
Fine tuning OpenAI embeddings can be a game-changer for your natural language processing (NLP) projects.
By adjusting the weights of the pre-trained embeddings, you can adapt them to your specific task and improve performance.
To fine tune OpenAI embeddings, you'll need to choose a suitable model architecture, such as a transformer or a CNN.
Curious to learn more? Check out: Evolutionary Algorithms
Preparation
Before diving into fine-tuning OpenAI embeddings, it's essential to prepare your data and infrastructure. This involves making design decisions about datasets and models.
To get started, you'll need to choose a dataset that's relevant to your project. This dataset will serve as the foundation for your fine-tuning process.
Datasets, Models, and Infrastructure
To fine-tune a model, you'll need to decide on a dataset. You can use datasets provided by Hugging Face.
Choosing the right dataset is crucial, as it will greatly impact the performance of your model. I've found that using a high-quality dataset can make a big difference in the accuracy of your results.
To create custom art of your pet, you'll need to define your base CLIP model using openai-clip. This will give you a solid foundation to work from.
In terms of infrastructure, you'll need to install relevant libraries. You'll be using torch to facilitate model loading, device management, tensor manipulation, and inference, as well as tqdm to track the progress of fine-tuning.
Having the right infrastructure in place will save you a lot of time and hassle in the long run.
Load Model and Preprocess
Loading the CLIP model and its preprocessing function is the first step in our preparation process.
The CLIP model used is ViT-B/32, which is a pre-trained model that has already been trained on a large dataset.
This model is then moved to the appropriate device, either the GPU if available, or the CPU if not.
The CLIP model is used for image classification by comparing the visual features of images with textual descriptions of subcategories.
The model processes and normalizes the features of the images and subcategory texts, calculates their similarity, and predicts the subcategory for each image.
For example, the model was used to classify three example images from our dataset, but it only identified one of the three images correctly.
Expand your knowledge: How to Fine Tune T5 Model
Processing the Dataset
To process a dataset, it's essential to split it into training and validation sets, taking 80% for training and the remaining 20% as validation data. This allows us to evaluate the performance of our machine learning model on unseen data.
We can then create a custom dataset class, which initializes the dataset object with data and sets up a series of transformations to preprocess the images. These transformations include resizing images to 224x224 pixels, converting them to tensors, and normalizing them with specific mean and standard deviation values.
A custom dataset class typically includes three key methods: __init__, __len__, and __getitem__. The __init__ method initializes the dataset object, the __len__ method returns the number of samples in the dataset, and the __getitem__ method retrieves an image and its corresponding subcategory from the dataset.
Here are the three methods in a custom dataset class:
- __init__ method: Initializes the dataset object with data and sets up transformations.
- __len__ method: Returns the number of samples in the dataset.
- __getitem__ method: Retrieves an image and its corresponding subcategory from the dataset.
After creating the custom dataset class, we can create DataLoaders for the training and validation sets. The training DataLoader has a batch size of 32 and shuffling enabled to randomize the order of samples, while the validation DataLoader has a batch size of 32 and shuffling disabled to maintain the order of samples.
By following these steps, we can effectively process our dataset and prepare it for training and validation.
Model Selection
Choosing a base model is crucial for fine-tuning OpenAI embeddings. We'll be focusing on permissively licensed models with available weights.
You can find these models on Hugging Face's git LFS-based model repository hub. This is where we'll get our models.
Each model is trained differently and with a specific use-case in mind, so it's essential to compare their performance on public benchmarks like MTEB.
Base Model Selection
Choosing a base model can be a daunting task, but it's essential to get it right. We'll be focusing on permissively licensed models with available weights, which can be downloaded and modified like open source code.
These models are commonly released via Hugging Face's git LFS-based model repository hub, where you can find a wide range of options.
Each model is trained differently and with a specific use-case in mind, so it's crucial to compare their performance on public benchmarks like MTEB. This will give you an idea of how well they'll perform in your specific use case.
The embedding dimension, or the number of entries in the output embeddings of the model, is an important consideration. Larger vectors can store more information, but can also result in higher costs as you embed more data over time.
Ultimately, the right base model for you will depend on your specific needs and resources.
GPT-3.5 vs GPT-4
Fine-tuning GPT-3.5 to distill GPT-4 responses is a viable option, as shown in the example of using GPT-4 to automatically generate questions from unstructured context.
We can use GPT-4 to generate "ground-truth" answers through a GPT-4 query engine flow, which can then be logged to a dataset using an OpenAIFineTuningHandler callback.
This process allows us to launch a fine-tuning job and obtain a distilled model that can be evaluated against a naive GPT-3.5 workflow using Ragas.
GPT-3.5 can be fine-tuned using OpenAI's finetuning endpoints, as demonstrated in the GPT-3.5 Fine-tuning Notebook (Colab) and GPT-3.5 Fine-tuning Notebook (Notebook link) /examples/finetuning/react_agent/react_agent_finetune.ipynb.
The fine-tuning process involves launching a job and getting back a distilled model, which can be benchmarked against a naive GPT-3.5 workflow.
Here are the resources available for fine-tuning GPT-3.5:
- GPT-3.5 Fine-tuning Notebook (Colab)
- GPT-3.5 Fine-tuning Notebook (in Repo)
Base vs Models
Comparing base and fine-tuned models can be a crucial step in model selection. The base model, for instance, does not support MRL.
Fine-tuning a model can lead to significant improvements in performance. In fact, fine-tuning for just 10 epochs on a domain-specific dataset led to an 8% improvement in NDCG@10.
The fine-tuned model outperformed the base model in various metrics, including at 64 dimensions. The fine-tuned model at 64 dimensions was even better than the base model at 768 dimensions.
Interestingly, the fine-tuned model performed best at 512 and 256 dimensions, showcasing the strength of MRL.
What Is Clip?
CLIP is a groundbreaking model developed by OpenAI that combines computer vision and natural language processing. It's trained on a massive dataset of images paired with textual descriptions.
CLIP's innovative approach lies in its ability to process both images and text into a shared embedding space. This is made possible through two main components: an image encoder and a text encoder.
The image encoder converts images into embeddings, while the text encoder performs the same function for text. These embeddings are then aligned using contrastive learning, which brings the embeddings of matching image-text pairs closer together and pushes apart those of non-matching pairs.
CLIP's capacity to learn from extensive image-text pairs empowers it to perform a variety of tasks without needing task-specific fine-tuning. This versatility makes CLIP an attractive choice for applications like image classification and object detection.
Using a GPU like the T4 GPU on Google Colab can significantly speed up the training process for CLIP models.
You might like: How to Fine Tune Clip Model
Context
Fine-tuning your embedding model is a powerful concept, but it has its limitations.
The SentenceTransformersFinetuneEngine is limited to fine-tuning sentence_transformers models.
Fine-tuning the embedding model requires reindexing your documents, which can be a hassle.
This problem was also highlighted by Jo (Vespa) in the Finetuning + RAG webinar, who explored the concept of freezing document embeddings using a foundation model.
Hyperparameter Tuning
Fine-tuning hyperparameters can be a complex task, but it's made simpler by using pre-existing training recipes. Determining how to fine-tune a model is a research project that should be scoped accordingly.
We selected three experimental parameters as most important: which pre-trained model to train, on how much data, and with how many output dimensions. These parameters determine the values of the parameters in the model, so they're known as hyperparameters.
The simplest way to explore hyperparameters is to define a set of possible values for each hyperparameter and then check all combinations - a grid search. This approach is effective and easy to parallelize.
We added two additional models to the original bge-base-en-v1.5 model and tried two different embedding dimensions. For each configuration, we tested a different dataset size, ranging from one hundred to over one hundred thousand samples.
All other hyperparameters were held fixed. We generated all possible combinations of model, dataset size, and dense output features using the product function from itertools. This creates an iterator that returns every possible combination of elements from each input list.
We then used these combinations to generate configuration objects for our model fine-tuning process. Our training process fits on a single GPU, and each experiment runs in parallel, so we can scale it up to the maximum number of simultaneous GPU workers allowed in Modal.
For more insights, see: Grid Search Hyperparameter Tuning
Fine Tuning
Fine-tuning is a process that can significantly improve the performance of OpenAI embeddings. It allows you to customize the model to better suit your specific needs by training it on your own data.
You can fine-tune OpenAI embeddings using a scalable, serverless infrastructure on Modal, which can be easily run with just a few dozen users interacting with a service a few times a day. This can create hundreds of datapoints in a matter of days.
Fine-tuning gives you a 5-10% increase in retrieval evaluation metrics, making it a worthwhile investment of time and resources.
To fine-tune OpenAI embeddings, you'll need to follow a few steps: generate a synthetic question/answer dataset using LlamaIndex, fine-tune the model, and evaluate the model. This process can be done using a variety of tools and techniques, including the EmbeddingAdapterFinetuneEngine abstraction.
Fine-tuning can be done on top of any existing embedding model, including SBERT embeddings, OpenAI embeddings, Cohere embeddings, and more. This means you can plug in a fine-tuned linear adapter after you've generated embeddings for your documents.
Here are some ways to fine-tune OpenAI embeddings:
- Finetuning Embeddings
- Fine-tuning our Linear Adapter
- Fine-tuning Cross-Encoders for Re-Ranking
- Fine-tuning Llama 2 for Better Text-to-SQL
Featured Images: pexels.com





