Finetune Bart Model for Efficient Text Summaries

Author

Reads 864

Woman Left Eye
Credit: pexels.com, Woman Left Eye

Finetuning the BART model can be a game-changer for creating efficient text summaries. By adapting the pre-trained model to a specific task, you can significantly improve its performance.

The BART model is particularly well-suited for text summarization because it's designed to generate text from a given input, making it an ideal choice for condensing long pieces of text into concise summaries.

To get started with finetuning BART, you'll need to choose a suitable dataset for training, such as the CNN/Daily Mail dataset. This dataset consists of news articles and their corresponding summaries, providing a great starting point for training a text summarization model.

By using a dataset like CNN/Daily Mail, you can take advantage of the model's ability to learn from large amounts of labeled data and improve its summarization skills.

Here's an interesting read: Fine Tune Llm Colab

Preparation

Before we dive into fine-tuning BART, we need to prepare our test data set. This involves applying the same steps we used for the training data.

Credit: youtube.com, Fine-tuning Large Language Models (LLMs) | w/ Example Code

To prepare our test data, we need to add special tokens to the start and end of each sentence. This is a crucial step that we'll look at in more detail.

We also need to pad and truncate all sentences to a single constant length. This ensures that our model can process all input data consistently.

There are three required formatting steps: adding special tokens, padding and truncating, and explicitly differentiating real tokens from padding tokens with the "attention mask".

Data

Data is a crucial aspect of fine-tuning BART. The model was trained on 160GB of text from the English Wikipedia and BookCorpus dataset.

The training data includes "corrupted" or "noisy" text, which is mapped to clean or original text. This is done using denoising autoencoder techniques, where the model learns to remove noise and improve the quality of the text.

The authors of BART use five noising schemes: Token Masking, Token Deletion, Text Infilling, Sentence Permutation, and Document Rotation. However, only Text Infilling and Sentence Permutation are employed in training the final BART model, with about 30% of tokens being masked and all sentences permuted.

The dataset vocabulary size is around 29000, and the maximum length of the sequences is 512 characters in the clean data. To work with the dataset, we need to tokenize it, which involves splitting the sentence into tokens, adding special tokens, and mapping them to their IDs.

What Is Huggingface Transformer?

Credit: youtube.com, What is Hugging Face? (In about a minute)

HuggingFace Transformer models provide an easy-to-use implementation of some of the best performing models in natural language processing. They offer an implementation of the current state-of-the-art models in several NLP tasks such as text classification, text generation, text summarization, and question answering.

The original Transformer is based on an encoder-decoder architecture and is a classic sequence-to-sequence model. This architecture introduced a new form of learning for language-related tasks.

The Transformer models are first pre-trained on a large text corpus, such as BookCorpus or Wikipedia. This pretraining makes sure that the model “understands language” and has a decent starting point to learn how to perform further tasks.

The ability of the model to understand language is highly significant since it will determine how well you can further train the model for something like text classification or text summarization. The pre-trained weights and weights fine-tuned on question-answering, text summarization, conditional text generation, mask filling, and sequence classification are available in the BART HuggingFace model.

The transformers package from Hugging Face provides a pytorch interface for working with BERT, which is currently the most widely accepted and powerful pytorch interface for working with BERT. The library includes pre-built modifications of these models suited to your specific task.

Obtain the Dataset

Credit: youtube.com, 10 Free Dataset Resources for Your Next Project!

Obtaining a dataset is a crucial step in any data-driven project. You'll need a dataset that's relevant to your task and model.

For a Seq2Seq model like BART-large, you'll need a dataset with text and a target, such as keywords. The size of the dataset will depend on the model, but 8,000 examples are ideal. You can create your own dataset or use an existing one shared by someone else.

If you're using a custom CSV file, you can use the transformers library to create a dataset dict. Alternatively, you can import a dataset from Hugging Face in Colab.

Some popular datasets on Hugging Face include the English Wikipedia and BookCorpus dataset, which was used to train the BART model. This dataset contains 160GB of text and has a vocabulary size of around 29,000.

Here are some noising techniques used to train the BART model:

  • Token Masking: Random tokens are replaced with [MASK]
  • Token Deletion: Random tokens are deleted
  • Text Infilling: A fixed number of contiguous tokens are deleted and replaced with a single [MASK] token
  • Sentence Permutation: Sentences are permuted randomly
  • Document Rotation: The document is rearranged to start with a random token

To prepare your test data set, you'll need to apply the same steps as you did for the training data, including tokenizing the dataset and deciding on a maximum sentence length for padding or truncating.

How Many Parameters?

Credit: youtube.com, How many parameters does Chat GPT have?

BART has nearly 140M parameters, which is more than BERT's 110M parameters and GPT's 117M parameters.

BART outperforms fine-tuned BERT and its variations like RoBERTa on several tasks, likely due to its larger number of parameters.

RoBERTa, a model with 125M parameters in its base model, is still outperformed by BART on several fine-tuning tasks.

GPT also has a large number of parameters, but it's not clear if that's the reason for its performance.

The number of parameters in a model can greatly impact its performance on tasks, as seen with BART and RoBERTa.

Recommended read: Gpt3.5 Finetune

Model

The BART model is a seq2seq model that can be fine-tuned for various tasks. It's particularly useful for tasks that involve mapping speech to text or text to text, such as ASR.

To fine-tune the BART model, you can use datasets like LibriSpeech or choose from the many available at jim-schwoebel/voice_datasets. The model can also be fine-tuned on scholarly articles in specific domains such as AI or COVID-19 to generate summaries.

The BART model can be fine-tuned on datasets such as ScisummNet, which provides summaries of domain-specific literature, to generate context-aware, extractive paraphrasing for long research papers.

Readers also liked: Llama 3 8b Best Finetune Model

Architecture Explained

Credit: youtube.com, Transformers, explained: Understand the model behind GPT, BERT, and T5

A model's architecture is its underlying structure, which determines how it processes and transforms data.

It's made up of layers, which are like building blocks that work together to achieve a specific task.

Each layer has a specific function, such as feature extraction or classification.

In a neural network, layers are stacked on top of each other to form a hierarchical structure.

This allows the model to learn complex patterns and relationships in the data.

A typical architecture consists of an input layer, one or more hidden layers, and an output layer.

The input layer receives the data, the hidden layers process it, and the output layer produces the final result.

The number of layers and the type of layers used can be adjusted to suit the specific problem being solved.

For example, a model designed for image classification might use convolutional layers to extract features from the image data.

Fine-Tuned Model Management

Fine-tuning a model can result in large files, with the model weights taking up around 418 megabytes. This makes saving and loading models across sessions a bit of a challenge.

Credit: youtube.com, Fine Tune a model with MLX for Ollama

To save your model, you can download it to your local machine or copy it to your Google Drive. The largest file you'll need to manage is the model weights.

Saving your model to your Google Drive is a good idea, as it allows you to access your model from anywhere and share it with others.

Learning Rate Scheduler

The learning rate scheduler is a crucial component of our model's training process. It helps us adjust the learning rate during training, which is essential for convergence.

The authors recommend choosing from the following learning rate values for Adam: 5e-5, 3e-5, and 2e-5. These values are specified in Appendix A.3 of the BERT paper.

We've already set our learning rate to 2e-5, which is a good starting point. However, we should be aware that this might be too high for our specific model.

Here are the recommended learning rate values for Adam:

It's worth noting that the epsilon parameter eps = 1e-8 is used to prevent division by zero in the implementation. This is a small number that's added to the denominator to avoid any potential issues.

Training

Credit: youtube.com, 4 - Summarization Fine Tuning BART | GPT2 T5 PEGASUS using HuggingFace | NLP Hugging Face Project

Training a BART model involves several key steps. The training data includes "corrupted" or "noisy" text, which is mapped to clean or original text. The authors of BART use a combination of existing and new noising techniques for pretraining, including Token Masking, Token Deletion, Text Infilling, Sentence Permutation, and Document Rotation.

The noising schemes used are applied to 160GB of text from the English Wikipedia and BookCorpus dataset. The vocabulary size is around 29000, and the maximum length of the sequences is 512 characters in the clean data.

To train the model, you'll need to load the data onto a GPU for acceleration. This is done by unpacking the data inputs and labels, loading the data onto the GPU, and then clearing out the gradients calculated in the previous pass.

The training loop involves a forward pass, where the input data is fed through the network, and a backward pass, where the gradients are computed. The optimizer is then used to update the model parameters.

Credit: youtube.com, Tutorial 2- Fine Tuning Pretrained Model On Custom Dataset Using 🤗 Transformer

Here's a summary of the training process:

Notice that the validation loss is increasing, which suggests that the model is overfitting on the training data. Validation loss is a more precise measure than accuracy, as it takes into account the exact output value, not just which side of a threshold it falls on.

Evaluation

To evaluate our fine-tuned BART model, we use the Matthew's correlation coefficient (MCC) metric. This is because the classes are imbalanced, making MCC a more suitable choice.

The final score is based on the entire test set, but we can also look at the scores on individual batches to get a sense of the variability in the metric between batches. Each batch has 32 sentences, except for the last batch which has only 4 sentences.

We can combine the results for all batches and calculate our final MCC score. In fact, we can get a good score in about half an hour without doing any hyperparameter tuning.

Test Set Evaluation

Credit: youtube.com, Lec 5 Training and test set, Evaluation

Evaluating your model on a test set is a crucial step in the machine learning process. This is where you get to see how well your model performs on unseen data.

The Matthews correlation coefficient (MCC) is a popular metric used to evaluate performance on the CoLA benchmark, especially when classes are imbalanced.

To calculate MCC, you'll combine the results from all batches, each containing 32 sentences, except the last batch which has only 4 test sentences.

A good MCC score can be achieved without hyperparameter tuning, but keep in mind that accuracy can vary significantly between runs due to the small dataset size.

The expected accuracy for the CoLA benchmark is 49.23, according to the library documents.

Possible Problems

When working with large models like BART, you might encounter memory limits, especially if you're running on a free Colab account. This can be frustrating, but there are some things you can try to avoid this problem.

Credit: youtube.com, Episode 5 - Policy Evaluation: Challenges, Difficulties and What can be Improved

Lowering the batch size is often the first step to take. This involves reducing the number of samples processed together by the model at one time.

You can lower per_device_train_batch_size, which is the batch size to load on a single GPU/TPU/CPU. The default is 8, but you can lower it down to 1.

Another option is to lower gradient_accumulation_steps, which is the number of steps to accumulate the gradient before it performs a backwards pass. The default is 1, but a greater value means more memory used.

Lowering eval_accumulation_steps can also help. This is the number of prediction steps to accumulate the output tensors before moving the results to the CPU. If you leave it empty, the whole predictions are accumulated on the GPU, which is faster but takes more memory.

You can also try lowering the input data size by making a random sample dataset. Good results can be achieved with a smaller dataset.

Advantages and Further Work

Credit: youtube.com, Fine-tuning a Neural Network explained

Finetuning a pre-trained model like BART can save you a significant amount of time during development.

By leveraging the knowledge gained from the pre-training process, you can quickly adapt the model to your specific NLP task. Quicker Development is one of the key advantages of finetuning, as it allows you to get started with your project faster.

Here are some of the benefits of finetuning a pre-trained model:

With finetuning, you can achieve better results than training a model from scratch, even with less data. This is because the pre-trained model has already learned to recognize patterns in language, which can be fine-tuned for your specific task.

Advantages of Fine-Tuning

Fine-tuning pre-trained models like BERT can be a game-changer for text classification tasks. Specifically, it allows us to quickly develop a model for our task by adding an untrained layer on top of the pre-trained model.

By doing so, we can take advantage of the pre-trained model's knowledge and features, which can lead to better results. This is especially useful when we don't have a large dataset to train a specific deep learning model from scratch.

Credit: youtube.com, Prompt Engineering, RAG, and Fine-tuning: Benefits and When to Use

One of the main advantages of fine-tuning is that it's quicker than training a model from scratch. We can leverage the pre-trained model's weights and focus on fine-tuning them for our specific task.

Fine-tuning also requires less data than training a model from scratch. This is because the pre-trained model has already learned general features from a large dataset, so we don't need as much data to fine-tune it for our specific task.

Here are the key advantages of fine-tuning:

Further Work

The MCC score could be a more suitable metric for validation accuracy, but it was left out to avoid explaining it earlier in the process.

Using a fixed seed value at the beginning of the training loop may not actually guarantee reproducible results, as it's not clear if it's effective.

Running the example multiple times to demonstrate the variance in MCC score would be a valuable exercise.

In fact, running the example several times shows that the MCC score can vary substantially across different runs.

Installation and Setup

Credit: youtube.com, "BART" | UCLA CS 263 NLP Presentation

To finetune BART, you'll need to install the Hugging Face Library, which provides a PyTorch interface for working with BERT and other pre-trained language models.

The library is widely accepted and powerful, supporting a variety of pre-trained transformer models and pre-built modifications suited to specific tasks.

You can use the run_glue.py example script from Hugging Face to pick which GLUE benchmark task you want to run on and which pre-trained model to use, with options for CPU, single GPU, multiple GPUs, or 16-bit precision for speed up.

The code in this notebook is a simplified version of the run_glue.py script, with plenty of comments to make it clear what's going on.

To speed up training, you can use 16-bit precision, which is supported by the run_glue.py script.

The training epochs take around 2:57 each, as shown by the MAX_LEN = 64 setting.

Carrie Chambers

Senior Writer

Carrie Chambers is a seasoned blogger with years of experience in writing about a variety of topics. She is passionate about sharing her knowledge and insights with others, and her writing style is engaging, informative and thought-provoking. Carrie's blog covers a wide range of subjects, from travel and lifestyle to health and wellness.

Love What You Read? Stay Updated!

Join our community for insights, tips, and more.