Efficient Fine-Tuning Techniques for Large Language Models

DH Ragu's team optimizes large language models through parameter-efficient fine-tuning, balancing performance and resource constraints.

### Efficient Fine-Tuning of Large Language Models
DH Ragu, a senior researcher at IBM, discusses the evolution of fine-tuning large language models (LLMs). The transition from transfer learning to LLMs introduced **instruction fine-tuning and in-context learning** as key techniques. Despite the advantages of in-context learning, it has limitations in **performance and sensitivity to prompt engineering**, leading to the need for **more efficient fine-tuning methods**.

### Challenges of In-Context Learning and Fine-Tuning for LLMs
In-context learning for language models faces issues with **model variability** and **unclear inference from prompts**, making it difficult to **trust their outputs**. The need for **detailed instructions and in-context examples** can lead to inefficiencies in resource usage and **increased latency**. Full fine-tuning of large language models is challenging due to **high memory requirements**, **storage difficulties**, and the need for **extensive infrastructure**, limiting accessibility and hindering **research progress**.

### Parameter-Efficient Fine-Tuning for LLMs
Parameter-efficient fine-tuning (PFT) addresses the challenges of fully fine-tuning large language models (LLMs) by **freezing most parameters and training a small subset**, reducing hardware and memory requirements. This approach helps **mitigate overfitting and catastrophic forgetting**, allowing the model to learn tasks without losing its **acquired world knowledge**. PFT offers a practical solution for organizations with limited resources, enabling them to effectively utilize **older GPUs** and achieve **faster model convergence**.

### Parameter-Efficient Fine-Tuning Techniques
Parameter-efficient fine-tuning (PFT) offers advantages like **generalization to new domains** and **reduced storage needs**. Soft prompting, a key PFT technique, involves using **task-specific tokens** to train the model's prompt, while keeping the rest of the architecture **frozen**. This approach allows for **multitask serving** and **efficient scaling**, making it popular among LLM hosting companies.

### Fine-Tuning vs. Prompt Tuning in Large Models
Full fine-tuning is more effective than prompt tuning for smaller models, but as the model size increases, **prompt tuning starts to catch up**. Hard prompting, which involves giving explicit instructions and examples, is a **tedious process**, while soft prompting allows the model to **learn from a small amount of data**. The paper highlights that **increasing prompt length improves performance**, with a peak effect at **20 tokens**, and emphasizes the importance of **proper weight initialization for soft prompts**.

### The Importance of Layering in Neural Networks
Layering is crucial in neural networks to **stabilize training** and avoid issues like **unstable gradients**. The authors found that using a **smaller dimension for the parameterization of prefixes**, combined with a **larger MLP**, helped achieve stable training. Prefix tuning and adapter layers are two techniques used to optimize model training, with prefix tuning requiring only a **small percentage of parameters** for comparable results to full fine-tuning.

### The Role and Impact of Adapter Layers in Transformers
Adapter layers in transformers use a **bottleneck structure** with two feed-forward networks to **reduce the number of parameters significantly**. This structure allows for a trade-off between **performance and overfitting**, making it a useful hyperparameter for model training. However, adapter layers have faced challenges in terms of **performance, inference overhead, and increased latency**, leading to the exploration of alternative techniques like **prefix tuning and low-rank adaptation (LoRA)**.

### Intrinsic Dimensionality in Fine-Tuning Models
The paper discusses the importance of **intrinsic dimensionality** in fine-tuning models, suggesting that not all tasks require the **same number of dimensions**. A random projection matrix is proposed to achieve the **same performance with fewer dimensions**, and intrinsic dimensionality is defined as the value that provides **90% accuracy**. The concept is then applied to large language models, using a **structure-aware technique** to measure intrinsic dimensionality layer-wise, aiming to **reduce memory requirements and improve efficiency**.

### The Impact of Parameters on Intrinsic Dimensionality
The intrinsic dimensionality of a model is **high with a small number of parameters** and decreases steadily as the number of parameters increases. Structure-aware intrinsic dimensionality **reduces memory requirements and training time**, while also **lowering the dimensionality needed to learn a task**. The Lura paper offers a novel approach to fine-tuning by identifying which weights to modify and using a bottleneck structure to **factorize the weight update matrix**, leading to **better performance with fewer trainable parameters**.

### The Role of Rank in Model Training
The rank of a model influences its **intrinsic dimensionality** and the **number of parameters needed for task learning**. The A and B matrices are initialized strategically to ensure stable training, with B set to **zero** and A using a **GAN distribution**. Various extensions of the popular Laura technique have been explored, including QA, long Laura, Lura plus, and Dora, each offering different approaches to **optimize training and hyperparameter selection**.

### Efficient Fine-Tuning of Large Language Models: Strategies and Challenges
* 👨 ‍💻 DH Ragu, a senior researcher at IBM Research, leads the conversational AI team, focusing on **large language models and conversational AI** for over **12 years**.
* 📚 Ragu's academic background includes a master's and PhD from IIT Delhi, where he explored **electrical engineering** and **computer science**, respectively.
* 🔍 He discusses the evolution from traditional transfer learning to the current landscape of large language models (LLMs), highlighting the **two-phase approach of pre-training and fine-tuning**.
* 💡 The introduction of instruction fine-tuning and alignment phases has improved the way models leverage **unlabeled data**, although in-context learning has emerged as a popular method for applying LLMs to **various tasks**.
* ⚠ However, in-context learning has limitations, including **sensitivity to prompt design**, **lack of clarity in model assumptions**, and **inefficiencies in resource usage**, leading to concerns over **accuracy and reliability**.
* 🔄 Full fine-tuning offers a solution but comes with challenges such as **high hardware requirements**, **storage issues**, and the risk of **overfitting** due to the vast number of parameters in large models.
* 🔑 Parameter-efficient fine-tuning (PFT) emerges as a middle ground, allowing for the training of a **small subset of parameters** while freezing the majority, thus **reducing memory requirements** and **mitigating overfitting risks**.
* 🛠 Ragu concludes with a discussion on techniques like **prompt tuning** and **prefix tuning**, which optimize the fine-tuning process and enhance the model's ability to **generalize across tasks** while maintaining **efficiency**.

### Understanding Prefix Tuning and Its Variants in NLP Models
* 🧠 Prefix tuning introduces a new layer to convert token sequences into embeddings, which can be **directly fed into a neural network**, but must be **architected carefully to avoid unstable training**.
* ⚙ The authors discovered that using a simple parameterization for prefix tokens led to **unstable training**, where gradients could either **explode or vanish**, necessitating a **more complex approach**.
* 🔍 By representing the function F Theta with a smaller dimension, they achieved **more stable training**, allowing for a large multi-layer perceptron (MLP) while **reducing the risk of instability**.
* 📊 Evaluating prefix tuning on tasks like text-to-text and table-to-text generation demonstrated that only **0.1% of parameters** could yield results comparable to full fine-tuning, even on **unseen domains**.
* 🔗 Adapters, another popular technique, involve **adding new layers within transformer blocks**, projecting hidden dimensions **down and then back up**, which helps to **maintain stability during training**.
* 📉 Despite their efficiency, adapters face challenges like **increased latency and complexity** in swapping out configurations compared to other techniques like prefix tuning.
* 📈 Low-rank adaptation (LoRA) emerged as a significant advancement, leveraging **intrinsic dimensionality** to optimize the **number of trainable parameters** needed for effective fine-tuning in large language models.
* 🔄 LoRA's approach allows for **flexible parameter adjustments** and has inspired various extensions, ensuring its place as a **leading technique** in the field of NLP model fine-tuning.

LiveWiki

LLMs | Parameter Efficient Fine-Tuning (PEFT) | Lec 14.1

LLMs | Parameter Efficient Fine-Tuning (PEFT) | Lec 14.1