LLM / SLM Parameters

What do we understand by LLM or SLM Parameters?

**Parameters** in deep learning, including language models, are adjustable values that control the behavior of neural networks. These parameters are learned during training and determine how the model processes input data.

In LLMs and SLMs, parameters typically include:

1. **Weight matrices**: These matrices contain the numerical values that are multiplied by input vectors to produce output activations.

2. **Bias terms**: These are additive constants added to the weighted sum of inputs to adjust the activation function's output.

3. **Learned embeddings**: These are fixed-size vector representations of words, phrases, or tokens learned during training.

The number and complexity of these parameters directly impact the model's performance, accuracy, and computational requirements. More parameters often allow for more nuanced learning and better representation of complex linguistic patterns, but also increase the risk of overfitting and computational costs.

In the context of LLMs, having **billions** of parameters means that the model has an enormous number of adjustable values, allowing it to capture subtle relationships between words, contexts, and meanings. This complexity enables LLMs to achieve impressive results in tasks like language translation, question answering, and text generation.

Conversely, SLMs typically have fewer parameters (often in the tens or hundreds of thousands), which makes them more efficient but also less capable of capturing complex linguistic patterns.

LLM Quality vs Size small language models

 Courtesy: Microsoft.com 

LLM Architecture

LLMs - words vs tokens

https://kelvin.legal/understanding-large-language-models-words-versus-tokens/#:~:text=The%20size%20of%20text%20an,the%20cost%20and%20vice%20versa.

Tokens can be thought of as pieces of words. Before the model processes the prompts, the input is broken down into tokens. These tokens are not cut up exactly where the words start or end - tokens can include trailing spaces and even sub-words. -- Llama.

The size of text an LLM can process and generate is measured in tokens. Additionally, the operational expense of LLMs is directly proportional to the number of tokens it processes - the fewer the tokens, the lower the cost and vice versa. 

Tokenizing language translates it into numbers – the format that computers can actually process. Using tokens instead of words enables LLMs to handle larger amounts of data and more complex language. By breaking words into smaller parts (tokens), LLMs can better handle new or unusual words by understanding their building blocks.

NLP and LLM

Natural Language Processing and Large Language Models.

Natural Language Processing

NLP stands for Natural Language Processing. Imagine you're teaching a computer to understand and interact with human language, just like how ChatGPT and I are communicating right now. NLP involves developing algorithms and techniques to enable computers to understand, interpret, and generate human language in a way that's meaningful to us. It's what powers virtual assistants like Siri or Alexa, language translation services like Google Translate, and even spell checkers or autocomplete features in your smartphone keyboard.

NLP is the intersection of computer science, artificial intelligence, and linguists. 

For a computer to be able to process language, below steps are required:

Understanding Language Structure: At its core, NLP aims to teach computers how to understand the structure, meaning, and context of human language. This involves breaking down language into its fundamental components such as words, phrases, sentences, and paragraphs.

Tokenization: One of the initial steps in NLP is tokenization, where text is divided into smaller units called tokens. These tokens could be words, subwords, or characters, depending on the specific task and language being processed.

Syntax Analysis: NLP algorithms analyze the syntactic structure of sentences to understand the grammatical rules and relationships between words. Techniques like parsing help identify the subject, verb, object, and other parts of speech in a sentence.

Semantic Analysis: Beyond syntax, NLP also focuses on understanding the meaning of words and sentences. This involves techniques such as semantic parsing, word sense disambiguation, and semantic role labeling to extract the underlying semantics from text.

Named Entity Recognition (NER): NER is a crucial task in NLP where algorithms identify and classify entities such as names of people, organizations, locations, dates, and numerical expressions within text.

Sentiment Analysis: This branch of NLP involves determining the sentiment or emotion expressed in a piece of text. Sentiment analysis techniques range from simple polarity classification (positive, negative, neutral) to more nuanced approaches that detect emotions like joy, anger, sadness, etc.

Machine Translation: NLP plays a key role in machine translation systems like Google Translate, which translate text from one language to another. These systems employ techniques such as statistical machine translation or more modern neural machine translation models.

Question Answering Systems: NLP powers question answering systems like chatbots and virtual assistants. These systems understand user queries and generate appropriate responses by analyzing the semantics and context of the questions.

Text Generation: Another exciting area of NLP is text generation, where algorithms produce human-like text based on input prompts or contexts. Large language models, such as GPT (like the one you're talking to!), are capable of generating coherent and contextually relevant text across various domains.

NLP Success

NLP has seen remarkable success over the past few decades, with continuous advancements driven by research breakthroughs and technological innovations. Here are some key areas where NLP has made significant strides:

1. Machine Translation: NLP has revolutionized the field of translation, making it possible for people to communicate seamlessly across language barriers. Systems like Google Translate employ sophisticated NLP techniques to provide reasonably accurate translations for a wide range of languages.
2. Virtual Assistants and Chatbots: Virtual assistants such as Siri, Alexa, and Google Assistant have become integral parts of our daily lives, thanks to NLP. These systems understand and respond to spoken or typed queries, perform tasks like setting reminders, sending messages, and even provide personalized recommendations.
3. Information Retrieval and Search Engines: NLP powers search engines like Google to understand user queries and return relevant search results. Techniques like natural language understanding help search engines interpret the user's intent and deliver more accurate results.
4. Sentiment Analysis: NLP enables businesses to analyze large volumes of text data, such as customer reviews and social media posts, to gauge public sentiment towards products, services, or brands. Sentiment analysis tools help companies make informed decisions and improve customer satisfaction.
5. Text Summarization and Extraction: NLP techniques are used to automatically summarize long documents or extract key information from unstructured text data. This is particularly useful in fields like news aggregation, document summarization, and information retrieval.
6. Healthcare Applications: In healthcare, NLP is used for clinical documentation, medical record analysis, and extracting valuable insights from patient data. NLP-powered tools assist healthcare professionals in diagnosis, treatment planning, and medical research.
7. Language Generation [LLM which is a subset of NLP]: Recent advancements in large language models (LLMs) have enabled machines to generate human-like text with impressive coherence and fluency. These models can write articles, generate code, compose music, and even engage in creative writing tasks.
8. Accessibility Tools: NLP has contributed to the development of accessibility tools for individuals with disabilities, such as text-to-speech and speech-to-text systems, which enable people with visual or auditory impairments to interact with digital content more effectively.
   

Large Language Models (LLM)

While NLP has been successful in many tasks, LLMs like GPT (Generative Pre-Trained Transformer) have addressed several limitations and brought about significant advancements in the field. Below are the reasons why LLMs were developed despite the success of NLP.

1. Contextual Understanding: 
1. Traditional NLP approaches often struggled with understanding context across longer pieces of text or in ambiguous situations. LLMs, on the other hand, leverage deep learning techniques to capture contextual dependencies effectively, enabling them to generate more coherent and contextually relevant text.    

1. Scalability: 
2. Transfer Learning: 
3. Language Generation: 
4. Data Efficiency: 
5. Continual Learning: