Power of Generative AI models
Output produced by Generative AI models closely resembles that of humans.
- Relevant
- Contextual
- Imaginitive
- Nuanced
- Linguistically Accurate
There are 3 modules in this course
- Module 1:
- Concept of prompt engineering: how to define a prompt and its elements.
- Best practicers for effective prompts
- Common prompt engineering tools
- IBM Watsonx, Prompt Lab, Spellbook, Dust
- Module 2:
- Prompt engineering approaches:
- Interview pattern
- Chain of thoughts
- Tree of thoughts
- Techniques for skillfully crafting prompts
- Zero shot
- Few shot
- Precise and relevant responses.
- Module 3:
- Final project
- Graded quiz
1. What is a prompt?
A prompt is an "instruction" you provide to the Generative AI model. It is an input that produces a desired output. Example, write a small paragraph describing your favorite holiday destination. Another one is Write HTML code to generate dropdown selection of cities within an online form.
Prompts can also be a series of instructions that refine an output step by step to acheive a desired results.
> Write a short story about a scientist studying life in Mars.
> What were some of the challenges he faced during the research?
Gen AI then,
Building blocks of a well-constructed prompt:
Prompts contain questions, contextual texts, guiding patterns or examples and partial examples for the model.
Instructions: Give distinct guidelines regarding the task you wish to execute.
Example:
Write an essay in 600 words analyzing the effects of global warming on marine life.
Context: Provides a framework for generating relevant content.
Example
In the recent decades, global warming has undergone significant shifts, leading to rising sea levels, increased storm intensity, and changin weather patterns. These changes have had a severe impact on marine life. Write an essay in 600 words analyzing the effects of global warming on marine life.
Input Data: Any piece of information provided as a part of prompt. This can be used by the Generative AI model to obtain specefic data.
Example
You have been provided with a data set containing temperature records and measurements of sea levels in the Pacific Ocean. Write an essay in 600 words analyzing the effects global warming on marine life in the Pacific Ocean.
Output Indicator: Offers benchmarks for assessing attributes of the output. (Tone, length, quality of output)
Example
The output generated should be an essay of 600 words. It will be evaluated based on the clarity of analysis and incorporation of relevant data or case studies.
2. Prompt Engineering
The process of designing effective prompts to generate better and desired responses is called prompt engineering.
Prompt Engineering is a blend of:
- Critical Analysis
- Creativity
- Technical Acumen
It includes framing the question in the right context with the right information and your expectation of desired outcomes to elicit the most appropriate response.
Example
The captain of a ship is planning a strategic voyage in the Atlantic Ocean. To help the captain navigate effectively, provide weather forecasts for the upcoming week from 28th August 2023 to 1st September 2023. The coordinates of the target location are between 20 degrees north and 30 degrees north latitude and 40 degrees west and 20 degrees west longitude
To help plan an effective navigation in the Atlantic Ocean, provide detailed information about expected wind patterns, wave heights, precipitation probabilities, cloud cover, and any potential storms that might affect the voyage during a specified time frame and location
Process involved in prompte engineering
Define the Goal
Craft Initial Prompt
Test the Prompt
Analyze the Prompt
Specificity, providing additional context, or rephrasing.
Iterate the process
Last 3 of the above steps are repeated until results are satisfactory.
Benefits of prompt engineering
- Optimizes model efficiency
- Prompt engineering helps design intelligent prompts that allow the users to harness the full capabilities of these models without requiring extensive retraining.
- Boosting performance for specific tasks.
- Prompt engineering empowers generative AI models to deliver responses that are nuanced and have a context, rendering them more effective for specific tasks.
- Understanding model constraints
- Refining prompts through each iteration and studying the corresponding responses of the model can help us understand its strengths and weaknesses. Helps in enhancing future development of the models.
- Enhancing model security:
- Skilled prompt engineering can prevent issues of harmful content generation due to poorly designed prompts, thereby enhancing safe utilization of the model.
EXERCISES
Before doing the exercises let us learn some basics.
LLMs
Large language models (LLMs), like ChatGPT, are advanced artificial intelligence systems capable of understanding and generating human-like text. These models are trained on vast amounts of text data, learning the patterns, structures, and nuances of human language. They can perform various language-related tasks such as text generation, language translation, summarization, question answering, and more. The "large" in "large language models" refers to the vast size of the neural networks that power these models, typically containing hundreds of millions to billions of parameters.
Neural Network
A neural network is a computational model inspired by the structure and functioning of the human brain's interconnected network of neurons. It consists of layers of interconnected nodes, known as neurons or units. Each neuron receives input signals, processes them using weighted connections, and produces an output signal.
LLMs are pretrained on vast amounts of data to generate novel content, texts, images, and video.
Example with Prompt Instructions fed to drive the output in a specific format.
Adding Context-specifics to prompt to influence the output
Prompts in different formats
Prompts can be presented in different formats, such as questions, statements, or instructions. Let's experiment with how prompts in different formats can yield different results.
- What are the benefits of water reservoir?
- Discuss the advantages of water reservoirs.
- List the top five benefits of water reservoirs.
Limit the output length
- Create an announcement for starting a new job at ABCTech company as a lead data scientist in a tweet-length message.
- Supervise the style, tone and content of the output generated.
- Clarity: includes using simple and concise language.
- User clear and concise language.
- Avoid jaron, specialized terminology or complex terms.
- Provide explicit instructions.
- Context: provides background and required details.
- Establish the context
- Include relevant information.
- Precision: means being specific and providing examples.
- Be specific
- Use examples
- Role play / persona pattern: role play enhances responses by assuming a persona and offering relevant context.
- Assume a persona
- Provide a context for role play
LAB
Naive prompting and persona pattern
> What is the best way to get fit.
Now the same approach using persona.
> Acting as a fitness expert, tell me the best way to get fit.
You will act as a fitness expert who is current with the latest research data and provide very detailed step-by-step instructions in reply to my queries.
You'll see that we structured this in three parts:
We told the LLM the persona it should act as;
We specified a qualifier about being research-based;
We told the AI the response format we wanted.
Now the actual query is:
Create a gym workout program to lose weight and build strength for an out-of-shape beginner.
Output will look like below:
For an out-of-shape beginner looking to lose weight and build strength, here is a sample gym workout program that combines cardiovascular exercise, strength training, and flexibility work:
1. Warm-up (5-10 minutes):
- Start with a light cardio warm-up such as brisk walking on the treadmill or cycling.
- Follow with dynamic stretches to prepare your muscles for the workout ahead.
2. Cardiovascular Exercise (20-30 minutes):
- Choose a cardio machine such as the treadmill, elliptical, or stationary bike.
- Aim for a moderate intensity level where you can still hold a conversation but are slightly out of breath.
- Alternate between periods of higher intensity (1-2 minutes) and lower intensity (1-2 minutes) to increase calorie burn and cardiovascular fitness.
3. Strength Training (30-45 minutes):
- Perform compound exercises that target multiple muscle groups at once, such as squats, deadlifts, bench presses, and rows.
- Start with 2-3 sets of 10-12 repetitions for each exercise, using lighter weights to focus on form and technique.
- Gradually increase the weight as you get stronger and more comfortable with the movements.
- Include exercises for all major muscle groups, including legs, back, chest, shoulders, arms, and core.
4. Flexibility Work (10-15 minutes):
- Finish your workout with static stretches to improve flexibility and prevent injury.
- Focus on stretching all major muscle groups, holding each stretch for 15-30 seconds.
- Incorporate yoga poses or foam rolling to release tension and improve range of motion.
5. Cool Down and Hydration:
- End your workout with a light cardio cool down and more static stretches.
- Remember to hydrate throughout your workout and after to replenish fluids lost through sweat.
This sample gym workout program can be done 3-4 times per week, with rest days in between to allow your muscles to recover. It's important to listen to your body, start at a pace that feels comfortable, and gradually progress as you get stronger and more fit. Consult with a fitness professional or personal trainer to tailor the program to your individual needs and goals.
When the persona is someone famous
Naive prompt: Give me a list of 10 article titles to promote my new book about dog training
Now with a famous persona
Acting as marketing expert Seth Godin, give me a list of 10 article titles to promote my new book about dog training.
Common Prompt engineering tools
Prompt engineering tools provide features and functionalites to optmize prompts.
Functionalities
- Suggestions for prompts
- Contextual understanding
- Iterative refinement
- Bias mitigation
- Domain specific aid
- Libraries of predefined prompts
- Explicitly specify objective of LLM.
- Provide context
- Domain expertise - use domain specific terminology when needed - law, engineering, medicine.
- Bias mitigation - provide explicit instructions to generate neutral responses if you see any bias in responses.
- Framing - limit the word length or scope out the subject matter to ensure relevance.
Zero shot prompting - a method wherein model can generate meaningful responses to prompts without prior training. Example Select the adjective in this sentence. Anita bakes the best cakes in the neighbourhood - output is Best.
- User feedback loop - keep on looping until the output is close to your need.
Explainability
Is how well a user can understand and interpret the model's decision making process and the reasons behind the generated output. Explainability helps users, developers and stakeholders understand how the model works.
Explainability helps users, developers, and stakeholders understand how the model works. Why it makes certain predictions or generates specific text. And whether it can be trusted in various applications. Explainability is crucial to addressing ethical concerns related to AI. It helps all stakeholders evaluate and ensure the LLM's behavior is consistent with the specific domain's ethical guidelines and legal requirements.
Interview pattern approach
Involves designing prompts by simulating a conversation or interacting in the interview style.
Chain of thought approach
The Chain-of-Thought (CoT) methodology significantly bolsters the cognitive performance of AI models by segmenting complex tasks into more manageable steps. By adopting this prompting strategy, AI models can demonstrate heightened cognitive abilities and offer a deeper understanding of their reasoning processes.
Constructing series of prompts or questions to guide model to generate desired response.
It involves breaking down a complex task into smaller and easier ones through a sequence of more straightforward prompts, with each prompt building upon the previous one to guide the models toward the intended outcome. Before posing a question directly to the model, you feed it with related questions along with their corresponding solutions. This chain-of-prompts helps the model think about the problem and use the same strategy to answer more such questions correctly. In simpler words, the prompt includes a question and an accurate answer to the question to provide the required context and step-by-step reasoning for the model, then it poses a different question to be answered using the same line of reasoning.
Tree of Thought approach (ToT)
At its core, Chain-of-Thought prompting solicits a step-by-step thinking process from the LLM. Compared to the naive/standard/Input-Output prompting, we get far better results with it.
- ToT is built to expand the capabilities of Chain of Thought.
- Involves hirarchically structuring a prompt or query akin to a tree structure.
- Enables Generative AI models to demonstrate advanced reasoning capabilities.
- This approach is particularly useful when you want to provide explicit instructions or constraints to the model to ensure it generates the desired output.
- It involves generating multiple lines of thought resembling a decision tree to explore different possibilities and ideas. Unlike traditional linear approaches, this technique allows the model to evaluate and pursue multiple paths simultaneously. Each thought or idea branches out, creating a treelike structure of interconnected thoughts. The model proceeds by assessing every possible route, assigning numerical values according to its predictions of outcomes. And eliminating lesser promising lines of thought, ultimately pinpointing the most favorable choices.
- Generate multiple lines of thought resembling a decision tree to explore different possibilities and ideas.
- Each thought or idea branches out, creating a treelike structure of interconnected thoughts.
- The model proceeds by assessing every possible route, assigning numerical values according to its predictions of outcomes. And eliminating lesser promising lines of thought, ultimately pinpointing the most favorable choices.
Act as a human resource specialist design a recruitment and retention strategy for an e-commerce business, focusing on attracting and retaining skilled remote employees. Building such prompt instruction will allow the generative AI model to consider a step-by-step process and think logically. It will also make it consider intermediate thoughts, building upon them, and exploring branches that may or may not lead somewhere. This practice will maximize the use and capabilities of the model, rendering more useful results.
Exercise that shows GPT3.5 inadequacies
Dave Hulbert suggested a few rather convincing prompts that leverage this approach and yield, anedotically, great results. I particularly like how he incorporates the Persona pattern and recommend you approach ToT prompting using his prompts or similar variations you might develop yourself.
Bob is in the living room.
He walks to the kitchen, carrying a cup.
He puts a ball in the cup and carries the cup to the bedroom.
He turns the cup upside down, then walks to the garden.
He puts the cup down in the garden, then walks to the garage.
Where is the ball?
My output
I set up this context
Imagine three different experts are answering this question.
All experts will write down 1 step of their thinking, then share it
with the group. Then all experts will go on to the next step, etc.
If any expert realises they're wrong at any point then they leave. The question is...
and then prompted the question. Output was a below:
Then used the tree method
Simulate three brilliant, logical experts collaboratively answering a question. Each one verbosely explains their thought process in real-time, considering the prior explanations of others and openly acknowledging mistakes. At each step, whenever possible, each expert refines and builds upon the thoughts of others, acknowledging their contributions. They continue until there is a definitive answer to the question. For clarity, your entire response should be in a markdown table. The question is...
| Step | Expert 1 | Expert 2 | Expert 3 |
|------|----------|----------|----------|
| 1 | Bob is in the living room. | Bob is in the living room. | Bob is in the living room. |
| 2 | Bob walks to the kitchen, carrying a cup. | Bob walks to the kitchen, carrying a cup. | Bob walks to the kitchen, carrying a cup. |
| 3 | Bob puts a ball in the cup and carries the cup to the bedroom. | Bob puts a ball in the cup and carries the cup to the bedroom. | Bob puts a ball in the cup and carries the cup to the bedroom. |
| 4 | Bob turns the cup upside down in the bedroom. | Bob turns the cup upside down in the bedroom. | Bob turns the cup upside down in the bedroom. |
| 5 | Bob walks to the garden and puts the cup down. | Bob walks to the garden and puts the cup down. | Bob walks to the garden and puts the cup down. |
| 6 | Bob walks to the garage. | Bob walks to the garage. | Bob walks to the garage. |
| 7 | **Where is the ball?** | **Where is the ball?** | **Where is the ball?** |
Expert 1: The ball is in the cup in the bedroom.
Expert 2: The ball is in the cup in the bedroom.
Expert 3: The ball is in the cup in the bedroom.
| Step | Expert 1 | Expert 2 | Expert 3 |
|------|----------|----------|----------|
| 1 | 5 | 5 | 5 |
| 2 | 5 | 5 | 5 |
| 3 | 5 | 5 | 5 |
| 4 | 5 | 5 | 5 |
| 5 | 5 | 5 | 5 |
| 6 | 5 | 5 | 5 |
The consensus solution is that the ball is in the cup in the bedroom.
The same prompt when it was given to LLAMA 3 (80GB), response was below.
A classic lateral thinking puzzle!
Let's break it down step by step:
1. Bob starts in the living room.
2. He walks to the kitchen, carrying a cup.
3. In the kitchen, he puts a ball in the cup and carries the cup to the bedroom.
4. Then, he turns the cup upside down (which means the ball would fall out).
5. After that, he walks to the garden and puts the cup down there.
Now, let's think about what happens to the ball:
* When Bob puts it in the cup in the kitchen, the ball is inside the cup.
* When he turns the cup upside down in the bedroom, the ball would fall out of the cup. So, at this point, the ball is on the floor or bed, but we don't know exactly where.
* Finally, Bob leaves the cup (with no ball) in the garden.
So, where is the ball? Ah-ha! The answer is... it's still in the bedroom, on the floor or bed, after Bob turned the cup upside down and the ball fell out!
Let's break it down step by step:
1. Bob starts in the living room.
2. He walks to the kitchen, carrying a cup.
3. In the kitchen, he puts a ball in the cup and carries the cup to the bedroom.
4. Then, he turns the cup upside down (which means the ball would fall out).
5. After that, he walks to the garden and puts the cup down there.
Now, let's think about what happens to the ball:
* When Bob puts it in the cup in the kitchen, the ball is inside the cup.
* When he turns the cup upside down in the bedroom, the ball would fall out of the cup. So, at this point, the ball is on the floor or bed, but we don't know exactly where.
* Finally, Bob leaves the cup (with no ball) in the garden.
So, where is the ball? Ah-ha! The answer is... it's still in the bedroom, on the floor or bed, after Bob turned the cup upside down and the ball fell out!
Text to Image Prompt Techniquees using Stable Diffusion
An image prompt is a text descritpion of an image you want to generate.
Style Modifiers: are descriptors used to influence the artistic style or visual attributes of images.
- You can include Visual Elements: colour, contrast, texture, shape, size.
- You can include:
> Art styles
> Historical periods of art
> Photography techniques
> Types of art materials
> Traits of well known brands or artists
Examples:
A man running through the park, charming, cozy water colour illustration on a grey background.
A big, fat cow admist an ancient marketplace, (medieval manuscript drawing)
A dreamy, surreal landscape, pastel colours, with high contrast
Quality booster: are terms used to enhance the visual appeal
- Avoid blurriness and pixelation which leads to low quality images
- Noise reduction, sharpening, colour correction, resolution enhancement
- Use terms like:
High Resolution
2k, 4k, hyperdetailed
sharp focus
complementary colours
Examples
- Generate close-up image that highlights the texture of tree bark with 4k resolution
- Create a human portrait with sharp, crisp details and fine lines
- Create an image of a tall, gigantic fort with complementary colours and a blurred background for the subject to stand out.
Repetition: leverages the power of iterative sampling to enhance image diversity.
Repetition involves emphasizing a particular visual element within an image to create a sense of familiarity for the model, allowing it to focus on a specific idea or concept you want to highlight. This can be accomplished by repeating the same word or similar phrase within the image prompt. Repetition helps reinforce the message conveyed through the image and increase the memorability of the model. Rather than producing just one image based on a prompt, the model generates multiple images with subtle differences, resulting in a diverse set of potential outputs. This technique is particularly valuable when generative models are confronted with abstract or ambiguous prompts to which numerous valid interpretations are possible.
Weighted terms: are words or phrases that have a powerful emotion or psychological impact.
Words like, limited time offer, guaranteed, elicit a sense of urgency, security and trust.
Words like exclusive, premium, luxury, elicit a sense of exclusivity and sophistication.
Examples
- Craft an image of a cozy living room with a warm:10 | crackling:8 | fireplace.
- Generate a vibrant cityscape with shimmering:6 | neon-lit: 8 | skyscrappers.
- Depict a bustling street market, with colourful:-6 | exotic: 10 | food stalls.
Negative weight of 6 to colourful - so model will de-emphasise colourful and adds more weight to exotic.
Fix deformed generation
This is a technique used to fix deformities or anamolies that may impact image effectiveness ( such as distortion of human limbs).
Example
- Mother Teresa with waving hand [disfigured, deformed hands, distorted hands, distorted fingers, bad anatomy, bad hands]
- A smiling little girl [bad, ugly, deformed body, distorted face, cross-eyed, blurry]
- A man running on treadmill [bad legs, disfigured body, bad anatomy, photoshopped]
Negative prompts are in the square brackets which tell model what not to do.
Best Practices for writing Prompts
