A beginner-friendly guide with example (Python)\u00a0code<\/h4>\n
This is the third article in a larger series<\/a> on multimodal AI. In the previous posts, we discussed multimodal LLMs<\/a> and embedding models<\/a>, respectively. In this article, we will combine these ideas to enable the development of multimodal RAG systems. I\u2019ll start by reviewing key concepts and then share example code for implementing such a\u00a0system.<\/p>\n Language models like GPT, LLaMA, and Claude learn a tremendous amount of world knowledge via their pre-training. This makes them powerful tools for solving custom problems and answering complex questions.<\/p>\n However, there is knowledge that even the most advanced language models are ignorant of<\/strong>. This includes proprietary information within organizations, events that occurred after a model’s pre-training data collection, and specialized knowledge that is not prevalent on the internet.<\/p>\n Although this ignorance limits a model\u2019s out-of-the-box capabilities, there is a popular technique to overcome these limitations<\/strong>: retrieval augmented generation (or RAG for\u00a0short).<\/p>\n RAG<\/strong> is an approach for improving a model\u2019s response quality by dynamically providing the relevant context<\/strong> for a given prompt. Here\u2019s an example of when this might be\u00a0helpful.<\/p>\n Say, I forgot the name of a Python library a colleague mentioned in yesterday\u2019s meeting. This isn\u2019t something ChatGPT can help me with because it does not know the meeting\u2019s contents.<\/p>\n However, RAG could help with this by taking my question (e.g. \u201cWhat was the name of that Python library that Rachel mentioned in yesterday\u2019s meeting?\u201d), automatically pulling the meeting transcript, then providing my original query and the transcript to an\u00a0LLM.<\/p>\n Although improving LLMs with RAG unlocks several practical use cases, there are some situations where relevant information exists in non-text formats, e.g., images, videos, charts, and tables. In such cases, we can go one step further and build multimodal RAG systems<\/strong>, AI systems capable of processing text and non-text\u00a0data<\/strong>.<\/p>\n Multimodal RAG enables more sophisticated inferences beyond what is conveyed by text alone. For example, it could analyze someone\u2019s facial expressions and speech tonality to give a richer context to a meeting\u2019s transcription.<\/p>\n While there are several ways to implement a multimodal RAG (MRAG) system, here I will focus on three basic strategies at increasing levels of sophistication.<\/p>\n The following discussion assumes you already have a basic understanding of RAG and multimodal models<\/strong>. The following articles discussed these topics: RAG<\/a>, Multimodal LLMs<\/a>, and Multimodal Embeddings<\/a>.<\/p>\n A simple way to make a RAG system multimodal is by translating new modalities to text before storing them in the knowledge base<\/strong>. This could be as simple as converting meeting recordings into text transcripts, using an existing multimodal LLM (MLLM) to generate image captions, or converting tables to a readable text format (e.g.,\u00a0.csv or\u00a0.json).<\/p>\n The key upside of this approach is that it requires minimal changes to an existing RAG system<\/strong>. Additionally, by explicitly generating text representations of non-text modalities, one has better control over the features of the data to extract. For instance, captions of analytical figures may include both a description and key insights.<\/p>\n Of course, the downside of this strategy is that the model\u2019s responses cannot directly use non-textual data<\/strong>, which means that the translation from, say, image to text can create a critical information bottleneck.<\/p>\n Another approach is to generate text representations of all items in the knowledge base, e.g., descriptions and meta-tags, for retrieval, but to pass the original modality to a multimodal LLM (MLLM)<\/strong>. For example, image metadata is used for the retrieval step, and the associated image is passed to a model for inference.<\/p>\n This maintains many of the benefits of Level 1 while mitigating its limitations. Namely, text features of items in the knowledge base can be optimized for search, but the downstream model can use the full richness of each item\u2019s original modality.<\/p>\n The key difference with this approach is that it requires an MLLM<\/strong>, which is an LLM capable of processing non-text data<\/strong>. This unlocks more advanced reasoning capabilities, as demonstrated by models like GPT-4o or LLaMA 3.2\u00a0Vision.<\/p>\n Although we could use keyword-based search in the retrieval processes for Level 1 and Level 2, it is a common practice to use so-called vector search<\/strong>. This consists of generating vector representations (i.e., embeddings)<\/strong> of items in the knowledge base and then performing a search by computing similarity scores<\/strong> between an input query and each item in the knowledge base.<\/p>\n Traditionally, this requires that the query and knowledge base items are text-based. However, as we saw in the previous article<\/a> of this series, there exist multimodal embedding models<\/strong> that generate aligned vector representations of both text and non-text\u00a0data<\/strong>.<\/p>\n Therefore, we can use multimodal embeddings to perform multimodal retrieval. This works the same way as text-based vector search, but now the embedding space co-locates similar concepts independent of its original modality. The results of such a retrieval strategy can then be passed directly to a\u00a0MLLM.<\/p>\n With a basic understanding of how Multimodal RAG works, let\u2019s see how we can build such a system. Here, I will create a question-answering assistant that can access the text and figures from the previous two blogs in this\u00a0series.<\/p>\n The Python code for this example is freely available at the GitHub\u00a0repo<\/a>.<\/p>\n We start by importing a few handy libraries and\u00a0modules.<\/p>\n Next, we\u2019ll import text and image chunks from the Multimodal LLMs<\/a> and Multimodal Embeddings<\/a> blog posts. These are saved in\u00a0.json files, which can be loaded into Python as a list of dictionaries.<\/p>\n While I won\u2019t review the data preparation process here, the code I used is on the GitHub\u00a0repo<\/a>.<\/p>\n We will also load the multimodal embeddings (from CLIP) for each item in text_content_list <\/em>and image_content_list<\/em>. These are saved as pytorch\u00a0tensors.<\/p>\n Printing the shape of these tensors, we see they are represented via 512-dimensional embeddings. And we have 86 text chunks and 17\u00a0images.<\/p>\n With our knowledge base loaded, we can now define a query for vector search. This will consist of translating an input query into an embedding using CLIP. We do this similarly to the examples from the previous\u00a0post<\/a>.<\/p>\n Printing the shape, we see we have a single vector representing the\u00a0query.<\/p>\n To perform a vector search over the knowledge base, we need to do the following.<\/p>\n Here\u2019s what that looks like in code for the text\u00a0chunks.<\/p>\n Above, we see the top text results. Notice we only have one item, even though k<\/em>=5. This is because the 2nd-5th items were below the 0.1 threshold.<\/p>\n Interestingly, this item doesn\u2019t seem helpful to our initial query of \u201cWhat is CLIP\u2019s contrastive loss function?\u201d<\/em> This highlights one of the key challenges of vector search<\/strong>: items similar to a given query may not necessarily help answer\u00a0it<\/em>.<\/p>\n One way we can mitigate this issue is having less stringent restrictions on our search results by increasing k<\/em> and lowering the similarity threshold<\/em>, then hoping the LLM can work out what\u2019s helpful vs.\u00a0not.<\/p>\n To do this, I\u2019ll first package the vector search steps into a Python function.<\/p>\n Then, set more inclusive search parameters.<\/p>\n This results in 15 text results and 1 image\u00a0result.<\/p>\n Although most of these text item results do not seem helpful to our query, the image result is exactly what we\u2019re looking for. Nevertheless, given these search results, let\u2019s see how LLaMA 3.2 Vision responds to this\u00a0query.<\/p>\n We first will structure the search results as well-formatted strings.<\/p>\n Note the metadata that accompanies each text and image item. This will help the LLaMA better understand the context of the\u00a0content.<\/p>\n Next, we interleave the text and image results in a\u00a0prompt.<\/p>\n The final prompt is quite long, so I won\u2019t print it here. However, it is fully displayed in the example notebook<\/a> on\u00a0GitHub.<\/p>\n Finally, we can use ollama<\/a> to pass this prompt to LLaMA 3.2\u00a0Vision.<\/p>\n The model correctly picks up that the image contains the information it needs and explains the general intuition of how it works. However, it misunderstands the meaning of positive and negative pairs<\/strong>, thinking that a negative pair corresponds to a pair of the same modality.<\/p>\n While we went through the implementation details step-by-step, I packaged everything into a nice UI using Gradio in this notebook<\/a> on the GitHub\u00a0repo.<\/p>\n YouTube-Blog\/multimodal-ai\/3-multimodal-rag at main \u00b7 ShawhinT\/YouTube-Blog<\/a><\/p>\n Multimodal RAG systems can synthesize knowledge stored in a variety of formats, expanding what\u2019s possible with AI. Here, we reviewed 3 simple strategies for developing such a system and then saw an example implementation of a multimodal blog QA assistant.<\/p>\n Although the example worked well enough for this demonstration, there are clear limitations to the search process. A few techniques that may improve this include using a reranker to refine similarity search<\/strong> results and to improve search quality via fine-tuned multimodal embeddings<\/strong>.<\/p>\n If you want to see future posts on these topics, let me know in the comments\u00a0\ud83d\ude42<\/p>\n More on Multimodal models\u00a0\ud83d\udc47<\/strong><\/p>\n Multimodal AI<\/a><\/p>\n My website<\/strong>: https:\/\/www.shawhintalebi.com\/<\/a><\/p>\n [1] RAG<\/a><\/p>\n [2] Multimodal LLMs<\/a><\/p>\n [3] Multimodal Embeddings<\/a><\/p>\n Multimodal RAG: Process Any File Type with AI<\/a> was originally published in Towards Data Science<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>\n<\/div>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AKUfCT0odhqkTsPzf1ljx6A.png?ssl=1\")
What is\u00a0RAG?<\/strong><\/h3>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2ASTVyqpJkhoKZWYdR-2-xqA.png?ssl=1\")
Multimodal RAG<\/strong><\/h3>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AmqRCTYThFcZmGcVtw6s1cw.png?ssl=1\")
3 Levels of\u00a0MRAG<\/strong><\/h3>\n
\n
Level 1: Translate modalities to\u00a0text<\/strong><\/h4>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/839\/1%2A7QqhRIlnU7TQsCMnVDb6KA.png?ssl=1\")
Level 2: Text-only retrieval +\u00a0MLLM<\/strong><\/h4>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/825\/1%2AJoUZLYezY3q95zngSmDJIA.png?ssl=1\")
Level 3: Multimodal retrieval +\u00a0MLLM<\/strong><\/h4>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/839\/1%2AYwMdXXTGBMj9QSjAwkojdA.png?ssl=1\")
Example Code: Multimodal Blog Question-Answering Assistant<\/strong><\/h3>\n
Imports & Data\u00a0Loading<\/h4>\n
import json
from transformers import CLIPProcessor, CLIPTextModelWithProjection
from torch import load, matmul, argsort
from torch.nn.functional import softmax<\/pre>\n# load text chunks
with open('data\/text_content.json', 'r', encoding='utf-8') as f:
text_content_list = json.load(f)
# load images
with open('data\/image_content.json', 'r', encoding='utf-8') as f:
image_content_list = json.load(f)<\/pre>\n# load embeddings
text_embeddings = load('data\/text_embeddings.pt', weights_only=True)
image_embeddings = load('data\/image_embeddings.pt', weights_only=True)
print(text_embeddings.shape)
print(image_embeddings.shape)
# >> torch.Size([86, 512])
# >> torch.Size([17, 512])<\/pre>\nMultimodal Search<\/h4>\n
# query
query = \"What is CLIP's contrastive loss function?\"
# embed query (4 steps)
# 1) load model
model = CLIPTextModelWithProjection.from_pretrained(\"openai\/clip-vit-base-patch16\")
# 2) load data processor
processor = CLIPProcessor.from_pretrained(\"openai\/clip-vit-base-patch16\")
# 3) pre-process text
inputs = processor(text=[text], return_tensors=\"pt\", padding=True)
# 4) compute embeddings with CLIP
outputs = model(**inputs)
# extract embedding
query_embed = outputs.text_embeds
print(query_embed.shape)
# >> torch.Size([1, 512])<\/pre>\n\n
# define k and simiarlity threshold
k = 5
threshold = 0.05
# multimodal search over articles
text_similarities = matmul(query_embed, text_embeddings.T)
# rescale similarities via softmax
temp=0.25
text_scores = softmax(text_similarities\/temp, dim=1)
# return top k filtered text results
isorted_scores = argsort(text_scores, descending=True)[0]
sorted_scores = text_scores[0][isorted_scores]
itop_k_filtered = [idx.item()
for idx, score in zip(isorted_scores, sorted_scores)
if score.item() >= threshold][:k]
top_k = [text_content_list[i] for i in itop_k_filtered]
print(top_k)<\/pre>\n# top k results
[{'article_title': 'Multimodal Embeddings: An Introduction',
'section': 'Contrastive Learning',
'text': 'Two key aspects of CL contribute to its effectiveness'}]<\/pre>\ndef similarity_search(query_embed, target_embeddings, content_list,
k=5, threshold=0.05, temperature=0.5):
\"\"\"
Perform similarity search over embeddings and return top k results.
\"\"\"
# Calculate similarities
similarities = torch.matmul(query_embed, target_embeddings.T)
# Rescale similarities via softmax
scores = torch.nn.functional.softmax(similarities\/temperature, dim=1)
# Get sorted indices and scores
sorted_indices = scores.argsort(descending=True)[0]
sorted_scores = scores[0][sorted_indices]
# Filter by threshold and get top k
filtered_indices = [
idx.item() for idx, score in zip(sorted_indices, sorted_scores)
if score.item() >= threshold
][:k]
# Get corresponding content items and scores
top_results = [content_list[i] for i in filtered_indices]
result_scores = [scores[0][i].item() for i in filtered_indices]
return top_results, result_scores<\/pre>\n# search over text chunks
text_results, text_scores = similarity_search(query_embed, text_embeddings,
text_content_list, k=15, threshold=0.01, temperature=0.25)
# search over images
image_results, image_scores = similarity_search(query_embed, image_embeddings,
image_content_list, k=5, threshold=0.25, temperature=0.5)<\/pre>\n1 - Two key aspects of CL contribute to its effectiveness
2 - To make a class prediction, we must extract the image logits and evaluate
which class corresponds to the maximum.
3 - Next, we can import a version of the clip model and its associated data
processor. Note: the processor handles tokenizing input text and image
preparation.
4 - The basic idea behind using CLIP for 0-shot image classification is to
pass an image into the model along with a set of possible class labels. Then,
a classification can be made by evaluating which text input is most similar to
the input image.
5 - We can then match the best image to the input text by extracting the text
logits and evaluating the image corresponding to the maximum.
6 - The code for these examples is freely available on the GitHub repository.
7 - We see that (again) the model nailed this simple example. But let\u2019s try
some trickier examples.
8 - Next, we\u2019ll preprocess the image\/text inputs and pass them into the model.
9 - Another practical application of models like CLIP is multimodal RAG, which
consists of the automated retrieval of multimodal context to an LLM. In the
next article of this series, we will see how this works under the hood and
review a concrete example.
10 - Another application of CLIP is essentially the inverse of Use Case 1.
Rather than identifying which text label matches an input image, we can
evaluate which image (in a set) best matches a text input (i.e. query)\u2014in
other words, performing a search over images.
11 - This has sparked efforts toward expanding LLM functionality to include
multiple modalities.
12 - GPT-4o \u2014 Input: text, images, and audio. Output: text.FLUX \u2014 Input: text.
Output: images.Suno \u2014 Input: text. Output: audio.
13 - The standard approach to aligning disparate embedding spaces is
contrastive learning (CL). A key intuition of CL is to represent different
views of the same information similarly [5].
14 - While the model is less confident about this prediction with a 54.64%
probability, it correctly implies that the image is not a meme.
15 - [8] Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex
Capabilities<\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/800\/1%2Arq89PAcqQ_lHgkYhkf5T4g.png?ssl=1\")
Prompting MLLM<\/h4>\n
text_context = \"\"
for text in text_results:
if text_results:
text_context = text_context + \"**Article title:** \"
+ text['article_title'] + \"n\"
text_context = text_context + \"**Section:** \"
+ text['section'] + \"n\"
text_context = text_context + \"**Snippet:** \"
+ text['text'] + \"nn\"<\/pre>\nimage_context = \"\"
for image in image_results:
if image_results:
image_context = image_context + \"**Article title:** \"
+ image['article_title'] + \"n\"
image_context = image_context + \"**Section:** \"
+ image['section'] + \"n\"
image_context = image_context + \"**Image Path:** \"
+ image['image_path'] + \"n\"
image_context = image_context + \"**Image Caption:** \"
+ image['caption'] + \"nn\"<\/pre>\n# construct prompt template
prompt = f\"\"\"Given the query \"{query}\" and the following relevant snippets:
{text_context}
{image_context}
Please provide a concise and accurate answer to the query, incorporating
relevant information from the provided snippets where possible.
\"\"\"<\/pre>\nollama.pull('llama3.2-vision')
response = ollama.chat(
model='llama3.2-vision',
messages=[{
'role': 'user',
'content': prompt,
'images': [image[\"image_path\"] for image in image_results]
}]
)
print(response['message']['content'])<\/pre>\nThe image depicts a contrastive loss function for aligning text and image
representations in multimodal models. The function is designed to minimize the
difference between the similarity of positive pairs (text-image) and negative
pairs (text-text or image-image). This loss function is commonly used in CLIP,
which stands for Contrastive Language-Image Pre-training.
**Key Components:**
* **Positive Pairs:** Text-image pairs where the text describes an image.
* **Negative Pairs:** Text-text or image-image pairs that do not belong to
the same class.
* **Contrastive Loss Function:** Calculates the difference between positive
and negative pairs' similarities.
**How it Works:**
1. **Text-Image Embeddings:** Generate embeddings for both text and images
using a multimodal encoder (e.g., CLIP).
2. **Positive Pair Similarity:** Calculate the similarity score between each
text-image pair.
3. **Negative Pair Similarity:** Calculate the similarity scores between all
negative pairs.
4. **Contrastive Loss Calculation:** Compute the contrastive loss by
minimizing the difference between positive and negative pairs' similarities.
**Benefits:**
* **Multimodal Alignment:** Aligns text and image representations for better
understanding of visual content from text descriptions.
* **Improved Performance:** Enhances performance in downstream tasks like
image classification, retrieval, and generation.<\/pre>\nConclusion<\/h3>\n
<\/p>\n
\n