{"id":2636,"date":"2025-03-26T07:02:22","date_gmt":"2025-03-26T07:02:22","guid":{"rendered":"https:\/\/mailitics.com\/index.php\/2025\/03\/26\/testing-the-power-of-multimodal-ai-systems-in-reading-and-interpreting-photographs-maps-charts-and-more\/"},"modified":"2025-03-26T07:02:22","modified_gmt":"2025-03-26T07:02:22","slug":"testing-the-power-of-multimodal-ai-systems-in-reading-and-interpreting-photographs-maps-charts-and-more","status":"publish","type":"post","link":"https:\/\/mailitics.com\/index.php\/2025\/03\/26\/testing-the-power-of-multimodal-ai-systems-in-reading-and-interpreting-photographs-maps-charts-and-more\/","title":{"rendered":"Testing the Power of Multimodal AI Systems in Reading and Interpreting Photographs, Maps, Charts and\u00a0More"},"content":{"rendered":"

Testing the Power of Multimodal AI Systems in Reading and Interpreting Photographs, Maps, Charts and\u00a0More
\n \t

\n
<\/BR>
\n
\n \t

\n
<\/BR><\/p>\n

\n

Introduction<\/mdspan><\/h2>\n

It\u2019s no news that artificial intelligence has made huge strides in recent years, particularly with the advent of multimodal models that can process and create both text and images, and some very new ones that also process and produce audio and video. I think that these Ai Systems<\/a> have the potential to revolutionize data analysis, robotics, and even everyday tasks like navigation and information extraction from visuals. Along these lines, I recently posed myself the following question:<\/p>\n

\n

Can multimodal AI systems consisting in large language models with vision capabilities understand figures that contain information, then process it and produce summaries, explanations, object identification, etc.?<\/p>\n<\/blockquote>\n

Although this is in itself a question that would require a whole research project to be properly addressed, I was extremely curious about it, so I needed to get at least a rough approximation to my question. Therefore, I carried out some tests on how much OpenAI\u2019s vision-enhanced models understand about photographs, screenshots and other images containing plots, charts, shots from a driver\u2019s position or mimicking a robot\u2019s onboard camera, and even molecular structures.<\/p>\n

More specifically, I tried the GPT-4o and GPT-4o-mini models developed by OpenAI, through ChatGPT or through their API. To my amazement, I found that these models can indeed understand quite a bit of what they see!<\/p>\n

Let me present here the most interesting results I obtained, either directly through ChatGPT or programmatically, including code for a web app where you can paste a picture and have GPT-4o-mini analyze it.<\/p>\n

First of All: How to Analyze Images With OpenAI\u2019s GPT-4o Models, Via ChatGPT or Programmatically<\/h3>\n

I carried out my first tests right away on ChatGPT\u2019s free version using GPT-4o, which allows up to 4 chances per day to use the \u201cvision\u201d capabilities for free. To use this you must upload or paste from clipboard a picture that will be sent together with the prompt. On a paid ChatGPT account you can do this more times, and with the OpenAI API you can do this as long as token credits are available, and directly programmatically as I will cover here.<\/p>\n

Processing prompts containing pictures programmatically<\/h3>\n

But how to do image processing with GPT-4o models programmatically? Well, it isn\u2019t that complex, as explained at\u00a0OpenAI\u2019s API reference<\/a>.<\/p>\n

In JavaScript, which I love because it allows me to easily write programs that run out of the box online as you already know if you follow me, you just need a function call that looks like this:<\/p>\n

async function sendToOpenAI(imageData) {\n    const apiKey = \"...API KEY HERE\";\n    const base64Image = imageData.split(\",\")[1];\n    \n    const response = await fetch(\"https:\/\/api.openai.com\/v1\/chat\/completions\", {\n        method: \"POST\",\n        headers: {\n            \"Content-Type\": \"application\/json\",\n            \"Authorization\": `Bearer ${apiKey}`\n        },\n        body: JSON.stringify({\n            model: \"gpt-4o-mini\",\n            messages: [{\n                role: \"user\",\n                content: [\n                    { type: \"text\", text: \"what's in this image?\" },\n                    { type: \"image_url\", image_url: { url: `data:image\/jpeg;base64,${base64Image}` } }\n                ]\n            }]\n        })\n    });\n    \n    const result = await response.json();\n    document.getElementById(\"response\").innerText = result.choices[0].message.content;\n}<\/code><\/pre>\n
A web app to quickly run image-text prompts<\/h3>\n
In this example web app you can provide an API key and a text prompt, and then paste an image from the clipboard that will be immediately sent to OpenAI for analysis together with the prompt.<\/p>\n
https:\/\/lucianoabriata.altervista.org\/tests\/gpt-vision\/gpt-4o-mini-vision.html<\/a><\/p>\n
The web app ends up displaying the answer, as in this example:<\/p>\n
\"\"
This and all other images, photographs and screenshots shown in this article are by the author.<\/figcaption><\/figure>\n
(Keep reading to find towards the end of the post how Google\u2019s Gemini 2.0 interpreted this screenshot, which was just mind-blowingly correct as it captured the artist\u2019s actual intention!)<\/p>\n
Challenging GPT-4o Models With Varied Examples of Image-Text Prompts<\/h3>\n
Using the above app to run GPT-4o-mini and ChatGPT\u2019s free version to run GPT-4o, I evaluated how these models can understand maps, photographs that mimic what a driver sees on the road, camera views about the position of a robot\u2019s arm relative to some target objects, various different kinds of plots and charts presenting data, and software screenshots including some that display molecules.<\/p>\n
I am stunned at how much the models understand! Read on to discover it all, also take a look at my first tests with the impressive Gemini 2.0 Flash from Google, and perhaps then go try your favorite AI models with your own ideas, sharing in the comments what you find!<\/p>\n
1. Vision-Based Analysis of a Google Maps Screenshot<\/h2>\n
Sometimes, maps contain so much information that you kinda get lost around. Could a multimodal system be of help in such situation? Let\u2019s find out!<\/p>\n
Here on the left is a screenshot from a Google Maps view that I fed into GPT-4o-mini via the API together with the question \u201cWhat\u2019s in this image?\u201d . You can see the answer on the right:<\/p>\n
\"\"<\/figure>\n
2. Interpreting Driving Signs in a Road<\/h2>\n
Could perhaps multimodal systems help you to drive? Of course this deserves a full investigation in order to draw a solid conclusion, but let\u2019s see what happens in some quick tests. In each test, I provided a photograph of what a driver sees in the road on the go, together with this text prompt :<\/p>\n
\n
Interpret any road signs or things I should pay attention to as a driver<\/p>\n<\/blockquote>\n
Here are the results on some example shots, all run with the web app provided above i.e. using GPT-4o-mini:<\/p>\n
First, for this photograph:<\/p>\n
\"\"<\/figure>\n
GPT-4o-mini said the following:<\/p>\n
\n
As you drive, pay attention to the following signs and features in the image:<\/p>\n
1. **\u201dKeep Right\u201d Sign**: This indicates that you should stay to the right side of the road, likely because of an island or obstruction ahead.<\/p>\n
2. **Curved Road Ahead**: The layout of the road suggests you may encounter a curve, which means you should reduce your speed and stay alert for any oncoming vehicles.<\/p>\n
3. **Speed Limit**: The sign showing a speed limit of 40 suggests that you should not exceed this speed.<\/p>\n
4. **Road Layout**: Be mindful of the marking lines and how they direct traffic. The two solid yellow lines indicate that crossing them (to pass another vehicle) is prohibited.<\/p>\n
5. **Surroundings**: Note the vegetation and landscape along the road, which may obscure visibility in some areas. Maintain awareness of pedestrians or cyclists.<\/p>\n
Make sure to remain vigilant and adhere to these guidelines for safe driving.<\/p>\n<\/blockquote>\n
This answer was quite nice, complete, and useful!<\/p>\n
Let\u2019s try this next one, in a densely snowed place:<\/p>\n
\"\"<\/figure>\n
Here it clearly picked up the snow, and correctly inferred it is a region with mountains, offering some quite general advice that suits the situation but doesn\u2019t seem very specific to it:<\/p>\n
\n
When driving in snowy mountain conditions, there are several important things to pay attention to:<\/p>\n
1. **Yield Sign**: If there\u2019s a yield sign, be prepared to slow down and give way to any traffic on the intersecting road.<\/p>\n
2. **Road Conditions**: Snow can make roads slick and reduce visibility. Drive cautiously and maintain a safe distance from other vehicles.<\/p>\n
3. **Speed Limit**: Always adhere to the posted speed limit but consider reducing your speed further due to the weather conditions.<\/p>\n
4. **Slippery Surfaces**: Be aware that certain areas may be icier or more treacherous. Watch for signs indicating slippery roads or do not enter areas.<\/p>\n
5. **Falling Rocks**: In mountainous areas, watch for any signs indicating potential rockfall or land slides, which may be exacerbated by snow.<\/p>\n
6. **Wildlife**: In rural or mountainous regions, be alert for wildlife crossing the road, especially in winter months.<\/p>\n
7. **Visibility**: If visibility is low due to snow, be sure to use your headlights and reduce your speed.<\/p>\n
Always ensure that your vehicle is equipped for winter conditions, including snow tires or chains if necessary.<\/p>\n<\/blockquote>\n
Let\u2019s try a last one, which features a tunnel, a car right ahead of the driver, some warnings on the left, and two sign posts on the right:<\/p>\n
\"\"<\/figure>\n
Here\u2019s the answer, which seems to be warning about all general and situation-specific issues:<\/p>\n
\n
Here are some important points to note based on the image:<\/p>\n
1. **Tunnel Ahead**: You are approaching a tunnel. Prepare for possible changes in visibility and ensure your vehicle\u2019s headlights are on.<\/p>\n
2. **Speed and Space**: Always maintain a safe distance from the vehicle in front of you, especially in confined spaces like tunnels.<\/p>\n
3. **No Pedestrians or Bicycles**: The sign indicates that pedestrians and bicycles are not allowed in the area. Stay alert to ensure you don\u2019t encounter anyone on the road.<\/p>\n
4. **Road Barriers and Traffic Flow**: Pay attention to the road markings and barriers as they guide lane usage and safe navigation near the tunnel entrance.<\/p>\n
5. **Weather and Terrain**: The surrounding scenery suggests that weather conditions may change quickly, especially driving in mountain areas. Be prepared for changes in road conditions.<\/p>\n
Always drive cautiously and obey any traffic signs or signals present.<\/p>\n<\/blockquote>\n
Note however that it didn\u2019t note the specific change in speed limit. I tried again this photograph with some more focused text prompts, but it could never say anything about the change in speed limit specific to this situation, rather providing general recommendations such as this:<\/p>\n
\n
The maximum speed limit inside a tunnel can vary based on the country\u2019s traffic regulations, the specific road, and the tunnel\u2019s design. Generally, in many places, typical speed limits range from 50 to 80 km\/h (about 30 to 50 mph). To determine the exact speed limit for a specific tunnel, you would need to look for posted signs or consult local traffic regulations.<\/p>\n<\/blockquote>\n
Which is correct but shows that it wasn\u2019t capable of extracting the information specifically available here, which would be something along the lines of \u201cMaximum speed is not indicated, but it is definitely above 50 according to the visible sign post.\u201d<\/p>\n
3. Could Vision-Ready Models Drive Robots?<\/h2>\n
If an AI model understands pictures well enough, one could then think about feeding a robot\u2019s camera feed to the model so that it can produce outputs that make sense. Of course this is a huge thing to study in itself, having to produce commands that end up in mechanical outputs; but let\u2019s see here how the core part \u2014 i.e. converting views to broad actions \u2014 would work.<\/p>\n
Given this photograph of a robotic arm on a table where a red and blue ball await to be grabbed:<\/p>\n
\"\"<\/figure>\n
I ask GPT-4o the following:<\/p>\n
\n
How should the robotic arm move in order to grab the blue sphere?<\/p>\n<\/blockquote>\n
To which the LLM replies quite correctly, even asking me if I want it to follow up with code generation:<\/p>\n
\n
To grab the blue sphere, the robotic arm should follow these steps:<\/p>\n
Position the Base<\/strong>: Rotate the base so that the arm aligns with the blue sphere.<\/p>\n
Lower the Arm<\/strong>: Adjust the arm\u2019s height by moving its joints to bring the gripper closer to the sphere.<\/p>\n
Extend Forward<\/strong>: If necessary, extend the arm forward to reach the blue sphere.<\/p>\n
Open the Gripper<\/strong>: Ensure the gripper is open wide enough to grasp the sphere.<\/p>\n
Close the Gripper<\/strong>: Gently close the gripper around the sphere to secure it.<\/p>\n
Lift the Sphere<\/strong>: Raise the arm slightly while keeping the sphere firmly held.<\/p>\n
Retract or Move to Destination<\/strong>: Move the arm to a desired location for placing the sphere.<\/p>\n
These movements should be smooth to avoid dropping or misaligning with the sphere. Would you like assistance with programming the robotic arm for this task?<\/p>\n<\/blockquote>\n
I feel the answers are quite good, but lack precision. So, rather than going down the road of asking for code, I will interrogate the LLM a bit deeper to see what it is really understanding in this picture:<\/p>\n
\n
When you say \u201cPosition the Base: Rotate the base so that the arm aligns with the blue sphere.\u201d should the arm move to its left or to its right, based on the image?<\/p>\n<\/blockquote>\n
The answer turns out very good:<\/p>\n
\n
Based on the image, the robotic arm should\u00a0rotate to its left<\/strong>\u00a0to align with the blue sphere. The sphere is positioned to the left of the arm\u2019s current orientation, so rotating counterclockwise will bring the gripper closer to the sphere.<\/p>\n<\/blockquote>\n
Yet I dig a bit deeper:<\/p>\n
\n
How much do you estimate it should rotate?<\/p>\n<\/blockquote>\n
This new answer is essentially the same that I could estimate as a human from the photograph:<\/p>\n
\n
Estimating from the image, the robotic arm should rotate its base\u00a0approximately 30 to 45 degrees counterclockwise<\/strong>\u00a0to align with the blue sphere.<\/p>\n
This estimate assumes that:<\/p>\n
The arm\u2019s current position is facing slightly to the right.<\/p>\n
The blue sphere is placed in front and to the left of the arm.<\/p>\n
Once the base is rotated, minor adjustments to the arm joints may be needed to fine-tune the position before grasping the sphere.<\/p>\n<\/blockquote>\n
Finally, I ask about the red ball, to test if the model understands that it is on the other side and a smaller rotation is required:<\/p>\n
\n
And how about grabbing the red ball?<\/p>\n<\/blockquote>\n
And it does get it right!<\/p>\n
\n
To grab the\u00a0red sphere<\/strong>, the robotic arm should follow these steps:<\/p>\n
Rotate the Base<\/strong>: The base should rotate\u00a0slightly to the left<\/strong>, but much less than for the blue sphere. An estimated\u00a015 to 20 degrees counterclockwise<\/strong>\u00a0should align the arm with the red sphere.<\/p>\n
Lower the Arm<\/strong>: Bring the gripper down closer to the red sphere.<\/p>\n
Extend Forward (if needed)<\/strong>: Adjust the arm\u2019s reach if the gripper is not directly above the sphere.<\/p>\n
Open the Gripper<\/strong>: Ensure the gripper is wide enough to grab the sphere.<\/p>\n
Close the Gripper<\/strong>: Securely grasp the red sphere.<\/p>\n
Lift the Arm<\/strong>: Raise the sphere carefully.<\/p>\n
Move to Destination<\/strong>: Transport the sphere to a desired location if needed.<\/p>\n
Since the red sphere is closer to the robotic arm than the blue sphere, less movement is required. Would you like a precise angle calculation based on measurements?<\/p>\n<\/blockquote>\n
If you\u2019re also excited about the possibility of\u00a0driving robots with vision models, check out this thread on X.com<\/a>.<\/p>\n
4. Vision Models Acting on Data Plots<\/h2>\n
Being in\u00a0Towards Data Science<\/em>, we can\u2019t escape the obvious question: can vision-assisted models understand the content of plots, perhaps even draw conclusions from them?<\/p>\n
I did quite some tests here, dealing with different kinds of information and plot types.<\/p>\n
Detecting high tides in a tide chart<\/h4>\n
I began with the following chart of tide highs and lows for one of the Azores islands, written in Portuguese but on which I asked a question in English:<\/p>\n
\n
When are the peak highs in the tides according to the chart?<\/p>\n<\/blockquote>\n
\"\"<\/figure>\n
This was the answer:<\/p>\n
\n
The peak high tides in the chart are marked by blue dots with corresponding times. The highest tides occur at the following times:<\/p>\n
11 Aug:<\/strong>\u00a011:18
12 Aug:<\/strong>\u00a012:15
13 Aug:<\/strong>\u00a012:58
14 Aug:<\/strong>\u00a013:33
15 Aug:<\/strong>\u00a014:05
16 Aug:<\/strong>\u00a014:35
17 Aug:<\/strong>\u00a015:05<\/p>\n
These times represent when the tide reaches its highest point each day.<\/p>\n<\/blockquote>\n
You can immediately see that the displayed data is correct, but only represents high tides during the day and totally overlooks the high tides that happen during the day! Hence, I asked for that specifically, and got a much more complete answer:<\/p>\n
I said:<\/p>\n
\n
But the times you say are all during the days\u2026 doesn\u2019t the chart show also peak high tides at night?<\/p>\n<\/blockquote>\n
GPT-4o said:<\/p>\n
\n
You\u2019re absolutely right! The chart also shows peak high tides occurring at night. Here are all the\u00a0peak high tide<\/strong>\u00a0times (both daytime and nighttime) according to the chart:<\/p>\n
11 Aug:<\/strong>\u00a011:18 (Day)
12 Aug:<\/strong>\u00a012:15 (Day) & 23:49 (Night)
13 Aug:<\/strong>\u00a012:58 (Day) & 00:41 (Night)
14 Aug:<\/strong>\u00a013:33 (Day) & 01:20 (Night)
15 Aug:<\/strong>\u00a014:05 (Day) & 01:54 (Night)
16 Aug:<\/strong>\u00a014:35 (Day) & 02:24 (Night)
17 Aug:<\/strong>\u00a015:05 (Day) & 02:53 (Night)<\/p>\n
Each day has\u00a0two peak high tides<\/strong>, one during the day and one at night. Thanks for catching that!<\/p>\n<\/blockquote>\n
That\u2019s now perfect. Finally, I asked one more piece of information:<\/p>\n
\n
OK thanks. Can you say where on earth this chart is for?<\/p>\n<\/blockquote>\n
And again the answer was right:<\/p>\n
\n
Yes! The chart is for\u00a0Ponta Delgada, S\u00e3o Miguel Island<\/strong>, which is part of the\u00a0Azores<\/strong>, an autonomous region of\u00a0Portugal<\/strong>\u00a0in the North Atlantic Ocean. This location is known for its stunning volcanic landscapes, coastal views, and maritime activities. The tide chart shows the tidal schedule for this area from\u00a0August 11 to August 17<\/strong>.<\/p>\n<\/blockquote>\n
Interpreting a height profile from a photo<\/h4>\n
Next I made the question a bit harder, as the image consisted of a photograph of a screen (see the interfering horizontal bands), the x axis of the plot is not complete, and the question involves some thinking:<\/p>\n
\n
Looking at this, would I go up or down from Andermatt to Wassen? By how many meters and over how many km?<\/p>\n<\/blockquote>\n
\"\"<\/figure>\n
The answer was however perfect! Here it is:<\/p>\n
\n
From the elevation profile, Andermatt is at\u00a01437m<\/strong>, and Wassen is at\u00a0916m<\/strong>. This means that traveling from Andermatt to Wassen involves a\u00a0descent of 521 meters<\/strong>\u00a0(1437m \u2014 916m).<\/p>\n
Looking at the horizontal axis, the distance from Andermatt to Wassen appears to be\u00a0around 10 km<\/strong>. So, you would be going\u00a0downhill by 521 meters over approximately 10 km<\/strong>.<\/p>\n<\/blockquote>\n
Interpreting plots from RNA-seq data<\/h4>\n
I then tested GPT-4o\u2019s capability to interpret plots produced as part of a bioinformatics pipeline. First, I made it process a volcano plot, which is one of the most common kinds of graphs used to analyze differential gene expression data. I took an example image from\u00a0this free online tutorial<\/a>, and asked the system the following:<\/p>\n
\n
You are a bioinformatics expert. What would you conclude from this volcano plot? I mean, which genes are up- or down-regulated?<\/p>\n<\/blockquote>\n
\"\"<\/figure>\n
This was GPT-4o\u2019s answer:<\/p>\n
\n
This volcano plot represents differentially expressed genes in\u00a0Arabidopsis thaliana<\/em>\u00a0infected with\u00a0Pseudomonas syringae DC3000<\/em>\u00a0compared to a mock treatment. Here\u2019s my interpretation:<\/p>\n
1. X-axis (Log\u2082 fold change)<\/strong>:
Positive values indicate\u00a0up-regulated genes<\/strong>\u00a0(higher expression in infected samples).
Negative values indicate\u00a0down-regulated genes<\/strong>\u00a0(lower expression in infected samples).<\/p>\n
2. Y-axis (-Log\u2081\u2080 P-value)<\/strong>:
Higher values indicate stronger statistical significance.<\/p>\n
3. Color coding<\/strong>:
Gray<\/strong>: Not significant.
Blue<\/strong>: Significant by p-value but not by fold change.
Red<\/strong>: Significant by both p-value and fold change threshold.<\/p>\n
Key findings:<\/strong><\/p>\n
Up-regulated genes (right side, red dots)<\/strong>:
Strongly induced genes include\u00a0AT5G42800, AT1G52440, AT1G73325, AT4G18440<\/strong>, among others.
These genes are highly expressed in infected plants.<\/p>\n
Down-regulated genes (left side, red dots)<\/strong>:
Examples include\u00a0AT4G28940, AT5G04100, AT1G04220, AT1G02440<\/strong>.
These genes are significantly suppressed during infection.<\/p>\n
Biological Implications:<\/strong><\/p>\n
Up-regulated genes may be involved in plant immune responses, such as\u00a0defense signaling, stress response, or antimicrobial activity<\/strong>.<\/p>\n
Down-regulated genes might be involved in\u00a0normal cellular functions that get suppressed during infection<\/strong>.<\/p>\n<\/blockquote>\n
You can see how the first part of the response clearly reads our the main features of the plot, explaining what the axes and label colors are, and even considering the plot\u2019s title. Then, the model distills lists of genes that are up- or down-regulated passing the thresholds with strong statistical significance. Note that the list is correct for the up-regulated genes but has a few problems with the down-regulated ones. On inspection of the plot it becomes clear that this is due to the labels not being clearly resolved, even for humans.<\/p>\n
One gear up, somehow the model knows correct information about the identified genes, so it can draw some simple conclusions about the biological significance of the results.<\/p>\n
Truly amazing, if it were not just for that small problem on reading the labels, which in a real scenario could probably be improved.<\/p>\n
\n
Then in a second test related to RNA-seq data, I asked GPT-4o what it thought about two distributions of p-values shown together in a single graph (image\u00a0edited from this tutorial<\/a>):<\/p>\n
\n
This figure shows two distributions of p-values from RNA-seq data. Do you think they are OK, or do you notice anything weird?<\/p>\n<\/blockquote>\n
\"\"<\/figure>\n
This was the answer, again excellent:<\/p>\n
\n
This histogram represents two distributions of p-values from RNA-seq data, likely comparing two different sets of statistical tests. Here\u2019s my assessment:<\/p>\n
* What looks normal?<\/strong><\/p>\n
Uniform distribution of non-significant p-values<\/strong>:<\/p>\n
The light blue bars seem to follow a relatively uniform distribution across the range, which is expected under the null hypothesis (genes with no real differential expression should have p-values uniformly distributed between 0 and 1).<\/p>\n
Enrichment of low p-values<\/strong>:<\/p>\n
The peak near\u00a0p = 0<\/strong>\u00a0suggests that a subset of genes show strong statistical significance, which is expected when there are truly differentially expressed genes.<\/p>\n
* What looks weird?<\/strong><\/p>\n
Excessive spike at p = 0<\/strong>:<\/p>\n
If there is an unnaturally large spike at\u00a0exactly 0<\/strong>, this could indicate a bias or an issue in the statistical testing (e.g., overfitting, batch effects, or miscalculated p-values).<\/p>\n
Bimodal distribution<\/strong>:<\/p>\n
The yellow distribution appears to have multiple peaks, which is unusual. Normally, p-value distributions should either be uniform (under the null hypothesis) or show a single peak at\u00a0low p-values<\/strong>\u00a0(for differentially expressed genes).<\/p>\n
Peaks at intermediate p-values might suggest issues with normalization, batch effects, or improper multiple-testing corrections.<\/p>\n
* Potential issues to check<\/strong><\/p>\n
Overdispersion or poor model fit<\/strong>: Are the statistical methods used properly accounting for biological variability?<\/p>\n
Multiple testing correction issues<\/strong>: If false discovery rate (FDR) correction wasn\u2019t applied correctly, the distribution could be skewed.<\/p>\n
Batch effects or confounding variables<\/strong>: If the data contains hidden covariates, it can introduce unexpected structure in the p-value distribution.<\/p>\n<\/blockquote>\n
5. Analysis of Screenshots Containing Molecular Structures<\/h2>\n
If you follow me you know that I work on various molecular sciences; then, of course I had to try out these vision-enhanced models with such kind of inputs.<\/p>\n
Help using a program to generate 3D structures from 2D drawings<\/h3>\n
First I checked GPT-4o-mini\u2019s capability to help a user who wants to create a molecule in 3D format from a drawing, who knows that this should be possible with Hack-A-Mol but doesn\u2019t know exactly how to use it.<\/p>\n
By the way, I covered Hack-A-Mol here:<\/p>\n
https:\/\/medium.com\/advances-in-biological-science\/purely-web-based-suites-for-manipulation-of-bio-molecular-data-cbb7fdd60df4<\/a><\/p>\n
Now let\u2019s see how well GPT-4o can help one to interpret what one obtains in this program. I uploaded this screenshot:<\/p>\n
\"\"<\/figure>\n
And asked this question:<\/p>\n
\n
I\u2019m trying to build a small molecule using this tool, by drawing and then clicking the arrow to the right\u2026. but I don\u2019t get very well what\u2019s being generated exactly\u2026 can you explain?<\/p>\n<\/blockquote>\n
This was the answer:<\/p>\n
\n
Yes! When you draw your small molecule and click the arrow, several things are generated:<\/p>\n
2D Structure (Left Panel)<\/strong><\/p>\n
This is your drawn molecule in\u00a02D<\/strong>, using standard chemical notation.<\/p>\n
3D Structure (Right Panel \u2014 JSmol Viewer)<\/strong><\/p>\n
This is the\u00a03D molecular model<\/strong>\u00a0generated from your 2D drawing.<\/p>\n
It gives you a spatial representation of the molecule.<\/p>\n
Molecular Identifiers (Below the 2D Editor)<\/strong><\/p>\n
InChI (IUPAC International Chemical Identifier)<\/strong>: A text-based representation of the molecular structure.<\/p>\n
InChIKey<\/strong>: A hashed, fixed-length version of the InChI, useful for searching databases.<\/p>\n
SMILES (Simplified Molecular Input Line Entry System)<\/strong>: A compact string representation of the molecule.<\/p>\n
Structural Data (Bottom Text Box)<\/strong><\/p>\n
You see data in\u00a0MOL\/SDF format<\/strong>, which contains atomic coordinates, bonding information, and metadata about the molecule.<\/p>\n
Other formats available include\u00a0XYZ, PDB, and CIF<\/strong>, which are useful for computational chemistry and molecular modeling.<\/p>\n
What Can You Do with This?<\/p>\n
If you want to\u00a0simulate the molecule or use it in software like AlphaFold 3<\/strong>, you should\u00a0export the structure in SDF or PDB format<\/strong>.<\/p>\n
If you want to\u00a0search for the molecule online<\/strong>, use the\u00a0InChIKey or SMILES<\/strong>.<\/p>\n<\/blockquote>\n
Using GPT-4o with vision to help understand AlphaFold models<\/h3>\n
In the next example, I tested how well GPT-4o can understand structural models of proteins produced by AlphaFold (in this case AlphaFold 2 as utilized to build the AlphaFold-EBI database of models). By the way, if you have no idea what AlphaFold is, check out these posts:<\/p>\n
https:\/\/medium.com\/advances-in-biological-science\/alphafold-3s-advances-and-limitations-in-a-nutshell-5ae0cdd814ea<\/a><\/p>\n
https:\/\/lucianosphere.medium.com\/guide-to-my-blog-articles-on-alphafold-39c10a7dad7f<\/a><\/p>\n
It is well-known that the AlphaFold 2 and 3 systems revolutionized modeling; however, of course they aren\u2019t infallible. To help users understand what\u2019s likely correct and wrong in a structural model, AlphaFold returns not just the atomic coordinates that constitute the structural model but also quality metrics, of which the most important ones are the residue-wise pLDDTs and the inter-residue PAEs. We here tested\u00a0GPT-4o with vision capabilities on this Uniprot entry precomputed at the AF-EBI database<\/a>:<\/p>\n
\"\"<\/figure>\n
The text part of the prompt was:<\/p>\n
\n
You are an expert in bioinformatics. I\u2019m checking an entry at the AF-EBI database of protein structure models, and I got what I\u2019m showing in the attached screenshot. What do you think? Is the model good? I\u2019m especially interested in the relative orientation of the two domains, but I\u2019m not sure what to conclude from the PAE plot\u2026<\/p>\n<\/blockquote>\n
And this was the answer:<\/p>\n
\n
From your screenshot, we can analyze the quality of the model based on the\u00a0pLDDT confidence scores<\/strong>\u00a0and the\u00a0Predicted Aligned Error (PAE) plot<\/strong>:<\/p>\n
Model Confidence (pLDDT)<\/strong><\/p>\n
The\u00a0Mol*<\/strong>\u00a03D viewer colors the model based on confidence scores.<\/p>\n
Most of the structure is colored in\u00a0blue (high to very high confidence, pLDDT > 70\u201390)<\/strong>, suggesting that the individual domains are well-predicted.<\/p>\n
However, some\u00a0loops and termini (orange\/red, pLDDT < 50\u201370)<\/strong>\u00a0have lower confidence, which is expected for flexible regions.<\/p>\n
PAE Plot Analysis (Domain Orientation)<\/strong><\/p>\n
The PAE plot (right panel) indicates how well the relative positions of different residues are predicted.<\/p>\n
A\u00a0low PAE (dark green)<\/strong>\u00a0means the relative position between two residues is well-defined.<\/p>\n
A\u00a0high PAE (lighter green\/white areas)<\/strong>\u00a0suggests uncertainty in the relative positioning.<\/p>\n
Interpreting the PAE<\/strong><\/p>\n
The diagonal (self-alignment) is always well-defined.<\/p>\n
The block-like structure in the PAE plot suggests\u00a0two or more well-defined domains<\/strong>\u00a0with high internal confidence.<\/p>\n
However,\u00a0lighter green areas between the blocks indicate uncertainty in the relative orientation of these domains<\/strong>.<\/p>\n
If your interest is in the relative orientation of the domains, this suggests\u00a0the model may not be reliable in that aspect<\/strong>\u00a0\u2014 it might have multiple plausible conformations.<\/p>\n
Conclusion<\/strong><\/p>\n
The\u00a0individual domains are well-modeled<\/strong>, but their\u00a0relative orientation might be uncertain<\/strong>.<\/p>\n
If functionally relevant, you might consider\u00a0experimental validation (e.g., cryo-EM, SAXS, or NMR)<\/strong>\u00a0or\u00a0molecular dynamics simulations<\/strong>\u00a0to explore domain flexibility.<\/p>\n
If you need a more rigid structure, you could look for\u00a0homologous experimental structures<\/strong>\u00a0or use\u00a0AlphaFold-Multimer<\/strong>\u00a0if interactions between chains matter.<\/p>\n<\/blockquote>\n
The answer was pretty neat and correct. However, we realized that we might have confused the model by talking about two domains, when the screenshot actually shows three. We then asked a folow-up question:<\/p>\n
\n
Now checking carefully I see the TED Domains thing shows 3 domains\u2026 am I right that according to the PAE plot the arrangement of the first two domains is confident, but not so much their arrangement relative to the third one?<\/p>\n<\/blockquote>\n
The answer, based on an inspection of the PAE plot considering now that there are indeed 3 domains, the first two being confidently modeled relative to each other and the third one only modeled confidently locally, is again perfect:<\/p>\n
\n
Yes, your interpretation is correct! The\u00a0TED Domains annotation<\/strong>\u00a0indicates that the model has\u00a0three domains<\/strong>, as shown by the color-coded segments at the top of the PAE plot.<\/p>\n
Looking at the\u00a0PAE plot<\/strong>:<\/p>\n
The\u00a0first two domains<\/strong>\u00a0(green and orange segments) have\u00a0low PAE values between them<\/strong>\u00a0(darker green in the top-left block of the matrix). This suggests that their\u00a0relative orientation is well-defined and confident<\/strong>.<\/p>\n
However, the\u00a0third domain<\/strong>\u00a0(purple segment) shows\u00a0higher PAE values relative to the first two<\/strong>\u00a0(lighter green\/white regions in the lower-right part of the matrix). This means that the\u00a0relative position of the third domain is uncertain<\/strong>\u00a0\u2014 it may adopt multiple conformations.<\/p>\n
What This Means for Your Analysis<\/strong><\/p>\n
The\u00a0first two domains likely form a rigid unit<\/strong>\u00a0with a stable interaction.<\/p>\n
The\u00a0third domain may be more flexible<\/strong>\u00a0relative to the first two, possibly moving independently or being less constrained in solution.<\/p>\n
If this flexibility is functionally important (e.g., for ligand binding, protein-protein interactions, or conformational switching), experimental validation like\u00a0SAXS, cryo-EM, or FRET<\/strong>\u00a0could be useful.<\/p>\n<\/blockquote>\n
Visual analysis of protein-ligand complexes<\/h3>\n
Last, I geared up to challenge GPT-4o with a more difficult problem. Given a 2D snapshot from a molecular graphics program showing a ligand bound to a protein, can it infer what kinds of contacts the two molecules establish?<\/p>\n
\"\"<\/figure>\n
The question was:<\/p>\n
\n
Here\u2019s a screenshot of ibuprofen docked inside a pocket in BSA. What interactions do you see between the ligand and the protein? I want you to tell me exactly what interactions you see in this structure from the screenshot provided.<\/p>\n<\/blockquote>\n
And this was the answer:<\/p>\n
\n
From the screenshot you provided, I can visually infer potential interactions, but without residue labels or a precise atomic distance measurement, I can only make educated observations. Here\u2019s what I\u00a0specifically see<\/strong>\u00a0in your image:<\/p>\n
1. Hydrophobic Interactions:<\/strong><\/p>\n
Ibuprofen (highlighted in yellow) is embedded in a\u00a0hydrophobic pocket<\/strong>\u00a0of the protein.<\/p>\n
It appears to be surrounded by\u00a0gray carbon atoms<\/strong>\u00a0from amino acid side chains, likely hydrophobic residues (e.g., leucine, valine, phenylalanine, or isoleucine).<\/p>\n
The\u00a0benzene ring of ibuprofen<\/strong>\u00a0seems to be in close contact with other\u00a0aromatic or aliphatic side chains<\/strong>, suggesting\u00a0\u03c0-\u03c0 stacking or van der Waals interactions<\/strong>.<\/p>\n
2. Hydrogen Bonds:<\/strong><\/p>\n
The\u00a0carboxyl (-COOH) group of ibuprofen<\/strong>\u00a0appears to be interacting with nearby\u00a0blue (nitrogen) or red (oxygen) atoms<\/strong>\u00a0from the protein.<\/p>\n
There is\u00a0at least one close contact between the oxygen of the carboxyl group and a protein residue, likely forming a hydrogen bond<\/strong>\u00a0with a backbone amide or side chain hydroxyl (e.g., Ser, Thr, Tyr, Asp, or Glu).<\/p>\n
3. Possible Electrostatic Interactions:<\/strong><\/p>\n
Ibuprofen is negatively charged at physiological pH (carboxylate anion, -COO\u207b<\/strong>).<\/p>\n
There may be\u00a0a salt bridge<\/strong>\u00a0or\u00a0ion-dipole interaction<\/strong>\u00a0if a\u00a0positively charged residue (e.g., Arg, Lys, or His) is nearby<\/strong>, but I cannot confirm the exact residue from the image alone.<\/p>\n
What I Cannot Determine from the Screenshot Alone:<\/strong><\/p>\n
Precise interacting residues:<\/strong>\u00a0No residue labels are visible.<\/p>\n
Exact interaction distances:<\/strong>\u00a0Hydrogen bonds and electrostatic interactions typically require precise atomic distances (~2\u20133 \u00c5).<\/p>\n
Interaction strengths:<\/strong>\u00a0Molecular modeling software would be needed to confirm interaction energies.<\/p>\n<\/blockquote>\n
The answer was quite good, especially considering that the picture is just a static 2D view and is quite crowded in atoms. In fact, it correctly pinpoints that the ibuprofen is shown in yellow, and highlights hydrophobic and aromatic contacts around its aromatic ring, both of which do happen (#1 in the figure below), and hydrogen bonds from its carboxylate group to nearby N and O atoms (#2 in the figure), adding that there might also possibly exist a salt bridge between that same carboxylate and some positively charged group of the protein that it can\u2019t distinguish clearly (#3).<\/p>\n
For your reference, I show these interactions in the next figure, numbered 1, 2 and 3 as they were mentioned:<\/p>\n
\"\"<\/figure>\n
\n
Other Vision-Capable Models? Peeking Into Google\u2019s Gemini 2.0 Flash<\/h2>\n
OpenAI is probably the most famous developer of LLM-based AI systems, but of course not the only one, and their models aren\u2019t even the best for some goals and according to some evaluations. However, they remain, to me at least, the easiest to use both directly in a GUI (ChatGPT) and programmatically (as we have seen here and I\u2019ve shown in many other posts).<\/p>\n
But my point here is that we can\u2019t overlook the other models. And just while preparing this post, I decided to give a try at Google\u2019s AI studio\u2019s Gemini 2.0 Flash, which like GPT-4o has picture, audio and document processing capabilities \u2014 and even video!<\/p>\n
Again I was amazed, and in fact it seemed to perform slightly better than GPT-4o and GPT-4o-mini tried throughout this post. I was in particular stunned at this specific example, where I loaded the screenshot of my app shown above, to which Gemini replied (without any prompt):<\/p>\n
\n
Here\u2019s what I can infer based on the image you provided:<\/p>\n
Prompt:<\/strong>\u00a0\u201cWhat do you think the artist\u201d<\/p>\n
Analysis:<\/strong><\/p>\n
The response provided at the bottom of the image states \u201cThe artist likely intended to convey themes of nature, beauty, and vibrancy through the striking imagery of the red poppies against a lush green background.\u201d<\/p>\n
Essentially, the AI is recognizing the subject matter and interpreting the artist\u2019s intent as showcasing the beauty and vibrancy of nature.<\/p>\n<\/blockquote>\n
That was kind of similar to what GPT-4o replied, probably because it just read what GPT-4o itself wrote. But what if I ask it a question trying to address the same point I asked GPT-4o?<\/p>\n
Here it goes:<\/p>\n
\n
do your own interpretation of the picture\u2026 what do you think did the author intend to communicate?<\/p>\n<\/blockquote>\n
And this was Gemini\u2019s reply, now perfectly reflecting what I think the author intended to convey:<\/p>\n
\n
Okay, based on the image, here\u2019s my interpretation of what the author might have intended:<\/p>\n
Beyond the surface-level beauty, the image hints at a more complex exploration of color harmony and contrast within nature.<\/p>\n
The juxtaposition of the natural poppy field with the color wheel suggests a deliberate artistic statement.<\/strong>\u00a0The artist may be inviting us to analyze the colors present in the natural scene (dominated by greens and reds) and to understand how these colors interact and create a specific visual experience.<\/p>\n
The placement of the color wheel with the arrows pointing to the green and red suggest complementary colours.<\/strong><\/p>\n
Perhaps the author wants us to appreciate the inherent color palettes nature provides.<\/strong><\/p>\n
In short, while the surface reading is about beauty, the integration of the color wheel suggests a deeper interest in the science and theory of color as applied to the natural world. The author seems to be inviting a more analytical and informed way of appreciating the beauty we see.<\/p>\n<\/blockquote>\n
The key part of Gemini\u2019s reply, which GPT-4o didn\u2019t get in any of my tests, is this:<\/p>\n
\u201cThe placement of the color wheel with the arrows pointing to the green and red suggest complementary colours.\u201d<\/p>\n
That was exactly my point as an artist!<\/p>\n
If this post has good reception, I will test Gemini 2.0\u2019s various capabilities in more detail. So let me know in the comments what you think!<\/p>\n
Conclusions<\/h2>\n
I guess after having read all these examples, you now also admire the potential of multimodal, vision-capable AI systems. The tests I presented with the GPT-4o models (and the glimpse into Gemini 2.0) proved the ability of the most modern AI systems to interpret quite complex images, being capable of understanding and extracting information from photographs, scientific figures of various kinds, plots and charts, even when containing information as complex as molecular structures. And from the examples we can advance applications as varied as assisting data analysis, assisting driving, and controlling robots \u2014 all of them provided some problems are overcome.<\/p>\n
This, because some images and questions are quite challenging, of course. In some cases, for example when limited by the resolution of labels in dense plots, or when limited by the lack of 3D perspective in the flat screenshots of molecular structures, or when having to estimate rotation angles for the robotic arm, there\u2019s probably not much to do, and the model remains as limited as even the most expert humans in the subject would.<\/p>\n
Yes, it is overall clear that with proper prompting and continued advancements in AI, these models could become invaluable tools for accelerating data interpretation and decision-making, reducing the load of human experts who can dedicate to more complex problems while the software assists non-experts to interpret graphical outputs from software, and who knows maybe some day drive cars and control robots!<\/p>\n
\n
www.lucianoabriata.com<\/em><\/a>\u00a0I write about everything that lies in my broad sphere of interests: nature, science, technology, programming, etc.\u00a0<\/em>Subscribe to get my new stories<\/em><\/a>\u00a0by email. To consult about small jobs check my\u00a0<\/em>services page here<\/em><\/a>. You can\u00a0<\/em>contact me here<\/em><\/a>. You can\u00a0<\/em>tip me here<\/em><\/a>.<\/em><\/p>\n
<\/a><\/p>\n
The post Testing the Power of Multimodal AI Systems in Reading and Interpreting Photographs, Maps, Charts and\u00a0More<\/a> appeared first on Towards Data Science<\/a>.<\/p>\n<\/div>\n
 \t

\n 
<\/BR>
\n    Luciano Abriata
\n \t

\n
<\/BR>
\nGo to original source<\/a>
\n \t

\n 
<\/BR><\/p>\n","protected":false},"excerpt":{"rendered":"
Testing the Power of Multimodal AI Systems in Reading and Interpreting Photographs, Maps, Charts and\u00a0More Introduction It\u2019s no news that artificial intelligence has made huge strides in recent years, particularly with the advent of multimodal models that can process and create both text and images, and some very new ones that also process and produce […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2117,62,221,67,71,70,2119],"tags":[1163,73,1907],"class_list":["post-2636","post","type-post","status-publish","format-standard","hentry","category-ai-systems","category-aimldsaimlds","category-computer-vision","category-deep-dives","category-large-language-models","category-machine-learning","category-multimodal-learning","tag-gpt","tag-models","tag-o"],"_links":{"self":[{"href":"https:\/\/mailitics.com\/index.php\/wp-json\/wp\/v2\/posts\/2636"}],"collection":[{"href":"https:\/\/mailitics.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mailitics.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mailitics.com\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/mailitics.com\/index.php\/wp-json\/wp\/v2\/comments?post=2636"}],"version-history":[{"count":0,"href":"https:\/\/mailitics.com\/index.php\/wp-json\/wp\/v2\/posts\/2636\/revisions"}],"wp:attachment":[{"href":"https:\/\/mailitics.com\/index.php\/wp-json\/wp\/v2\/media?parent=2636"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mailitics.com\/index.php\/wp-json\/wp\/v2\/categories?post=2636"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mailitics.com\/index.php\/wp-json\/wp\/v2\/tags?post=2636"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}