An open-source project to explore the capabilities and limitations of LLMs on coding challenges<\/h4>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AesIxuKdYQRkHlT1COUgv8g.jpeg?ssl=1\")
Image by author (created with Flux 1.1\u00a0Pro)<\/figcaption><\/figure>\nWhat is this\u00a0about?<\/h3>\n
If 2024 taught us anything in the realm of Generative AI, then it is that coding is one of the most promising applications for large language models\u00a0(LLMs).<\/p>\n
In this blog post, I will describe how I am using one of the most advanced LLMs, Gemini Experimental 1121, which currently leads the LMArena Leaderboard, to tackle the Advent of Code<\/a> challenge.<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2Ad_Hk8HfGxfgnbRgX6AONsg.png?ssl=1\")
Image by\u00a0author<\/figcaption><\/figure>\nI will outline my approach and share my open-source code repository<\/a> so that readers can explore it further and replicate the\u00a0results.<\/p>\nWhy should we\u00a0care?<\/h3>\n
There are many reasons why LLMs + Coding is an exciting area, to highlight a\u00a0few:<\/p>\n
\n- Code is just like language and can be learned the same way by transformer models<\/li>\n
- The output is easily validated\u200a\u2014\u200awe can just run the code and check if it\u00a0does<\/li>\n
- There is a huge demand for code assistant\u200a\u2014\u200athey can easily increase productivity of coders\u00a0manyfold<\/li>\n<\/ul>\n
So, this is definitely an interesting and exciting direction and I thought it might be fun to explore it a bit more with a hands-on-challenge.<\/p>\n
The Advent of Code challenge<\/h3>\n
For those not familiar with the Advent of Code challenge: It is an annual event that runs from December 1st to December 25th, offering daily programming puzzles similar to an advent calendar. Each day, a new two-part puzzle is released where coders can test their coding and problem-solving skills. It\u2019s a fun way for developers of all levels to practice\u00a0coding.<\/p>\n
Both parts of the daily challenge revolve around a similar problem and use the same input data. The idea is to write a Python program that will process the input data and produce a solution (typically a number). Once we run the code and the model calculated the solution, we can take it and paste it into the website, which then will tell us if the solution was correct. If so, the second part will be unlocked with the a similar procedure.<\/p>\n
The competition runs for 25 days and allows users to collect a maximum of 50 stars (2 per\u00a0day).<\/p>\n
A great challenge for\u00a0LLMs<\/h3>\n
As mentioned above, this is a great challenge for LLMs. We can just take the problem statement and plug it into an LLM of our choice, let it produce the code, run the code, and take the solution that was produced and paste it into the website to see if the LLM was successful.<\/p>\n
For this project I\u2019m using Gemini Experimental 1121, which has greatly improved coding and reasoning capabilities. It is available through Google\u2019s AI Studio<\/a>. I use the same system prompt throughout the challenge\u200a\u2014\u200ait is a zero-shot prompt (no chain-of-thought) with the addition that the code should expect the input via input redirection, like\u00a0so:<\/p>\npython day01\/part1.py < day01\/input.txt<\/pre>\nThe system prompt\u00a0is:<\/p>\n
Provide python code to solve a given puzzle.
Assume there is an input.txt file that can be read
via input redirection in the command line.<\/pre>\nI then post the actual challenge and Gemini will create the code that should produce the correct solution. I copy the code into the GH repo and run it and paste the produced solution into the Advent of Code website to see if it was\u00a0correct.<\/p>\n
The repository<\/h3>\n
Each day\u2019s challenge is organized in its own directory:<\/p>\n
dayXX\/
\u251c\u2500\u2500 input.txt # Challenge input
\u251c\u2500\u2500 part1-problem.txt # Problem description for part 1
\u251c\u2500\u2500 part2-problem.txt # Problem description for part 2
\u251c\u2500\u2500 part1.py # Solution for part 1
\u2514\u2500\u2500 part2.py # Solution for part 2<\/pre>\nThe part1 and part2-problem text files contain the problems of the challenge as stated by Advent of Code. I also appended the correct solution to the end of each text\u00a0file:<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2Adc4FjBgbhXyC9-EnELbcWA.png?ssl=1\")
Image by\u00a0author<\/figcaption><\/figure>\nThe python scripts contain teh code as produced by Gemini. To be fully transparent I also link to the actual conversations so that everyone can see and review the\u00a0steps:<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AdZfQ7rrkVYH-Ql-RyE-fjw.png?ssl=1\")
Image by\u00a0author<\/figcaption><\/figure>\nTo see an example of one of these chats, head over to my chat with Gemini about the day 1 challenge.<\/a><\/p>\nI will record all the results in a table that will give the readers a good first overview how the model has fared so\u00a0far:<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AivivI6IOJTmjEQHTAj43Rg.png?ssl=1\")
Image by\u00a0author<\/figcaption><\/figure>\nExample<\/h3>\n
To get a better idea what this will look like, let\u2019s have a look at part 1 the day 1 challenge. Here is the problem statement:<\/p>\n
The Chief Historian is always present for the big Christmas sleigh launch, but nobody has seen him in months! Last anyone heard, he was visiting locations that are historically significant to the North Pole; a group of Senior Historians has asked you to accompany them as they check the places they think he was most likely to visit.
As each location is checked, they will mark it on their list with a star. They figure the Chief Historian must be in one of the first fifty places they'll look, so in order to save Christmas, you need to help them get fifty stars on their list before Santa takes off on December 25th.
Collect stars by solving puzzles. Two puzzles will be made available on each day in the Advent calendar; the second puzzle is unlocked when you complete the first. Each puzzle grants one star. Good luck!
You haven't even left yet and the group of Elvish Senior Historians has already hit a problem: their list of locations to check is currently empty. Eventually, someone decides that the best place to check first would be the Chief Historian's office.
Upon pouring into the office, everyone confirms that the Chief Historian is indeed nowhere to be found. Instead, the Elves discover an assortment of notes and lists of historically significant locations! This seems to be the planning the Chief Historian was doing before he left. Perhaps these notes can be used to determine which locations to search?
Throughout the Chief's office, the historically significant locations are listed not by name but by a unique number called the location ID. To make sure they don't miss anything, The Historians split into two groups, each searching the office and trying to create their own complete list of location IDs.
There's just one problem: by holding the two lists up side by side (your puzzle input), it quickly becomes clear that the lists aren't very similar. Maybe you can help The Historians reconcile their lists?
For example:
3 4
4 3
2 5
1 3
3 9
3 3
Maybe the lists are only off by a small amount! To find out, pair up the numbers and measure how far apart they are. Pair up the smallest number in the left list with the smallest number in the right list, then the second-smallest left number with the second-smallest right number, and so on.
Within each pair, figure out how far apart the two numbers are; you'll need to add up all of those distances. For example, if you pair up a 3 from the left list with a 7 from the right list, the distance apart is 4; if you pair up a 9 with a 3, the distance apart is 6.
In the example list above, the pairs and distances would be as follows:
The smallest number in the left list is 1, and the smallest number in the right list is 3. The distance between them is 2.
The second-smallest number in the left list is 2, and the second-smallest number in the right list is another 3. The distance between them is 1.
The third-smallest number in both lists is 3, so the distance between them is 0.
The next numbers to pair up are 3 and 4, a distance of 1.
The fifth-smallest numbers in each list are 3 and 5, a distance of 2.
Finally, the largest number in the left list is 4, while the largest number in the right list is 9; these are a distance 5 apart.
To find the total distance between the left list and the right list, add up the distances between all of the pairs you found. In the example above, this is 2 + 1 + 0 + 1 + 2 + 5, a total distance of 11!
Your actual left and right lists contain many location IDs. What is the total distance between your lists?
### Solution: 1666427<\/pre>\nWhen giving this challenge to Gemini it produces this\u00a0code:<\/p>\n
import sys
def solve():
left_list = []
right_list = []
for line in sys.stdin:
left, right = map(int, line.strip().split())
left_list.append(left)
right_list.append(right)
left_list.sort()
right_list.sort()
total_distance = 0
for i in range(len(left_list)):
total_distance += abs(left_list[i] - right_list[i])
print(total_distance)
solve()<\/pre>\nRunning this code\u00a0with<\/p>\n
python day01\/part1.py < day01\/input.txt<\/pre>\ngives the\u00a0result:<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2ARLQJPBDdzocVUt2uhvJzKg.png?ssl=1\")
Image by\u00a0author<\/figcaption><\/figure>\nAs we can see, Gemini\u2019s code produced the correct\u00a0result.<\/p>\n
Bonus: Get an explanation<\/h4>\n
As a bonus, we can also ask Gemini to provide an explanation what the code does and why it solves the challenge:<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AHVyqkwkG73bDq047MTSVXg.png?ssl=1\")
Image by\u00a0author<\/figcaption><\/figure>\nYou can find the chat with the explanation here<\/a>.<\/p>\nConclusion<\/h3>\n
With this project I want to explore how capable state-of-the-art LLMs are currently at solving coding challenges. I have the hypothesis that Gemini (and other LLMs) have gotten good enough to solve most of these challenges. This does, of course, not mean that they are fit (yet) to solve real software challenges that are much more\u00a0complex.<\/p>\n
That being said, I was just curious about this and decided to hop onto this fun little project. I hope you enjoy it and it gives you some insight into where we are headed with LLMs + Coding\u00a0\ud83e\udd17<\/p>\n
Heiko Hotz<\/h3>\n
\ud83d\udc4b Follow me on Medium<\/a> and LinkedIn<\/a> to read more about Generative AI, Machine Learning, and Natural Language Processing.<\/p>\n\ud83d\udc65 If you\u2019re based in London join one of our NLP London\u00a0Meetups<\/a>.<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/750\/0%2ACq-ZbBArQ4czUbnB.png?ssl=1\")
Image by\u00a0author<\/figcaption><\/figure>\n![\"\"](\"https:\/\/medium.com\/_\/stat?event=post.clientViewed&referrerSource=full_rss&postId=dfd88ffa12a6\")
<\/p>\n
\nGoogle Gemini Is Entering the Advent of Code Challenge<\/a> was originally published in Towards Data Science<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>\n<\/div>\n
What is this\u00a0about?<\/h3>\n
If 2024 taught us anything in the realm of Generative AI, then it is that coding is one of the most promising applications for large language models\u00a0(LLMs).<\/p>\n
In this blog post, I will describe how I am using one of the most advanced LLMs, Gemini Experimental 1121, which currently leads the LMArena Leaderboard, to tackle the Advent of Code<\/a> challenge.<\/p>\n I will outline my approach and share my open-source code repository<\/a> so that readers can explore it further and replicate the\u00a0results.<\/p>\n There are many reasons why LLMs + Coding is an exciting area, to highlight a\u00a0few:<\/p>\n So, this is definitely an interesting and exciting direction and I thought it might be fun to explore it a bit more with a hands-on-challenge.<\/p>\n For those not familiar with the Advent of Code challenge: It is an annual event that runs from December 1st to December 25th, offering daily programming puzzles similar to an advent calendar. Each day, a new two-part puzzle is released where coders can test their coding and problem-solving skills. It\u2019s a fun way for developers of all levels to practice\u00a0coding.<\/p>\n Both parts of the daily challenge revolve around a similar problem and use the same input data. The idea is to write a Python program that will process the input data and produce a solution (typically a number). Once we run the code and the model calculated the solution, we can take it and paste it into the website, which then will tell us if the solution was correct. If so, the second part will be unlocked with the a similar procedure.<\/p>\n The competition runs for 25 days and allows users to collect a maximum of 50 stars (2 per\u00a0day).<\/p>\n As mentioned above, this is a great challenge for LLMs. We can just take the problem statement and plug it into an LLM of our choice, let it produce the code, run the code, and take the solution that was produced and paste it into the website to see if the LLM was successful.<\/p>\n For this project I\u2019m using Gemini Experimental 1121, which has greatly improved coding and reasoning capabilities. It is available through Google\u2019s AI Studio<\/a>. I use the same system prompt throughout the challenge\u200a\u2014\u200ait is a zero-shot prompt (no chain-of-thought) with the addition that the code should expect the input via input redirection, like\u00a0so:<\/p>\n The system prompt\u00a0is:<\/p>\n I then post the actual challenge and Gemini will create the code that should produce the correct solution. I copy the code into the GH repo and run it and paste the produced solution into the Advent of Code website to see if it was\u00a0correct.<\/p>\n Each day\u2019s challenge is organized in its own directory:<\/p>\n The part1 and part2-problem text files contain the problems of the challenge as stated by Advent of Code. I also appended the correct solution to the end of each text\u00a0file:<\/p>\n The python scripts contain teh code as produced by Gemini. To be fully transparent I also link to the actual conversations so that everyone can see and review the\u00a0steps:<\/p>\n To see an example of one of these chats, head over to my chat with Gemini about the day 1 challenge.<\/a><\/p>\n I will record all the results in a table that will give the readers a good first overview how the model has fared so\u00a0far:<\/p>\n To get a better idea what this will look like, let\u2019s have a look at part 1 the day 1 challenge. Here is the problem statement:<\/p>\n When giving this challenge to Gemini it produces this\u00a0code:<\/p>\n Running this code\u00a0with<\/p>\n gives the\u00a0result:<\/p>\n As we can see, Gemini\u2019s code produced the correct\u00a0result.<\/p>\n As a bonus, we can also ask Gemini to provide an explanation what the code does and why it solves the challenge:<\/p>\n You can find the chat with the explanation here<\/a>.<\/p>\n With this project I want to explore how capable state-of-the-art LLMs are currently at solving coding challenges. I have the hypothesis that Gemini (and other LLMs) have gotten good enough to solve most of these challenges. This does, of course, not mean that they are fit (yet) to solve real software challenges that are much more\u00a0complex.<\/p>\n That being said, I was just curious about this and decided to hop onto this fun little project. I hope you enjoy it and it gives you some insight into where we are headed with LLMs + Coding\u00a0\ud83e\udd17<\/p>\n \ud83d\udc4b Follow me on Medium<\/a> and LinkedIn<\/a> to read more about Generative AI, Machine Learning, and Natural Language Processing.<\/p>\n \ud83d\udc65 If you\u2019re based in London join one of our NLP London\u00a0Meetups<\/a>.<\/p>\n Google Gemini Is Entering the Advent of Code Challenge<\/a> was originally published in Towards Data Science<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>\n<\/div>\nWhy should we\u00a0care?<\/h3>\n
\n
The Advent of Code challenge<\/h3>\n
A great challenge for\u00a0LLMs<\/h3>\n
python day01\/part1.py < day01\/input.txt<\/pre>\n
Provide python code to solve a given puzzle.
Assume there is an input.txt file that can be read
via input redirection in the command line.<\/pre>\nThe repository<\/h3>\n
dayXX\/
\u251c\u2500\u2500 input.txt # Challenge input
\u251c\u2500\u2500 part1-problem.txt # Problem description for part 1
\u251c\u2500\u2500 part2-problem.txt # Problem description for part 2
\u251c\u2500\u2500 part1.py # Solution for part 1
\u2514\u2500\u2500 part2.py # Solution for part 2<\/pre>\nExample<\/h3>\n
The Chief Historian is always present for the big Christmas sleigh launch, but nobody has seen him in months! Last anyone heard, he was visiting locations that are historically significant to the North Pole; a group of Senior Historians has asked you to accompany them as they check the places they think he was most likely to visit.
As each location is checked, they will mark it on their list with a star. They figure the Chief Historian must be in one of the first fifty places they'll look, so in order to save Christmas, you need to help them get fifty stars on their list before Santa takes off on December 25th.
Collect stars by solving puzzles. Two puzzles will be made available on each day in the Advent calendar; the second puzzle is unlocked when you complete the first. Each puzzle grants one star. Good luck!
You haven't even left yet and the group of Elvish Senior Historians has already hit a problem: their list of locations to check is currently empty. Eventually, someone decides that the best place to check first would be the Chief Historian's office.
Upon pouring into the office, everyone confirms that the Chief Historian is indeed nowhere to be found. Instead, the Elves discover an assortment of notes and lists of historically significant locations! This seems to be the planning the Chief Historian was doing before he left. Perhaps these notes can be used to determine which locations to search?
Throughout the Chief's office, the historically significant locations are listed not by name but by a unique number called the location ID. To make sure they don't miss anything, The Historians split into two groups, each searching the office and trying to create their own complete list of location IDs.
There's just one problem: by holding the two lists up side by side (your puzzle input), it quickly becomes clear that the lists aren't very similar. Maybe you can help The Historians reconcile their lists?
For example:
3 4
4 3
2 5
1 3
3 9
3 3
Maybe the lists are only off by a small amount! To find out, pair up the numbers and measure how far apart they are. Pair up the smallest number in the left list with the smallest number in the right list, then the second-smallest left number with the second-smallest right number, and so on.
Within each pair, figure out how far apart the two numbers are; you'll need to add up all of those distances. For example, if you pair up a 3 from the left list with a 7 from the right list, the distance apart is 4; if you pair up a 9 with a 3, the distance apart is 6.
In the example list above, the pairs and distances would be as follows:
The smallest number in the left list is 1, and the smallest number in the right list is 3. The distance between them is 2.
The second-smallest number in the left list is 2, and the second-smallest number in the right list is another 3. The distance between them is 1.
The third-smallest number in both lists is 3, so the distance between them is 0.
The next numbers to pair up are 3 and 4, a distance of 1.
The fifth-smallest numbers in each list are 3 and 5, a distance of 2.
Finally, the largest number in the left list is 4, while the largest number in the right list is 9; these are a distance 5 apart.
To find the total distance between the left list and the right list, add up the distances between all of the pairs you found. In the example above, this is 2 + 1 + 0 + 1 + 2 + 5, a total distance of 11!
Your actual left and right lists contain many location IDs. What is the total distance between your lists?
### Solution: 1666427<\/pre>\nimport sys
def solve():
left_list = []
right_list = []
for line in sys.stdin:
left, right = map(int, line.strip().split())
left_list.append(left)
right_list.append(right)
left_list.sort()
right_list.sort()
total_distance = 0
for i in range(len(left_list)):
total_distance += abs(left_list[i] - right_list[i])
print(total_distance)
solve()<\/pre>\npython day01\/part1.py < day01\/input.txt<\/pre>\n
Bonus: Get an explanation<\/h4>\n
Conclusion<\/h3>\n
Heiko Hotz<\/h3>\n
<\/p>\n
\n