{"id":1352,"date":"2025-01-22T07:03:00","date_gmt":"2025-01-22T07:03:00","guid":{"rendered":"https:\/\/mailitics.com\/index.php\/2025\/01\/22\/lyrec-a-song-recommender-that-reads-between-the-lyrics-eca5ea2ae8c8\/"},"modified":"2025-01-22T07:03:00","modified_gmt":"2025-01-22T07:03:00","slug":"lyrec-a-song-recommender-that-reads-between-the-lyrics-eca5ea2ae8c8","status":"publish","type":"post","link":"https:\/\/mailitics.com\/index.php\/2025\/01\/22\/lyrec-a-song-recommender-that-reads-between-the-lyrics-eca5ea2ae8c8\/","title":{"rendered":"LyRec: A Song Recommender That Reads Between the Lyrics"},"content":{"rendered":"

LyRec: A Song Recommender That Reads Between the Lyrics
\n \t

\n
<\/BR>
\n
\n \t

\n
<\/BR><\/p>\n

\n

This is how I built an emotionally intelligent LLM-powered song recommendation system.<\/h4>\n
\"\"
Photo by David Pup\u0103z\u0103<\/a> on\u00a0Unsplash<\/a><\/figcaption><\/figure>\n

Do you remember the last time you found yourself obsessing over a song? Maybe it was the raw emotion that resonated with you, or perhaps it was the lyrics that kept you hooked. Or maybe you loved the story it tells. Wouldn\u2019t it be nice if there was a way to find songs that express a similar emotion, share similar lyrical elements, or paint similar\u00a0imagery?<\/p>\n

In this article, I\u2019ll show you how I built LyRec <\/em><\/strong>(see what I did there? \ud83d\ude09), a recommendation system for songs that lets you do this! Here\u2019s a little demo (it runs on my\u00a0Mac!).<\/p>\n

\"\"<\/figure>\n

Don\u2019t worry; you won\u2019t need to know the nitty-gritty of recommendation systems to understand this. Along the way, I\u2019ll explain my design choices, and hopefully, you\u2019ll learn a thing or two about semantic search and Retrieval-Augmented Generation (RAG). So let\u2019s get\u00a0started!<\/p>\n

Settings the Goals\u00a0\ud83c\udfaf<\/h3>\n

First, I needed to set some realistic<\/strong> goals for LyRec<\/em><\/strong>. Here\u2019s the list I finalized.<\/p>\n

1. Given a (seen or unseen) song lyrics, LyRec<\/strong> should be able to suggest similar songs from its database.
2. Given a free-form text input, LyRec <\/em><\/strong>should be able to find songs that match this description. The text input may describe an emotion, mood, story, specific elements, and so on.
3. Given both lyrics and description, LyRec <\/em><\/strong>should be able to find the best matching\u00a0songs.<\/p><\/blockquote>\n

The Approach\u00a0\ud83e\udd14<\/h3>\n

With the goals set, it was time to come up with a suitable approach for these tasks. I decided to go with an embedding-based<\/strong> semantic search<\/strong>. If you are unfamiliar with this concept, here\u2019s a quick overview.<\/p>\n

\nSemantic search<\/strong> is a technique where we search (given a query) a database by understanding the contextual meaning of words rather than relying solely on keyword matching. The increasingly popular way of doing this is with higher dimensional vector embeddings<\/strong>. First, using a language model, we compute a vector representation (embedding) for each entry in our database and store it alongside. When a new query comes, we generate its embedding using the same model and compute the similarity between the query embedding and all the embeddings stored in the database. Finally, we return the entry with the highest embedding similarity.<\/p><\/blockquote>\n

For LyRec<\/em><\/strong>, the entries would be song lyrics, and the query would be either new song lyrics or a user-written free-form text. So, we would essentially compute semantic similarity between lyrics embeddings and the query. Initially, this is where I\u00a0stopped.<\/p>\n

After giving it some more thought, I realized that while computing the similarity between two lyrics embeddings was fine, it was probably not ideal to compute the similarity between textual description and lyrics. So, I decided to generate another set of embeddings that would capture the description of the songs. But how do I get those song descriptions? Easy! I just asked another LLM to summarize each song in our database. Then, I used the previous model to generate embeddings from these summaries. So, when the query is a free-form text, LyRec<\/em><\/strong> would use these summary embeddings to compute semantic similarities.<\/p>\n

Okay, now LyRec<\/em><\/strong> has two ways to find similar songs. But what if we wanted to use both song lyrics and a description for a recommendation? Well, there could be many ways to combine lyrics and summary similarity scores. But I took a slightly different route of re-ranking<\/strong>.<\/p>\n

In RAGs, re-ranking<\/strong> is often done with a model (trained for predicting relevance scores), which is more accurate (but costlier) than embedding similarity. The initially retrieved documents (based on embedding similarity) and the query are passed to this re-ranker, which assigns a relevance score to each document. Then, based on this new score, the documents are reordered.<\/p><\/blockquote>\n

Taking inspiration from this, I came up with the following approach. First, LyRec<\/em><\/strong> will fetch the most similar songs based on the lyrics embedding, and then these songs will be re-ranked based on the summary embedding similarity scores. It\u2019s worth pointing out that I am not using a re-ranker model. Instead, I\u2019m just using a different embedding for the re-ranking step. You may ask, why not use summary embeddings in the first step and lyrics embeddings in the second? Well, to be honest, I don\u2019t have a good answer to that. I just preferred the outputs more (for a small set of queries) using this method. Maybe you can try the other\u00a0way!<\/p>\n

I hope the overall approach is now clear. Let\u2019s get into the implementation!<\/p>\n

The Implementation \ud83d\udc68\u200d\ud83d\udcbb<\/h3>\n

Dataset<\/h4>\n

Of course, the first thing I needed was a song lyrics dataset. Fortunately, I found one on Kaggle! This dataset is under a Creative Commons (CC0: Public Domain)\u00a0license.<\/p>\n

Spotify Million Song Dataset<\/a><\/p>\n

This dataset contains about 60K<\/em> <\/em><\/strong>song lyrics along with the title and artist name. I know 60K<\/em> might not cover all the songs you love, but I think it\u2019s a good starting point for\u00a0LyRec.<\/em><\/strong><\/p>\n

songs_df = pd.read_csv(f\"{root_dir}\/spotify_millsongdata.csv\")
songs_df = songs_df.drop(columns=[\"link\"])
songs_df[\"song_id\"] = songs_df.index + 1<\/pre>\n
I didn\u2019t need to perform any pre-processing on this data. I just removed the link<\/em> column and added an ID<\/em> for each\u00a0song.<\/p>\n
Models<\/h4>\n
I needed to select two LLMs: One for computing the embeddings and another for generating the song summaries. Picking the correct LLM for your task may be a little tricky because of the sheer number of them! It\u2019s a good idea to look at the leaderboard to find the current best ones. For the embedding model, I checked the MTEB leaderboard hosted by HuggingFace.<\/p>\n
MTEB Leaderboard – a Hugging Face Space by mteb<\/a><\/p>\n
I was looking for a smaller model (obviously!) without compromising too much accuracy; hence, I decided on GTE-Qwen2-1.5B-Instruct<\/strong><\/a>.<\/p>\n
from sentence_transformers import SentenceTransformer
import torch

model = SentenceTransformer(
    \"Alibaba-NLP\/gte-Qwen2-1.5B-instruct\",
    model_kwargs={\"torch_dtype\": torch.float16}
)<\/pre>\n
For the summarizer, I just needed a small enough instruction following LLM, so I went with Gemma-2\u20132b-It<\/strong><\/a>. In my experience, it\u2019s one of the best small models as of\u00a0now.<\/p>\n
import torch
from transformers import pipeline

pipe = pipeline(
    \"text-generation\",
    model=\"google\/gemma-2-2b-it\",
    model_kwargs={\"torch_dtype\": torch.bfloat16},
    device=\"cuda\",
)<\/pre>\n
Pre-computing the Embeddings<\/h4>\n
Computing the lyrics embeddings was pretty straightforward. I just used the\u00a0.encode(\u2026)<\/em> method with a batch_size<\/em> of 32 for faster processing.<\/p>\n
song_lyrics = songs_df[\"text\"].values

lyrics_embeddings = model.encode(
    song_lyrics,
    batch_size=32,
    show_progress_bar=True
)

np.save(f\"{root_dir}\/60k_song_lyrics_embeddings.npy\", lyrics_embeddings)<\/pre>\n
At this point, I stored these embeddings in a\u00a0.npy<\/em> file. I could have used a more structured format, but it did the job for\u00a0me.<\/p>\n
Coming to the summary embeddings, I first needed to generate the summaries. I had to ensure that the summary captured the emotion and the song\u2019s theme while not being too lengthy. So, I came up with the following prompt for\u00a0Gemma-2.<\/p>\n
You are an expert song summarizer. 
You will be given the full lyrics to a song. 
Your task is to write a concise, cohesive summary that 
captures the central emotion, overarching theme, and 
narrative arc of the song in 150 words.

{song lyrics}<\/pre>\n
Here\u2019s the code snippet for summary generation. For simplicity, the following shows a sequential processing. I have included the batch-processing version in the GitHub\u00a0repo.<\/p>\n
def get_summary(song_lyrics):
    messages = [
        {\"role\": \"user\", 
         \"content\": f'''You are an expert song summarizer. 
You will be given the full lyrics to a song. 
Your task is to write a concise, cohesive summary that 
captures the central emotion, overarching theme, and 
narrative arc of the song in 150 words.nn{song_lyrics}'''},
    ]

    outputs = pipe(messages, max_new_tokens=256)
    assistant_response = outputs[0][\"generated_text\"][-1][\"content\"].strip()
    return assistant_response

songs_df[\"summary\"] = songs_df[\"text\"].progress_apply(get_description)<\/pre>\n
Unsurprisingly, this step took the most time. Luckily, this needs to be done only once, and of course, when we want to update the database with new\u00a0songs.<\/p>\n
Then, I computed and stored the embedding just like the last\u00a0time.<\/p>\n
song_summary = songs_df[\"summary\"].values

summary_embeddings = model.encode(
    song_summary,
    batch_size=32,
    show_progress_bar=True
)

np.save(f\"{root_dir}\/60k_song_summary_embeddings.npy\", summary_embeddings)<\/pre>\n
Vector Search<\/h4>\n
With the embeddings in place, it was time to implement the semantic search based on embedding similarity. There are a lot of awesome open-source vector databases available for this job. I decided to use a simple one called FAISS<\/strong> (Facebook AI Similarity Search). It just takes two lines to add the embeddings into the database. First, we create a FAISS index. Here, we need to mention the similarity metric you want to utilize for searching and the dimension of the vectors. I used the dot product<\/strong> (inner product) as the similarity measure. Then, we add the embeddings to the\u00a0index.<\/p>\n
Note: Our database is small enough to do an exhaustive search using dot product. For larger databases, it\u2019s recommended to perform an approximate nearest neighbor (ANN) search. FAISS has support for\u00a0that.<\/p><\/blockquote>\n
import faiss

lyrics_embeddings = np.load(f\"{root_dir}\/60k_song_lyrics_embeddings.npy\")
lyrics_index = faiss.IndexFlatIP(lyrics_embeddings.shape[1])
lyrics_index.add(lyrics_embeddings.astype(np.float32))

summary_embeddings = np.load(f\"{root_dir}\/60k_song_summary_embeddings.npy\")
summary_index = faiss.IndexFlatIP(summary_embeddings.shape[1])
summary_index.add(summary_embeddings.astype(np.float32))<\/pre>\n
To find the most similar songs given a query, we first need to generate the query embedding and then call the\u00a0.search(\u2026) <\/em>method on the index. Under the hood, this method computes the similarity between the query and every entry in our database and returns the top k<\/em> entries and the corresponding scores. Here\u2019s the code performing a semantic search on lyrics embeddings.<\/p>\n
query_lyrics = 'Imagine the last song you fell in love with'
query_embedding = model.encode(f'''Instruct: Given the lyrics, 
retrieve relevant songsnQuery: {query_lyrics}''')
query_embedding = query_embedding.reshape(1, -1).astype(np.float32)
lyrics_scores, lyrics_ids = lyrics_index.search(query_embedding, 10)<\/pre>\n
Notice that I added a simple prompt in the query. This is recommended for this model. The same applies to the summary embeddings.<\/p>\n
query_description = 'Describe the type of song you wanna listen to'
query_embedding = model.encode(f'''Given a description, 
retrieve relevant songsnQuery: {query_description}''')
query_embedding = query_embedding.reshape(1, -1).astype(np.float32)
summary_scores, summary_ids = summary_index.search(query_embedding, k)<\/pre>\n
Pro tip: How do you do a sanity check?
Just put any entry from the database in the query and see if the search returns the same as the top-scoring entry!<\/p><\/blockquote>\n
Implementing the\u00a0Features<\/h4>\n
At this stage, I had the building blocks of LyRec<\/em><\/strong>. Now, it was the time to put these together. Remember the three goals I set in the beginning? Here\u2019s how I implemented those.<\/p>\n
To keep things tidy, I created a class named LyRec <\/em><\/strong>that would have a method for each feature. The first two features are pretty straightforward to implement.<\/p>\n
The method\u00a0.get_songs_with_similar_lyrics(\u2026) <\/em>takes a song (lyrics) and a whole number k <\/em>as input and returns a list of k<\/em> most similar songs based on the lyrics similarity. Each element in the list is a dictionary containing the artist\u2019s name, song title, and\u00a0lyrics.<\/p>\n
Similarly,\u00a0.get_songs_with_similar_description(\u2026) <\/em>takes a free-form text and a whole number k <\/em>as input and returns a list of k<\/em> most similar songs based on the description.<\/p>\n
Here\u2019s the relevant code\u00a0snippet.<\/p>\n
class LyRec:
    def __init__(self, songs_df, lyrics_index, summary_index, embedding_model):
        self.songs_df = songs_df
        self.lyrics_index = lyrics_index
        self.summary_index = summary_index
        self.embedding_model = embedding_model

    def get_records_from_id(self, song_ids):
        songs = []
        for _id in song_ids:
            songs.extend(self.songs_df[self.songs_df[\"song_id\"]==_id+1].to_dict(orient='records'))
        return songs

    def get_songs_with_similar_lyrics(self, query_lyrics, k=10):
        query_embedding = self.embedding_model.encode(
            f\"Instruct: Given the lyrics, retrieve relevant songsn Query: {query_lyrics}\"
        ).reshape(1, -1).astype(np.float32)

        scores, song_ids = self.lyrics_index.search(query_embedding, k)
        return self.get_records_from_id(song_ids[0])

    def get_songs_with_similar_description(self, query_description, k=10):
        query_embedding = self.embedding_model.encode(
            f\"Instruct: Given a description, retrieve relevant songsn Query: {query_description}\"
        ).reshape(1, -1).astype(np.float32)

        scores, song_ids = self.summary_index.search(query_embedding, k)
        return self.get_records_from_id(song_ids[0])<\/pre>\n
The final feature was a little tricky to implement. Recall that we need to first retrieve the top songs based on lyrics and then re-rank them based on the textual description. The first retrieval was easy. For the second one, we only need to consider the top-scoring songs. I decided to create a temporary FAISS index with the top songs and then search for the songs with the highest summary similarity scores. Here\u2019s my implementation.<\/p>\n
def get_songs_with_similar_lyrics_and_description(self, query_lyrics, query_description, k=10):
    query_lyrics_embedding = self.embedding_model.encode(
        f\"Instruct: Given the lyrics, retrieve relevant songsn Query: {query_lyrics}\"
    ).reshape(1, -1).astype(np.float32)

    scores, song_ids = self.lyrics_index.search(query_lyrics_embedding, 500)
    top_k_indices = song_ids[0]

    summary_candidates = []
    for idx in top_k_indices:
        emb = self.summary_index.reconstruct(int(idx))
        summary_candidates.append(emb)
    summary_candidates = np.array(summary_candidates, dtype=np.float32)

    temp_index = faiss.IndexFlatIP(summary_candidates.shape[1])
    temp_index.add(summary_candidates)

    query_description_embedding = self.embedding_model.encode(
        f\"Instruct: Given a description, retrieve relevant songsn Query: {query_description}\"
    ).reshape(1, -1).astype(np.float32)

    scores, temp_ids = temp_index.search(query_description_embedding, k)
    final_song_ids = [top_k_indices[i] for i in temp_ids[0]]

    return self.get_records_from_id(final_song_ids)<\/pre>\n
Voila! Finally, LyRec<\/em><\/strong> is ready. You can find the complete implementation on this repo. Please leave a star if you find this helpful!\u00a0\ud83d\ude03<\/p>\n
GitHub – Suji04\/LyRec: LyRec: Recommending Songs from User Query using LLMs<\/a><\/p>\n
The Result\u00a0\ud83e\udd41<\/h3>\n
Using Lyrics<\/h4>\n
Now it\u2019s time to see LyRec<\/em><\/strong> in action. For the first example, I\u2019m taking Ed Sheeran\u2019s Perfect<\/em> \u2764\ufe0f. <\/em>Here are the top few songs suggested by LyRec<\/em><\/strong> solely based on the lyrics. If you listen to these songs (which I highly recommend you do), you\u2019ll find many similar elements to Perfect\u2019s<\/em> lyrics!<\/p>\n