Because thy neighbour matters<\/h4>\nImage taken from unsplash.com<\/figcaption><\/figure>\n
First, let\u2019s talk about where ranking comes into play. Ranking is a big deal in e-commerce and search applications\u200a\u2014\u200aessentially, any scenario where you need to organize documents based on a query. It\u2019s a little different from classic classification or regression problems. For instance, in the Titanic dataset, you predict whether a passenger survives or not, and in house price prediction, you estimate the price of a house. But with ranking, the game changes. Instead of predicting a single value or category, you\u2019re trying to order documents based on relevance.<\/p>\n
Take an example: You search for \u201csaree\u201d on an e-commerce website like Amazon. You don\u2019t just want a random list of sarees; you want the most relevant ones to appear at the top, right? That\u2019s where Learning to Rank (LTR) steps in\u200a\u2014\u200ait ranks documents (or products) based on how well they match your\u00a0query.<\/p>\n
Now that we know where ranking fits in, let\u2019s dive into the nitty-gritty of different approaches and\u00a0methods.<\/p>\n
There are three main methods for Learning to Rank\u00a0(LTR):<\/p>\n
\n
Pointwise<\/strong><\/li>\n
Pairwise<\/strong><\/li>\n
Listwise<\/strong><\/li>\n<\/ol>\n
To make things easier to follow, let\u2019s establish some notation that we\u2019ll use to explain these\u00a0methods.<\/p>\n
We\u2019ll work with a set of queries q1,q2,\u2026,qn <\/em><\/strong>and each query has a corresponding set of documents d1,d2,d3,\u2026,dm<\/em><\/strong>\u200b. For\u00a0example:<\/p>\n
\n
Query q1<\/em><\/strong> is associated with documents d1<\/em><\/strong>,d2<\/em><\/strong>,d3<\/em><\/strong>\n<\/li>\n
Query q2<\/em><\/strong> associated with documents d4<\/em><\/strong>,d5<\/em><\/strong>.<\/li>\n<\/ul>\n
With this setup in mind, let\u2019s break down each method and how they approach the ranking\u00a0problem.<\/p>\n
Pointwise<\/h3>\n
In the pointwise approach<\/strong>, we treat the ranking problem as a simple classification task. For each query-document pair, we assign a target label that indicates the relevance of the document to the query. For\u00a0example:<\/p>\n
\n
Label 1 if the document is relevant.<\/li>\n
Label 0 if the document is not relevant.<\/li>\n<\/ul>\n
Using our earlier example, the data would look like\u00a0this:<\/p>\n
We train the model using this labeled data, leveraging features from both the queries and the documents to predict the label. After training, the model predicts the relevance of each document to a given query as a probability (ranging from 0 to 1). This probability can be interpreted as the relevance score.<\/p>\n
For example, after training, the model might produce the following scores:<\/p>\n
Using these scores, we re-rank the documents in descending order of relevance: d1,d3,d2<\/em><\/strong>. This new ranking order is then presented to the user, ensuring the most relevant documents appear at the\u00a0top.<\/p>\n
Pairwise<\/strong><\/h3>\n
The main drawback of the pointwise approach<\/strong> is that it misses the context<\/strong> in which the user interacts with a document. When a user clicks on or finds a document relevant, there are often multiple factors at play\u200a\u2014\u200aone of the most important being the neighboring items<\/strong>.<\/p>\n
For instance, if a user clicks on a document, it might not necessarily mean that the document is highly relevant. It could simply be that the other documents presented were of poor quality. Similarly, if you had shown a different set of documents for the same query, the user\u2019s interaction might have been entirely different.<\/p>\n
Imagine presenting d4<\/em><\/strong>\u200b for query q1<\/em><\/strong>\u200b. If d4\u200b<\/em><\/strong> is more relevant than d1<\/em><\/strong>\u200b, the user might have clicked on d4\u200b<\/em><\/strong> instead. This context\u200a\u2014\u200ahow documents compare to each other is completely overlooked in the pointwise approach.<\/p>\n
To capture this relative relevance<\/strong>, we turn to the pairwise approach<\/strong>.<\/p>\n
In the pairwise method, instead of looking at query-document pairs in isolation, we focus on pairs of documents<\/strong> for the same query and try to predict which one is more relevant. This helps incorporate the context of comparison between documents.<\/p>\n
We\u2019ll generate the data similarly for now, but the way we use it will be slightly more complex. Let\u2019s break that down\u00a0next.<\/p>\n
Imagine the training data for the pairwise approach<\/strong> structured as\u00a0follows:<\/p>\n
\n
\nq1,(d1,d2)<\/em><\/strong>\u2192label: 1(indicating d1<\/em><\/strong>\u200b is more relevant than\u00a0d2<\/em><\/strong>\u200b)<\/li>\n
\nq1,(d2,d3)<\/em><\/strong>\u2192label: 0 (indicating d2<\/em><\/strong>\u200b is less relevant than\u00a0d3<\/em><\/strong>\u200b)<\/li>\n
\nq1,(d1,d3)<\/em><\/strong>\u2192label: 1 (indicating d1<\/em><\/strong> is more relevant than\u00a0d3\u200b<\/em><\/strong>)<\/li>\n
\nq2,(d4,d5)<\/em><\/strong>\u2192label: 0(indicating d4<\/em><\/strong> is less relevant than\u00a0d5<\/em><\/strong>\u200b)<\/li>\n<\/ul>\n
Here, we assign the labels based on user interactions. For instance, d1<\/em><\/strong>\u200b and d3<\/em><\/strong>\u200b both being clicked indicates they are relevant, so we maintain their order for simplicity in this explanation.<\/p>\n
Model Training\u00a0Process:<\/h4>\n
Although the training data is in pairs, the model doesn\u2019t directly process these pairs. Instead, we treat it similarly to a classification problem, where each query-document pair<\/strong> is passed to the model separately.<\/p>\n
The model generates scores s1,s2,s3<\/em><\/strong>\u200b for the documents. These scores are used to compare the relevance of document\u00a0pairs.<\/p>\n
Penalizing the\u00a0Model:<\/strong><\/p>\n
If the model predicts scores that violate the true order of relevance, it is penalized. For\u00a0example:<\/p>\n
\n
If s1<s2<\/em><\/strong>, but the training data indicates d1>d2<\/em><\/strong>\u200b, the model is penalized because it failed to rank d1<\/em><\/strong>\u200b higher than\u00a0d2<\/em><\/strong>\u200b.<\/li>\n
If s2<s3<\/em><\/strong>\u200b, and the training data indicates d2<d3<\/em><\/strong>\u200b, the model did the right thing, so no penalty is\u00a0applied.<\/li>\n<\/ul>\n
This pairwise comparison helps the model learn the relative order of documents for a query, rather than just predicting a standalone relevance score like in the pointwise approach.<\/strong><\/p>\n
Challenges:<\/strong><\/p>\n
One of the main challenges of implementing pairwise models is the computational complexity<\/strong>\u200a\u2014\u200asince we need to compare all possible pairs of documents, the process scales as O(n\u00b2). Additionally, pairwise methods don\u2019t consider the global ranking<\/strong> of documents; they focus only on individual pairs during comparisons, which can lead to inconsistencies in the overall\u00a0ranking.<\/p>\n
Listwise<\/h3>\n
In listwise ranking, the goal is to optimize the entire list of documents based on their relevance to a query. Instead of treating individual documents separately, the focus is on the order in which they appear in the\u00a0list.<\/p>\n
Here\u2019s a breakdown of how this works in ListNet and LambdaRank:<\/p>\n
NDCG (Normalized Discounted Cumulative Gain)<\/strong>: I\u2019ll dive deeper into NDCG in another blog, but for now, think of it as a way to measure how well the ordering of items matches their relevance. It rewards relevant items appearing at the top of the list and normalizes the score for easier comparison.<\/p>\n
In listwise ranking, if you have a list of documents (d1, d2, d3), the model considers all possible permutations of these documents:<\/p>\n
\n
(d1, d2,\u00a0d3)<\/em><\/strong><\/li>\n
(d1, d3,\u00a0d2)<\/em><\/strong><\/li>\n
(d2, d1,\u00a0d3)<\/em><\/strong><\/li>\n
(d2, d3,\u00a0d1)<\/em><\/strong><\/li>\n
(d3, d1,\u00a0d2)<\/em><\/strong><\/li>\n
(d3, d2,\u00a0d1)<\/em><\/strong><\/li>\n<\/ul>\n
Training Process:<\/strong><\/p>\n\n
\nScore Prediction<\/strong>: The model predicts a score for each document in the list, and the documents are ranked according to these scores.For example: s1 = f(q1,d1), s2 =\u00a0f(q1,d2)<\/em><\/strong>\n<\/li>\n
\nIdeal Ranking<\/strong>: The ideal ranking is calculated by sorting the documents based on their true relevance<\/strong>. For example, d1<\/em><\/strong> might be the most relevant, followed by d2<\/em><\/strong>, and then\u00a0d3.<\/em><\/strong>\n<\/li>\n
\nNDCG Calculation<\/strong>: NDCG is calculated for each permutation of the document list. It checks how close the predicted ranking is to the ideal ranking, considering both relevance and the positions of the documents.<\/li>\n
\nPenalizing Incorrect Rankings<\/strong>: If the predicted ranking differs from the ideal, the NDCG score will drop. For example, if the ideal ranking is (d1, d3, d2)<\/em><\/strong> but the model ranks (d2, d1, d3)<\/em><\/strong>, the NDCG score will be lower because the most relevant document (d1<\/em><\/strong>) isn\u2019t ranked at the\u00a0top.<\/li>\n
\nGradient Calculation<\/strong>: The model calculates gradients based on how much the NDCG score would change if the order of documents was adjusted. These gradients guide the model on how to improve its predictions.<\/li>\n<\/ol>\n
This process helps the model learn to optimize the entire ranking list, improving the relevance of documents presented to\u00a0users.<\/p>\n
Summary<\/strong><\/h3>\n
When it comes to Learning to Rank, there\u2019s no one-size-fits-all approach. Pointwise models are super easy to set up and update, but they don\u2019t always take into account how documents relate to each other. That said, if you need something simple and fast, they\u2019re a great\u00a0option.<\/p>\n
On the other hand, pairwise<\/em><\/strong> and listwise<\/em><\/strong> methods are more powerful because they look at how documents compare to one another. But with that power comes more complexity<\/strong> \ud83d\ude1b, and listwise can be a real challenge because of its high complexity in training.<\/p>\n
Personally, I find the pairwise<\/em><\/strong> approach to be the sweet spot. It strikes a good balance between complexity and performance, making it ideal for many situations.<\/p>\n
At the end of the day, the method you choose really depends on your situation. How big and complicated is your dataset? Knowing the pros and cons of each method will help you pick the one that works best for what you\u2019re trying to\u00a0do.<\/p>\n
That\u2019s a wrap for today! Stay tuned for the next part, Until then happy ranking!\u00a0\ud83d\ude0a<\/p>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AogM3hQ5j8lYdU2Gb8xA3nw.jpeg?ssl=1\")
![\"\"](\"https:\/\/medium.com\/_\/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd5318f86e1b\")
<\/p>\n