Running experiments is a task that often falls to data scientists. If that\u2019s you, congrats! It can be a rewarding and high-impact area of work, but also requires tools found outside the typical ML-heavy data science curriculum.<\/p>\n
Even with the best tools, only a small share of experiments deliver meaningful business value. I\u2019ve been lucky to design and execute many experiments. Of those, I have a few winners. From these, I\u2019m sharing some stories to illustrate key concepts related to experiments.<\/p>\n
- \n
- Multiple Comparisons & How Random Assignment Saved us $1M in Marketing Spend<\/a><\/li>\n
- Choosing what to test & How IntelyCare tested its referral bonus program into existence (coming soon)<\/li>\n<\/ul>\n
Background: <\/strong>I work at a company called IntelyCare. We help connect nurses with various work opportunities (full-time, part-time, contracts, per-diem\u2026 the whole menu).<\/p>\n
- \n
- One of our core offerings is a nursing-only job board<\/a>. If you take a look in the year 2025, you\u2019ll notice two possible ways of sorting jobs by date and by relevance.<\/li>\n<\/ul>\n
Why it matters: <\/strong>The sort-by-relevance feature is our current best lever to guarantee a good experience for paying customers. It also gives us an opportunity to improve the overall efficiency of our job board by steering eyeballs away from low-quality jobs.<\/p>\n
Unfortunately, we can\u2019t put every<\/em> job at the top of a search result. We face a tradeoff between the quantity<\/em> of top-page listings and the quality<\/em> of the experience in the form of increased applies.<\/p>\n
How it works: \u201c<\/strong>Relevance\u201d doesn\u2019t mean what it normally means. Sorry!<\/p>\n
We give each job a score between 0 and 100. When filling a page with jobs, sorting by relevance means we sort the results by that score. That\u2019s it! For brevity, we\u2019ll say any job with a score higher than 0 is \u201cboosted.\u201d<\/p>\n
I know what you\u2019re thinking, \u201cThis isn\u2019t relevance!\u201d And you\u2019re right, at least in the normal sense of the word. The score doesn\u2019t vary across job-seekers or search terms. A better name would be \u201crelevant to Google.\u201d We\u2019re OK with that because a huge share of our job-board traffic comes from Google, as shown below.<\/p>\n
![\"Screenshot](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/03\/Experiments-1024x622.png?resize=1024%2C622&ssl=1\")
\u201cSort by relevance\u201d here is shorthand for \u201crelevant to Google.\u201d (Image by author)<\/figcaption><\/figure>\n In Math<\/a>: <\/strong>We have N jobs. Every day we generate a vector of N integers between 0 and 100. We feed this vector into a black box named Google. If we do a good job, the black box rewards us with many job applications.<\/p>\n
By putting the \u201cright\u201d jobs at the top of the page (loaded word there), we can improve upon a chronological sort. Before we can identify the right jobs, we need to know how much Google actually rewards higher-placed jobs.<\/p>\n
Day 0: Making progress when you know nothing<\/h2>\n
Sometimes, just to justify all the simplifying assumptions I\u2019m going to make later, I start a project by writing down the math equation I\u2019d lik<\/em>e to solve. I imagine ours looks something like this:<\/p>\n
![\"\"](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/03\/unnamed-64-1024x105.png?resize=1024%2C105&ssl=1\")
<\/figure>\n- \n
- \nS<\/em><\/strong> is our vector of relevancy scores. There are N jobs, so each s_i (<\/em>an element of S) corresponds to a different job. A function called applies<\/em> turns S<\/strong> into a scalar. Each day we\u2019d like to find the S<\/strong> that makes that number as large as possible \u2014 the relevancy scores that generate the greatest number of job applications for intelycare.com\/jobs.<\/li>\n
- \napplies<\/em> is a fine objective function on Day 0. Later on our objective function could change (e.g. revenue, lifetime value). Applies are easy to count, though, and lets me spend my complexity tokens<\/a> elsewhere. It\u2019s Day 0, people. We\u2019ll come back to these questions on Day 1.<\/li>\n
- Problem. We know nothing about the applies<\/em> function until we start feeding it relevancy scores.
![\"\ud83d\ude31\"](\"https:\/\/i0.wp.com\/s.w.org\/images\/core\/emoji\/15.0.3\/72x72\/1f631.png?ssl=1\")
\n<\/li>\n<\/ul>\nFirst things first: <\/strong>Seeing that we know nothing about the applies<\/em> function, our first question is, \u201chow do we choose an ongoing wave of daily S vectors so we can learn<\/em> what the applies<\/em> function looks like?\u201d<\/p>\n
- \n
- We know (1) which jobs are boosted and when, (2) how many applies each job receives each day. Note the absence of page-load data. It\u2019s Day 0! You might not have all the data you want on Day 0, but if we\u2019re clever, we can make do with what we have.<\/li>\n
- Note the subtle change in our objective. Earlier, our goal was to accomplish some business objective (maximize applies), and eventually, we\u2019ll come back to that goal. We took off our business hat for a minute and put on our science hat. Our only goal now is to learn something. If we can learn something, we can use it (later) to help achieve some business objective.
![\"\ud83e\udd13\"](\"https:\/\/i0.wp.com\/s.w.org\/images\/core\/emoji\/15.0.3\/72x72\/1f913.png?ssl=1\")
\n<\/li>\n - Since our goal is to learn something, <\/em>above all we want to avoid learning nothing<\/em>. Remember it\u2019s Day 0 and we have no guarantee that the Google Monster will pay any attention to how we sort things. We may as well go for broke and make sure this thing works before throwing more time at improving it.<\/li>\n<\/ul>\n
How do we choose an initial wave of daily S vectors? <\/strong>We\u2019ll give every job a score of 0 (default score), and choose a random<\/em> subset of jobs to boost to 100.<\/p>\n
- \n
- Maybe I\u2019m stating the obvious, but it has to be random if you want to isolate the effect of page-position on job applications. We want the only difference between boosted jobs and other jobs to be their relative ordering on the page as determined by our relevance scores. [I can\u2019t tell you how many phone screens I\u2019ve conducted where a candidate doubled down on running an A\/B test with the good customers in one group and the bad customers in the other group. In fairness, I\u2019ve also vetted marketing-tech vendors who do the same thing
![\"\ud83d\ude2d\"](\"https:\/\/i0.wp.com\/s.w.org\/images\/core\/emoji\/15.0.3\/72x72\/1f62d.png?ssl=1\")
].<\/li>\n - The randomness will be nice later on for other reasons. It\u2019s likely that some jobs benefit from page-placement more than others. We\u2019ll have an easier time identifying those jobs with a big, randomly-generated dataset.<\/li>\n<\/ul>\n
The plan: Subtle but important details<\/h2>\n
We know we can\u2019t boost every<\/em> job. Anytime I put a job at the top of the page, I bump all other jobs down the page (classic example of a \u201cspillover<\/a>\u201d).<\/p>\n
The spillover gets worse as I boost more and more jobs, I impose a greater and greater punishment on all other jobs by pushing them down in the sort (including other boosted jobs).<\/p>\n
- \n
- With little exception, nursing jobs are in-person and local, so any boosting spillovers will be limited to other nearby jobs. This is important.<\/li>\n<\/ul>\n
How do we choose an initial wave of daily S vectors? (final answer) <\/strong>We\u2019ll give every job a score of 0 (default score), and choose a random subset of jobs to boost to 100. The size of the random subset will vary across geographies.<\/em><\/p>\n
- \n
- We create 4 groups of distinct geographies with roughly the same amount of web traffic in each group. Each group is balanced along the key dimensions we think are important. We randomly boost a different percentage of jobs in each group.<\/li>\n<\/ul>\n
Here\u2019s how it looked\u2026<\/p>\n
![\"\"](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/03\/Experiments-4-1024x560.png?resize=1024%2C560&ssl=1\")
Daily Applies for boosted vs unboosted Jobs. Note how boosted jobs do better when there are fewer of them. (Image by author)<\/em><\/figcaption><\/figure>\n - \n
- Each black circle represents a different geography. Its elevation shows the difference in applies-per-job between boosted jobs and all other jobs (measured as a percent).<\/li>\n
- While groups are balanced in aggregate, the individual geographies vary considerably. The balance is still important though. Otherwise, what you see in the chart could be an artifact of the mix of urban\/rural or large\/small geographies in each group. As it is, we\u2019re confident the results come from our relevancy scores.<\/li>\n
- A quick-and-dirty interpretation of this chart is something like, \u201cthe 5% of jobs at the top of the page have ~26% more applies per day than the 95% of jobs placed below. The 10% of jobs at the top of the page have ~21% more applies per day than the 90% of jobs underneath\u2026\u201d and so on. I would never be so bold as to say that in real life, but in a perfect-experiment world it would be true.<\/li>\n
- By the time we boost 25% of jobs, the boost experience is entirely averaged out! We diluted the perks of premium placement to practically nothing for the median geography. \u201cAnd when everyone is super, no one will be! <evil laugh>.\u201d<\/a> Can you imagine learning this the hard way?<\/li>\n
- There are many other layers to peel back. Perhaps dilution happens more quickly for nursing specialties with many pages of listings? What about states that overlap with our long-standing per-diem staffing business? Many fine questions, we have answers for some, but all more than I can include in this post.<\/li>\n<\/ul>\n
What comes next? <\/strong>Day 1 is when the real fun begins!
![\"\ud83c\udf89\"](\"https:\/\/i0.wp.com\/s.w.org\/images\/core\/emoji\/15.0.3\/72x72\/1f389.png?ssl=1\")
<\/p>\n- \n
- We now have guardrails against diluting our premium experience (super important), but what is the best ~10% of jobs to boost each day? Obviously our paying customers have priority, but then what?<\/li>\n
- Does boost help some jobs more than others? The randomly-generated data from our experiment is well suited to answer this and many other questions. We\u2019ll save those questions for future posts.<\/li>\n
- Once we have a strategy for boosting, is our objective really to maximize the total<\/em> number of applies? Or do we only care about the applies for boosted jobs?
![\"\ud83e\udd14\"](\"https:\/\/i0.wp.com\/s.w.org\/images\/core\/emoji\/15.0.3\/72x72\/1f914.png?ssl=1\")
(Sometimes I miss the Day 0 days when all the jobs were equally relevant. Might be time to revisit those equations at the top of the post.)<\/li>\n<\/ul>\nKey takeaways for those who made it this far<\/h2>\n
- \n
- By being thoughtful about how we generated our initial data, we quickly found a convincing answer to our question, set ourselves up to answer many future questions, and saved ourselves a ton of time trying to build an uplift model on non-existent historical data.<\/li>\n
- Thinking of a test? Go for it! If you execute well, you can see the results clearly in a chart and avoid all the complicated statistics (obligatory xkcd<\/a> reference). [hmm, maybe *most* of the statistics. I still love a good regression table.]<\/li>\n
- Spillovers are everywhere. Sometimes varying the treatment across an aggregated group can help like it did here. That can quickly axe your sample-size, but I find it better to have a small data set with meaning than a big data set that\u2019s hot garbage.<\/li>\n<\/ul>\n
Bonus: We ran this experiment in 2023. How are things now?<\/h2>\n
At the time of our little geo-randomized experiment, you see in the charts that our premium job openings performed ~25% better than regular jobs (meaning they had 25% more applies on average).<\/p>\n
Why it matters: <\/strong>We\u2019ve taken over a year to grow and iterate our product to ensure our premium listings deliver the best possible experience. Looking at some recent numbers\u2026 (literally running the queries as I write this)<\/p>\n
- \n
- Boosted job openings receive 425% more applies <\/strong>than regular openings<\/li>\n
- Boosted jobs are 450% more likely to have receive at least one apply <\/strong>compared to regular openings<\/li>\n<\/ul>\n
Not bad! This isn\u2019t randomized, so that 425% includes all sorts of selection bias, additional product work, a crack SEO team, and a successful email operation, all in addition to the incremental effects from premium page position. Importantly, all the extra product and marketing work is focused on a small number of jobs as our initial testing recommends.
![\"\ud83c\udfc6\"](\"https:\/\/i0.wp.com\/s.w.org\/images\/core\/emoji\/15.0.3\/72x72\/1f3c6.png?ssl=1\")
<\/p>\nThe post Experiments Illustrated: How We Optimized Premium Listings on Our Nursing Job Board<\/a> appeared first on Towards Data Science<\/a>.<\/p>\n<\/div>\n
- Boosted jobs are 450% more likely to have receive at least one apply <\/strong>compared to regular openings<\/li>\n<\/ul>\n
- Spillovers are everywhere. Sometimes varying the treatment across an aggregated group can help like it did here. That can quickly axe your sample-size, but I find it better to have a small data set with meaning than a big data set that\u2019s hot garbage.<\/li>\n<\/ul>\n
- There are many other layers to peel back. Perhaps dilution happens more quickly for nursing specialties with many pages of listings? What about states that overlap with our long-standing per-diem staffing business? Many fine questions, we have answers for some, but all more than I can include in this post.<\/li>\n<\/ul>\n
- We create 4 groups of distinct geographies with roughly the same amount of web traffic in each group. Each group is balanced along the key dimensions we think are important. We randomly boost a different percentage of jobs in each group.<\/li>\n<\/ul>\n
- With little exception, nursing jobs are in-person and local, so any boosting spillovers will be limited to other nearby jobs. This is important.<\/li>\n<\/ul>\n
- Maybe I\u2019m stating the obvious, but it has to be random if you want to isolate the effect of page-position on job applications. We want the only difference between boosted jobs and other jobs to be their relative ordering on the page as determined by our relevance scores. [I can\u2019t tell you how many phone screens I\u2019ve conducted where a candidate doubled down on running an A\/B test with the good customers in one group and the bad customers in the other group. In fairness, I\u2019ve also vetted marketing-tech vendors who do the same thing
- \napplies<\/em> is a fine objective function on Day 0. Later on our objective function could change (e.g. revenue, lifetime value). Applies are easy to count, though, and lets me spend my complexity tokens<\/a> elsewhere. It\u2019s Day 0, people. We\u2019ll come back to these questions on Day 1.<\/li>\n
- \nS<\/em><\/strong> is our vector of relevancy scores. There are N jobs, so each s_i (<\/em>an element of S) corresponds to a different job. A function called applies<\/em> turns S<\/strong> into a scalar. Each day we\u2019d like to find the S<\/strong> that makes that number as large as possible \u2014 the relevancy scores that generate the greatest number of job applications for intelycare.com\/jobs.<\/li>\n
- Choosing what to test & How IntelyCare tested its referral bonus program into existence (coming soon)<\/li>\n<\/ul>\n