Running cool experiments is easily one of my favorite parts of working in data science.<\/p>\n
Most experiments don\u2019t deliver big wins, so the winners make for fun stories. We\u2019ve had a few of these at IntelyCare<\/a>, and I\u2019m sharing each story in a way that highlights a concept related to experimentation.<\/p>\n And in this post, we\u2019ll share a story about how we avoided doing something stupid by running an experiment first, and using it to discuss the multiple comparisons problem<\/a>.<\/p>\n IntelyCare connects nurses with work opportunities ranging from full-time work to individual shifts. When dealing with individual shifts, clinicians work for IntelyCare as employees (agency model). This means we\u2019re hiring nurses 24\/7.<\/p>\n You may have suppressed this memory, but in 2020 and 2021 we had this global pandemic. Hiring nurses during the pandemic was nothing short of a rock fight. We had full business permission to try everything and anything that could help us hire nurses more quickly and efficiently.<\/p>\n Working anywhere in healthcare means submitting a big pile of paperwork \u2014 licenses, immunizations, certifications, and more in addition to the regular resumes, references, and background checks.<\/p>\n IntelyCare is no different. And even though we make it all phone-friendly and digital, submitting all this paperwork is about as fun as filing your taxes. And that means many people who apply give up somewhere between creating an account and completing a shift.<\/p>\n We tried lots of things (including different referral incentives). One easy-to-try proposal was to just pay clinicians an extra $100 when they complete their first shift.<\/p>\n Why $100? Because it\u2019s a nice round number and looks good on Marketing<\/a> materials. You might be surprised how many business decisions are made this way (unless you\u2019re in marketing, in which case it\u2019s perfectly normal).<\/p>\n The idea was so easy we almost went live without a test. There was a lot of pressure to move quickly and we wanted to be fast. But science prevailed and instead of offering $100 to everybody, we randomly offered bonuses ranging from $0 to $100 in increments of $25.<\/p>\n Clinicians were informed of the bonus via email throughout the application process. (Unless you had a $0 bonus \u2014 no email for you.)<\/p>\n We ran this test for several months to give candidates sufficient time to complete their applications. By the time we circled back to make a decision, we had several thousand applicants at each bonus level.<\/p>\n Spillovers<\/a>? It\u2019s always a possibility but it seems unlikely. Demand for nursing talent was insanely high at the time. I have a hard time imagining clinicians with high bonuses stealing all the shifts from those with bonuses (thereby exaggerating the impact of the high bonus). There were plenty of shifts to go around.<\/p>\n If you ever run a test like this, chances are some higher up will ask you to \u201cslice and dice\u201d or \u201ccut\u201d or perhaps \u201cdig into\u201d the data 100 different ways. This is fun but also dangerous<\/a>. Wait, dangerous?! Let\u2019s discuss.<\/p>\n So what is an analyst to do? There are many heuristics, all of which make it harder to reject a null hypothesis.<\/p>\n If you\u2019re interested in a more detailed summary, I\u2019d recommend the following:<\/p>\n We\u2019re a curious bunch and so considered looking at several cuts of our experiment data: location, age, qualification, and more. Wouldn\u2019t it be amazing if bonuses were ineffective for nurses\u2026 except for nurses younger than 30 years old living in Rhode Island with active Netflix accounts? Many marketing teams are ready to jump at exactly these kinds of \u201cpatterns\u201d and I\u2018m kindly going to ask you to show me your Bonferroni receipts.<\/p>\n After taking multiple comparisons into account, we found one<\/em> dimension that was truly meaningful \u2014 whether the applicant was a nurse or a nursing assistant (CNA).<\/p>\n Without a bonus, nurses and nursing assistants went on to complete a shift at about the same rate. Nursing assistants were more likely to start working with a bonus of any amount. Nurses, on the other hand, were less likely<\/em> to start working! (And yes these are all stat sig different from no bonus, for all you skeptics out there).<\/p>\n For any readers from outside healthcare, it\u2019s important to know that nurses can easily earn between 2X and 4X the hourly rate of a nursing assistant. These populations differ in so many ways, which is why we put this dimension at the top of our sequential-testing list.<\/p>\n Years later, I still scratch my head at this chart and wonder why completion rates decreased<\/em> among nurses when we offered more<\/em> money. Maybe no gift is better than a cheap gift? Hospitals at the time were offering signing bonuses as high as $25,000 for full-time work<\/a>.<\/p>\n After running this test, we did away with bonuses for nurses. Maybe some bonus greater than $100 would have improved our funnel metrics? That\u2019s another test for another day.<\/p>\n For CNAs, note the large difference between the no bonus group and the $25 bonus group (nearly 5 full percentage points). From there, each additional $25 has a much smaller effect, and somewhere between $50 and $100 the marginal benefit from bigger bonuses reaches zero. We ended up going with $25 to give us room to bump things up at specific times and places as needed.<\/p>\n Remember the initial proposal was to give $100 to everyone. <\/em>Had we done that, we would have spent $1M extra in bonuses in one year<\/strong> and would likely have recruited the same number of people.<\/p>\n The post Experiments Illustrated: How Random Assignment Saved Us $1M in Marketing Spend<\/a> appeared first on Towards Data Science<\/a>.<\/p>\n<\/div>\n\n
Background: IntelyCare hires nurses at scale\u2026 and it\u2019s covid
![\"\ud83d\ude37\"](\"https:\/\/i0.wp.com\/s.w.org\/images\/core\/emoji\/15.0.3\/72x72\/1f637.png?ssl=1\")
\n<\/h2>\nThe problem: Lots of applies, but not so many new hires<\/h2>\n
The solution: Just throw money at it!
![\"\ud83d\udcb8\"](\"https:\/\/i0.wp.com\/s.w.org\/images\/core\/emoji\/15.0.3\/72x72\/1f4b8.png?ssl=1\")
\n<\/h2>\nTechnical aside: Multiple comparisons<\/h2>\n
\n
Call before you dig: 1-BON-FER-RONI<\/h3>\n
\n
\n
Back to the bonuses<\/h2>\n
![\"\"](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/03\/Experiments-illustrated-1-1024x617.png?resize=1024%2C617&ssl=1\")
What\u2019s the optimal bonus amount?<\/h2>\n
Key takeaways for those who made it this far<\/h2>\n
\n