Regression Discontinuity Design: How It Works and When to Use It<\/h1>\n
You\u2019re an avid data scientist and experimenter. You know that randomisation is the summit of Mount Evidence Credibility, and you also know that when you can\u2019t randomise, you resort to observational data and Causal Inference<\/a> techniques. At your disposal are various methods for spinning up a control group \u2014 difference-in-differences, inverse propensity score weighting, and others. With an assumption here or there (some shakier than others), you estimate the causal effect and drive decision-making. But if you thought it couldn\u2019t get more exciting than \u201cvanilla\u201d causal inference, read on.<\/p>\n Personally, I\u2019ve often found myself in at least two scenarios where \u201cjust doing causal inference\u201d wasn\u2019t straightforward. The common denominator in these two scenarios? A missing control group \u2014 at first glance, that is.<\/p>\n First, the cold-start scenario:<\/strong> the company wants to break into an uncharted opportunity space. Often there is no experimental data to learn from, nor has there been any change (read: \u201cexogenous shock\u201d), from the business or product side, to leverage in the more common causal inference frameworks like difference-in-differences (and other cousins in the pre-post paradigm).<\/p>\n Second, the unfeasible randomisation scenario:<\/strong> the organisation is perfectly intentional about testing an idea, but randomisation is not feasible\u2014or not even wanted. Even emulating a natural experiment might be constrained legally, technically, or commercially (especially when it\u2019s about pricing), or when interference bias arises in the marketplace.<\/p>\n These situations open up the space for a \u201cdifferent\u201d type of causal inference. Although the method we\u2019ll focus on here is not the only one suited for the job, I\u2019d love for you to tag along in this deep dive into Regression Discontinuity<\/a> Design (RDD).<\/p>\n In this post, I\u2019ll give you a crisp view of how<\/em> and why<\/em> RDD works. Inevitably, this will involve a bit of math \u2014 a pleasant sight for some \u2014 but I\u2019ll do my best to keep it accessible with classic examples from the literature.<\/p>\n We\u2019ll also see how RDD can tackle a thorny causal inference challenge in e-commerce and online marketplaces: the impact of listing position on listing performance. In this practical section we\u2019ll cover key modelling considerations that practitioners often face: parametric versus non-parametric RDD, choosing the right bandwidth parameter, and more. So, grab yourself a cup of of coffee and let\u2019s jump in!<\/p>\n Regression Discontinuity Design exploits cutoffs \u2014 thresholds \u2014 to recover the effect of a treatment on an outcome. More precisely, it looks for a sharp change in the probability of treatment assignment on a \u2018running\u2019 variable. If treatment assignment depends solely on the running variable, and the cutoff is arbitrary, i.e. exogenous, then we can treat the units around it as randomly assigned. The difference in outcomes just above and below the cutoff gives us the causal effect.<\/p>\n For example, a scholarship awarded only to students scoring above 90, creates a cutoff based on test scores. That the cutoff is 90 is arbitrary \u2014 it could have been 80 for that matter; the line had just to be drawn somewhere. Moreover, scoring 91 vs. 89 makes the whole difference as for the treatment: either you get it or not. But regarding capability<\/em>, the two groups of students that scored 91 and 89 are not really different, are they? And those who scored 89.9 versus 90.1 \u2014\u00a0if you insist? <\/p>\n Making the cutoff could come down to randomness, when it\u2019s just a bout a few points. Maybe the student drank too much coffee right before the test \u2014 or too little. Maybe they got bad news the night before, were thrown off by the weather, or anxiety hit at the worst possible moment. It\u2019s this randomness that makes the cutoff so instrumental<\/em> in RDD.<\/p>\n Without a cutoff, you don\u2019t have an RDD\u00a0\u2014\u00a0just a scatterplot and a dream. But, the cutoff by itself is not equipped with all it takes to identify the causal effect. Why it works hinges on one core identification assumption: continuity.<\/p>\n If the cutoff is the cornerstone of the technique, then its importance comes entirely from the continuity assumption. The idea is a simple, counterfactual one: had there been no treatment, then there would\u2019ve been no effect.<\/p>\n To ground the idea of continuity, let\u2019s jump straight into a classic example from public health: does legal alcohol access increase mortality?<\/p>\n Imagine two worlds where everyone and everything is the same. Except for one thing: a law that sets the minimum legal drinking age at 18 years (we\u2019re in Europe, folks).<\/p>\n In the world with the law (the factual world), we\u2019d expect alcohol consumption to jump right after age 18. Alcohol-related deaths should jump too, if there is a link.<\/p>\n Now, take the counterfactual world where there is no such law; there should be no such jump. Alcohol consumption and mortality would likely follow a smooth trend across age groups.<\/p>\n Now, that\u2019s a good thing for identifying the causal effect;\u00a0the absence of a jump in deaths in the counterfactual world is the necessary<\/em> condition to interpret a jump in the factual world as the impact of the law.<\/p>\n Put simply: if there is no treatment, there shouldn\u2019t be a jump in deaths. If there is, then something other than our treatment is causing it, and the RDD is not valid.<\/p>\n The continuity assumption can be written in the potential outcomes framework as:<\/p>\n begin{equation} Where (Y_i(0)) is the potential outcome, say, risk of death of subject (\/mathbb{i}) under no treatment.<\/p>\n Notice that the right-hand side is a quantity of the counterfactual world; not one that can be observed in the factual world, where subjects are treated if they fall above the cutoff.<\/p>\n Unfortunately for us, we only have access to the factual world, so the assumption cannot be tested directly. But, luckily, we can proxy it. We will see placebo groups achieve this later in the post. But first, we start by identifying what<\/em> can break the assumption:<\/p>\n Hopefully, it\u2019s clear what constitutes a RDD: the running variable, the cutoff, and most importantly, reasonable grounds to defend that continuity holds. With that, you\u2019ve gotten yourself a neat and effective causal inference design for questions that can\u2019t be answered by an A\/B test, nor by some of the more common causal inference techniques like diff-in-diff, nor with stratification.<\/p>\n In the next section, we continue shaping our understanding of how RDD works; how does RDD \u201ccontrol\u201d confounding relationships? What exactly does it estimate? Can we not just control for the running variable too? These are questions that we tackle next.<\/p>\n If you are already familiar with instrumental variables (IV), you may see the similarities: both RDD and IV leverage an exogenous variable that does not cause the outcome directly, but does influence the treatment assignment, which in turn may influence the outcome. In IV this is a third variable Z; in RDD it\u2019s the running variable that serves as an instrument.<\/p>\n Wait. A third variable; maybe. But an exogenous one? That\u2019s less clear. <\/p>\n In our example of alcohol consumption, it is not hard to imagine that age \u2014 the running variable \u2014 is a confounder. As age increases, so might tolerance for alcohol, and with it the level of consumption. That\u2019s a stretch, maybe, but not implausible.<\/p>\n Since treatment (legal minimum age) depends on age \u2014 only units above 18 are treated \u2014 treated and untreated units are inherently different. If age also influences the outcome, through a mechanism like the one sketched above, we got ourselves an apex confounder.<\/p>\n Still, the running variable plays a key role. To understand why, we need to look at how RDD and instruments leverage the frontdoor criterion to identify causal effects.<\/p>\n Perhaps almost instinctively, one may answer with controlling<\/em> for the running variable; that\u2019s what stratification taught us. The running variable is confounder, so we include it in our regression, and close the backdoor. But doing so would cause some trouble. <\/p>\n Remember, treatment assignment depends on the running variable so that everyone above the cutoff is treated with all<\/em> certainty, and certainly not <\/em>below it. So, if we control for the running variable, we run into two very related problems:<\/p>\n So no \u2014 conditioning on the running variable doesn\u2019t make the running variable the exogenous instrument that we\u2019re after. Instead, the running variable becomes exogenous by pushing it to the limit\u2014quite literally. There where the running variable approaches the cutoff from either side, the units are the same with respect to the running variable. Yet, falling just above or below makes the difference as for getting treated or not. This makes the running variable a valid instrument, if treatment assignment is the only thing that happens at the cutoff. Judea Pearl refers to instruments as meeting the front-door criterion.<\/p>\n So, in essence, we\u2019re controlling for the running variable \u2014 but only near the cutoff. That\u2019s why RDD identifies the local<\/em> average treatment effect (LATE), a special flavour of the average treatment effect (ATE). The LATE looks like:<\/p>\n $$delta_{SRD}=Ebig[Y^1_i \u2013 Y_i^0mid X_i=c_0]$$<\/p>\n The local<\/em> bit refers to the partial scope of the population we are estimating the ATE for, which is the subpopulation around the cutoff. In fact, the further away the data point is from the cutoff, the more the running variable acts as a confounder, working against<\/em> the RDD instead of in its favour.<\/p>\n Back to the context of the minimum age for legal alcohol consumption example. Adolescents who are 17 years and 11 months old are really not so different from those that are 18 years and 1 month old, on average. If anything, a month or two difference in age is not going to be what sets them apart. Isn\u2019t that the essence of conditioning on, or holding a variable constant? What sets them apart is that the latter group can consume alcohol legally for being above the cutoff, and not the former. <\/p>\n This setup enables us to estimate the LATE for the units around the cutoff and with that, the effect of the minimum age policy on alcohol-related deaths.<\/p>\n We\u2019ve seen how the continuity assumption has to hold to make the cutoff an interesting point along the running variable in identifying the causal effect of a treatment on the outcome. Namely, by letting the jump in the outcome variable be entirely attributable to the treatment. If continuity holds, the treatment is as-good-as-random near the cutoff, allowing us to estimate the local average treatment effect.<\/p>\n In the next section, we\u2019ll walk through the practical setup of a real-world RDD: we identify the key concepts; the running variable and cutoff, treatment, outcome, covariates, and finally, we estimate the RDD after discussing some crucial modelling choices, and end the section with a placebo test.<\/p>\n In e-commerce and online marketplaces, the starting point of the buyer experience is searching for a listing. Think of the visitor typing \u201cNikon F3 analogue camera\u201d in the search bar. Upon carrying out this action, algorithms frantically sort through the inventory looking for the best matching listings to populate the search results page.<\/p>\n Time and attention are two scarce resources.<\/strong> So, it is in the interest of everyone involved \u2014 the buyer, the seller and the platform \u2014\u00a0to reserve the most prominent positions on the page for the matches with the highest anticipated chance to become successful trades.<\/p>\n Additionally, position effects in consumer behaviour suggest that users infer higher credibility and desirability from items \u201cranked\u201d at the top<\/strong>. Think about high-tier products being placed at eye-height or above in supermarkets, and highlighted items on an e-commerce platform, at the top of the homepage. <\/p>\n So, the question then becomes: how does positioning on the search results page influence a listing\u2019s chances to be sold?<\/p>\n Hypothesis<\/em>: As with any good hypothesis, we need a bit of theory to ground it. Good for us is that we are not trying to find the cure for cancer. Our theory is about well-understood psychological phenomena and behavioural patterns, to put it overly sophisticated.\u00a0<\/p>\n Think of primacy effect<\/a><\/em>, anchoring bias<\/a><\/em> and the resource theory of attention<\/a>. These are well ideas in behavioural and cognitive psychology that back up our plan here.<\/p>\n Kicking off the conversation with a product manager will be more fun this way. Personally, I also get excited when I have to brush up on some psychology. <\/p>\n But I\u2019ve found through and through that a theory is really secondary to any initiative in my industry (tech). Except for a research team and project, arguably. And it\u2019s fair to say it helps us stay on-purpose: what we are doing is to bring business forward, not mother science.\u00a0<\/p>\n<\/blockquote>\n<\/details>\n Knowing the answer has real business value. Product and commercial teams could use it to design new paid features that help sellers get their listings on higher positions \u2014 a win for both the business and the user. It could also clarify the value of on-site real estate like banner positions and ad slots, helping drive growth in B2B advertising.<\/p>\n The question is about incrementality: would\u2019ve listing (mathbb{j}) been sold, had it been ranked 1st on the results page, instead of 15th. So, we want to make a causal statement. That\u2019s hard for at least two reasons: <\/p>\n Let\u2019s expand on that.<\/p>\n One experiment design could randomise the fetched listings across the page slots, independent of the listing relevance. Breaking the inherent link between relevance and position, we would learn the effect of position on listing performance. It\u2019s an interesting idea \u2014 but a costly one.\u00a0<\/p>\n While it\u2019s a reasonable design for statistical inference, this setup is kind of terrible for the user and business. The user might have found what they needed\u2014maybe even made a purchase. But instead, maybe half of the inventory they would have seen was remotely a good match because of our experiment. This suboptimal user experience likely hurts engagement in both the short and long term \u2014 especially for new users who are still to see what value the platform holds for them.\u00a0<\/p>\n Can we think of a way to mitigate this loss? Still committed to A\/B testing, one could expose a smaller set of users to the experiment. While it will scale down the consequences, it may also stand in the way of reaching sufficient statistical power by lowering the sample size. Moreover, even small audiences can be responsible for substantial revenue for some companies still \u2014\u00a0those with millions of users. So, cutting the exposed audience is not a silver bullet either.<\/p>\n Naturally, the way to go is to leave the platform and its users undisturbed \u2014\u00a0 and still find a way to answer the question at hand.\u00a0Causal inference is the right mindset for this, but the question is: how do we do that exactly? <\/p>\n Listings don\u2019t just make it to the top of the page on a good day; it\u2019s their quality, relevance, and the sellers\u2019 reputation that promote the ranking of a listing. Let\u2019s call these three variables W<\/strong>. <\/p>\n What makes W<\/strong> tricky is that it influences both the ranking of the listing and also the probability that the listing gets clicked, a proxy for performance.<\/p>\n In other words, W<\/strong> affects both our treatment (position) and outcome (click), helping itself with the status of confounder<\/em>.<\/p>\n Therefore, our task is to find a design that\u2019s fit for purpose; one that effectively controls the confounding effect of W<\/strong>.<\/p>\n Not all causal inference designs are just sitting around waiting to be picked. Sometimes they show up when you least need them, and sometimes you get lucky when you need them most \u2014 like today.<\/p>\n It looks like we can use the page cutoff to identify the causal impact of position on clicks-through rate.<\/p>\n Let\u2019s unpack the listing recommendation mechanism to see exactly how. Here\u2019s what happens under the hood when a results page is generated for a search:<\/p>\n So, what is exactly our design? Next, we walk through the reasoning and identification of the key ingredients of our design.<\/p>\n In our setup, the running variable is the relevance score (s_j) for listing j. This score is a continuous, complex function of both user and listing properties:<\/p>\n $$s_j = f(u_i, l_j)$$<\/p>\n The listing\u2019s rank (r_j) is simply a rank transformation of (s_j), defined as:<\/p>\n $$r_i = sum_{j=1}^{n} mathbf{1}(s_j leq s_i)$$<\/p>\n Practically speaking, this means that for analytic purposes\u2014such as fitting models, making local comparisons, or identifying cutoff points\u2014knowing a listing\u2019s rank conveys nearly the same information as knowing its underlying relevance score, and vice versa.<\/p>\n The relevance score (s_j) reflects how well a listing matches a specific user\u2019s query, given parameters like location, price range, and other filters. But this score is relative\u2014it only has meaning within the context of the listings returned for that particular search.<\/p>\n In contrast, rank (or position) is absolute. It directly determines a listing\u2019s visibility. I think of rank as a standardising transformation of (s_j). For example, Listing A in search Z might have the highest score of 5.66, while Listing B in search K tops out at 0.99. These raw scores aren\u2019t comparable across searches\u2014but both listings are ranked first in their respective result sets. That makes them equivalent in terms of what really matters here: how visible they are to users.<\/p>\n<\/details>\n If a listing just misses the first page, it doesn\u2019t fall to the bottom of page two \u2014 it is artificially bumped to the top. That\u2019s a lucky break. Normally, only the most relevant listings appear at the top, but here a listing of merely moderate relevance ends up in a prime slot \u2014albeit on the second page \u2014 purely due to the arbitrary position of the page break. Formally, the treatment assignment (D_j) goes like:<\/p>\n $$D_j = begin{cases} 1 & text{if } r_j > 30 \\ 0 & text{otherwise} end{cases}$$<\/p>\n (Note on global rank: Rank 31 isn\u2019t just the first listing on page two; it is still the 31st listing overall)<\/em><\/p>\n The strength of this setup lies in what happens near the cutoff: a listing ranked 30 may be nearly identical in relevance to one ranked 31. A small scoring fluctuation \u2014 or a high-ranking outlier \u2014 can push a listing over the threshold, flipping its treatment status. This local randomness is what makes the setup valid for RDD.<\/p>\n Finally, we operationalise the outcome of interest as the click-though rate from impressions to clicks. Remember that all listings are \u2018impressed\u2019 when when the page is populated. The click is the binary indicator of the desired user behaviour.<\/p>\n In summary, this is our setup:<\/p>\n Next we walk through how to estimate the RDD.\u00a0<\/p>\n In this section, we\u2019ll estimate the causal parameter, interpret it, and connect them back to our core hypothesis: how position affects listing visibility.<\/p>\n Here\u2019s what we\u2019ll cover:<\/p>\n We\u2019re working with impressions data from one of Adevinta\u2019s (ex-eBay Classifieds Group) marketplaces. It\u2019s real data, which makes the whole exercise feel grounded. That said, values and relationships are censored and scrambled where necessary to protect its strategic value.<\/p>\n An important note to how we interpret the RDD estimates and drive decisions, is how the data was collected: only those searches where the user saw both the first and second page were included. <\/p>\n This way, we partial out the page fixed effect if any, but the reality is that many users don\u2019t make it to the second page at all. So there is a big volume gap. We discuss the repercussion in the analysis recap.<\/p>\n The dataset consists of these variables:<\/p>\n The running variable, the cutoff, and the continuity assumption, give you all<\/em> you need to identify the causal effect. But including covariates can sharpen the estimator by reducing variance \u2014 if done right. And, oh is it easy to do it wrong. <\/p>\n The easiest thing to \u201cbreak\u201d about the RDD design, is the continuity assumption. Simultaneously, that\u2019s the last<\/em> thing we want to break (I already rambled long enough about this). <\/p>\n Therefore, the main quest in adding covariates is to it in such way that we reduce variance, while keeping the continuity assumption intact. One way to formulate that, is to assume continuity without covariates and<\/em> with<\/em> covariates:<\/p>\n begin{equation} begin{equation} Where (Z_i) is a vector of covariates, for subject i. Less mathy, two things should remain unchanged after adding covariates:<\/p>\n I did not find out the above myself; Calonico, Cattaneo, Farrell, and Titiunik (2018) did. They developed a formal framework for incorporating covariates into RDD. I\u2019ll leave the details to the paper. For now, some modelling guidelines can keep us going:<\/p>\n Our target model may look like this:<\/p>\n begin{equation} For letting the covariates interact with the treatment indicator, the sort of model we want to avoid<\/em> looks like this:<\/p>\n begin{equation} Now, let\u2019s distinguish between two ways of practically including covariates:<\/p>\n We\u2019ll use residualisation in our case. It\u2019s an effective way reduce noise, produces cleaner visualisations, and protects the strategic value of the data.<\/p>\n The snippet below defines the outcome de-noising model and computes the residualised outcome, In practice, though, the residualisation barely moved the needle this time. We can see that by checking the change in standard deviation:<\/p>\n Even though the denoising didn\u2019t have much effect, we\u2019re still in a good spot. The original outcome variable already has low conditional variance, and patterns around the cutoff are visible to the naked eye, as we can see below.<\/p>\n We move on to a few other modelling decisions that generally have a bigger impact: choosing between parametric and non-parametric RDD, the polynomial degree and the bandwidth parameter (h).<\/p>\n You might wonder why we even have to choose between parametric and non-parametric RDD. The answer lies in how each approach trades off bias and variance<\/strong> in estimating the treatment effect.<\/p>\n Choosing parametric RDD<\/strong> is essentially choosing to reduce variance. It assumes a specific functional form for the relationship between the outcome and the running variable, (mathbb{E}[Y mid X]), and fits that model across the entire dataset. The treatment effect is captured as a discrete jump in an otherwise continuous function. The typical form looks like this:<\/p>\n $$Y = beta_0 + beta_1 D + beta_2 X + beta_3 D cdot X + varepsilon$$<\/p>\n Non-parametric RDD<\/strong>, on the other hand, is about reducing bias. It avoids strong assumptions about the global relationship between Y and X and instead estimates the outcome function separately on either side of the cutoff. This flexibility allows the model to more accurately capture what\u2019s happening right around the threshold. The non-parametric estimator is:<\/p>\n (tau = lim_{x downarrow c} mathbb{E}[Y mid X = x] \u2013 lim_{x uparrow c} mathbb{E}[Y mid X = x])<\/p>\n So, which should you choose? Honestly, it can feel arbitrary. And that\u2019s okay. This is the first in a series of judgment calls that practitioners often call the fun part of RDD. It\u2019s where modelling becomes as much an art as it is a science.<\/p>\n I\u2019ll walk through how I approach that choice. But first, let\u2019s look at two key tuning parameters (especially for non-parametric RDD) that will guide our final decision: the polynomial degree and the bandwidth, h.<\/p>\n The relationship between outcome and the running variable can take many forms, and capturing its true shape is crucial for estimating the causal effect accurately. If you\u2019re lucky, everything is linear and there is no need to think of polynomials \u2014 If you\u2019re a realist, then you probably want to learn how they can serve you in the process.\u00a0<\/p>\n In selecting the right polynomial degree, the goal is to reduce bias, without inflating the variance of the estimator. So we want to allow for flexibility, but we don\u2019t want to do it more than necessary. Take the examples in the image below: with an outcome of low enough variance, the linear form naturally invites the eyes to estimate the outcome at the cutoff. But the estimate becomes biased with only a slightly more complex form, if we enforce a linear shape in the model. Insisting on a linear form in such a complex case is like fitting your feet into a glove: It kind of works, but it\u2019s very ugly.\u00a0<\/p>\n Instead, we give the model more degrees of freedom with a higher-degree polynomial, and estimate the expected Working with polynomials in the way that\u2019s described above does not come free of worries. Two things are required and pose a challenge at the same time:\u00a0<\/p>\n Only then we reduce bias as intended; If one of these two is not the case, we risk adding more of it.\u00a0<\/p>\n The thing is that modelling the entire range properly is more difficult than modelling a smaller range, specially if the form is complex. So, it\u2019s easier to make mistakes. Moreover, the entire range is almost certain not to be relevant to estimate the causal effect \u2014\u00a0the \u201clocal\u201d in LATE gives it away. How do we work around this?<\/p>\n Enter the bandwidth parameter, h. The bandwidth parameters aids the model in leveraging data that is closer to the cutoff, dropping the global<\/em> data idea, and bringing it back to the local scope RDD estimates the effect for. It does so by weighting the data by some function (mathbb{w}(X)) so that more weight is given to entries near the cutoff, and less to the entries further away.<\/p>\n For example, with h = 10, the model considers the range of total length 20; 10 on each side of the cutoff.<\/p>\n The effective weight depends on the function, (mathbb{w}). A bandwidth function that has a hard-boundary behaviour is called a square, or uniform, kernel. Think of it as a function that gives weights 1 when the data is within bandwidth, and 0 otherwise. The gaussian and triangular kernels are two other frequently used kernels by practitioners. The key difference is that these behave less abruptly in weighting of the entries, compared to the square kernel. The image below visualises the behaviour of the three kernels functions.<\/p>\n To me, choosing the final model boils down to the question: what is the simplest model that does the good job? <\/em>Indeed \u2014\u00a0the principle of Occam\u2019s razor never goes out of fashion. In practise, this means:<\/p>\n * This brings us to the generally accepted recommendation in the literature: keep the polynomial degree lower than 3. In most use cases 2 works well enough. Just make sure you pick mindfully.<\/p>\n ** Also, note that h fits especially well in the non-parametric mentality; I see these two choices as co-dependent.<\/p>\n Back to the listing position scenario. This is the final model to me:<\/p>\n Let\u2019s look at the model output. The image below shows us the model summary. If you\u2019re familiar with that, it all will come down to interpreting the parameters.<\/p>\n The first thing to look at is that treated listings have ~1% point higher probability of being clicked, than untreated listings. To put that in perspective, that\u2019s a +20% change if the click rate of the control is 5%, and ~ +1% increase if the control is 80%. When it comes to practical significance<\/em> of this causal effect, these two uplifts are day and night. I\u2019ll leave this open-ended with a few questions to take home: when would you and your team label this impact as an opportunity to jump on? What other data\/answers do we need to declare this track worthy of following?<\/p>\n The remainder of the parameters don\u2019t really add much to the interpretation of the causal effect. But let\u2019s go over them quickly, nonetheless. The second estimate (x) is that of the slope below cutoff slope; the third one (D x (mathbb(x))) is the additional [negative] points added to the previous slope to reflect the slope above<\/em> the cutoff; Finally, the intercept is the average for the units right below the cutoff. Because our outcome variable is residualised, the value -0.012 is the demeaned outcome; it no longer is on the scale of the original outcome.<\/p>\n I\u2019ve put this image together to show a collection of other possible models, had we made different choices in bandwidth, polynomial degree, and parametric-versus-not. Although hardly any of these models would have put the decision maker on a totally wrong path in this particular dataset, each model comes with its bias and variance properties. This does<\/em> colour our confidence<\/em> of the estimate.<\/p>\n In any causal inference method, the identification assumption is everything. One thing is off, and the entire analysis crumbles. We can pretend everything is alright, or we put our methods to the test ourselves (believe me, it\u2019s better when you break your own analysis before it goes out there)<\/p>\n Placebo testing is one way to corroborate the results. Placebo testing checks the validity of results by using a setup identical to the real one, minus the actual treatment. If we still see an effect, it signals a flawed design \u2014 continuity can\u2019t be assumed, and causal effects can\u2019t be identified.<\/p>\n Good for us, we have a placebo group. The 30-listing page cut only exists on the desktop version of the platform. On mobile, infinite scroll makes it one long page; no pagination, no page jump. So the effect of \u201cgoing to the next page\u201d shouldn\u2019t appear, and it doesn\u2019t.<\/p>\n I don\u2019t think we need to do much inference. The graph below already tells us the entire story: without pages, going from the 30th position to the 31st is not different from going from any other position to the next. More importantly, the function is smooth at the cutoff. This finding adds a great deal of credibility to our analysis by showcasing that continuity holds in this placebo group.<\/p>\n The placebo test is one of the strongest checks in an RDD. It tests the continuity assumption almost directly<\/em>, by treating the placebo group as a stand-in for the counterfactual. <\/p>\n Of course, this relies on a new assumption: that the placebo group is valid; that it is a sufficiently good counterfactual. So the test is powerful only if that assumption is more credible than assuming continuity without evidence.<\/p>\n Which means that we need to be open to the possibility that there is no proper placebo group. How do we stress-test our design then?<\/p>\n Quick recap. There are two related sources of confounding and hence to violating the continuity assumption: <\/p>\nOutline<\/h2>\n
\n
\n
\n
\n
How and why RDD works\u00a0<\/h2>\n
The continuity assumption, and parallel worlds<\/h3>\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/04\/RDD_-_Parallel_worlds-3-1024x538.png?resize=1024%2C538&ssl=1\")
lim_{x to c^-} mathbb{E}[Y_i(0) mid X_i = x] = lim_{x to c^+} mathbb{E}[Y_i(0) mid X_i = x]
label{eq: continuity_po}
end{equation}<\/p>\n\n
When units can influence their position with regard to the cutoff, it may be that units who did so are inherently different from those who did not. Hence, cutoff manipulation can result in selection bias: a form of confounding. Especially if treatment assignment is binding, subjects may try their best to get one version of the treatment over the other. <\/li>\n<\/ol>\nRDD and instruments<\/h3>\n
Backdoor vs. frontdoor<\/em><\/h4>\n
\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/04\/DAG_-_IV-2-1.png?ssl=1\")
LATE, not ATE<\/em><\/h3>\n
RDD in Action: Search Ranking and listing performance Example<\/h2>\n
If a listing is ranked higher on the search results page, then it will have a higher chance of being sold, because higher-ranked listings get more visibility and attention from users.<\/p>\nIntermezzo: business or theory?<\/summary>\n
\n
\n
The cost of A\/B testing<\/h3>\n
Confounders<\/h3>\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/04\/DAG_-_Fork-4-1024x538.png?resize=1024%2C538&ssl=1\")
You don\u2019t choose regression discontinuity \u2014 it chooses you<\/em><\/h3>\n
Abrupt cutoff in search results pagination<\/h4>\n
\n
A coarse set of listings is pulled from the inventory, based on filters like location, radius, and category, etc.<\/li>\n
<\/strong>This step uses user history and listing quality proxies to predict what the user is most likely to click.<\/li>\n
<\/strong>Higher scores get higher ranks. Business rules mix in ads and commercial content with organic results.<\/li>\n
<\/strong>Listings are slotted by absolute relevance score. A results page ends at the k<\/em>th<\/sup> listing, so the k+1<\/sub><\/em>th<\/sup> listing appears at the top of the next<\/em> page. This is is going to be crucial to our design.<\/li>\n
<\/strong>Users see the results in order of relevance. If a listing catches their eye, they might click and view more details: one step closer to the trade.<\/li>\n<\/ol>\nPractical setup and variables<\/h3>\n
The running variable<\/h4>\n
Details: Relevance score vs. rank<\/summary>\n
The cutoff, and treatment<\/h4>\n
The outcome: Impression-to-click<\/h4>\n
\n
Estimating RDD<\/h3>\n
\n
Meet the data<\/h3>\n
\n
\n\n
\n click<\/td>\n rel_position<\/td>\n D<\/td>\n category<\/td>\n organic<\/td>\n boosted<\/td>\n<\/tr>\n \n 1<\/td>\n -3<\/td>\n 0<\/td>\n A<\/td>\n 1<\/td>\n 0<\/td>\n<\/tr>\n \n 1<\/td>\n -14<\/td>\n 0<\/td>\n A<\/td>\n 1<\/td>\n 0<\/td>\n<\/tr>\n \n 0<\/td>\n 3<\/td>\n 1<\/td>\n C<\/td>\n 1<\/td>\n 0<\/td>\n<\/tr>\n \n 0<\/td>\n 10<\/td>\n 1<\/td>\n D<\/td>\n 0<\/td>\n 0<\/td>\n<\/tr>\n \n 1<\/td>\n -1<\/td>\n 0<\/td>\n K<\/td>\n 1<\/td>\n 1<\/td>\n<\/tr>\n<\/tbody>\n<\/table> Covariates: how to include them to increase accuracy?<\/h3>\n
lim_{x to c^-} mathbb{E}[Y_i(0) mid X_i = x] = lim_{x to c^+} mathbb{E}[Y_i(0) mid X_i = x] text{(no covariates)}
end{equation}<\/p>\n
lim_{x to c^-} mathbb{E}[Y_i(0) mid X_i = x, Z_i] = lim_{x to c^+} mathbb{E}[Y_i(0) mid X_i = x, Z_i] text{(covariates)}
end{equation}<\/p>\n\n
\n
Y_i = alpha + tau D_i + f(X_i \u2013 c) + beta^top Z_i + varepsilon_i
end{equation}<\/p>\n
Y_i = alpha + tau D_i + f(X_i \u2013 c) + beta^top (Z_i cdot D_i) + varepsilon_i
end{equation}<\/p>\n\n
click_res<\/code>. The idea is simple: once we strip out the variance explained by the covariates, what remains is a less noisy version of our outcome variable\u2014at least in theory. Less noise means more accuracy.<\/p>\nSD(click_res) \/ SD(click) - 1<\/code> gives us about -3%, which is small practically speaking. <\/p>\n# denoising clicks\nmod_outcome_model <- lm(click ~ l1 + organic + boosted, \n data = df_listing_level)\n\ndf_listing_level$click_res <- residuals(mod_outcome_model)\n\n# the impact on variance is limited: ~ -3%\nsd(df_listing_level$click_res) \/ sd(df_listing_level$click) - 1<\/code><\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/04\/visible_jump-1-1024x576.png?resize=1024%2C576&ssl=1\")
Modelling choices in RDD<\/h3>\n
Parametric vs non-parametric RDD<\/h4>\n
Polynomial degree<\/h4>\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/04\/RDD_-_form-2-1024x538.png?resize=1024%2C538&ssl=1\")
The bandwidth parameter: h<\/h4>\n
\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/04\/DAG_-_Fork-3-1024x538.png?resize=1024%2C538&ssl=1\")
Everything put together: non- vs. parametric RDD, polynomial degree and bandwidth<\/h4>\n
\n
# modelling the residuals of the outcome (de-noised)\nmod_rdd <- lm(click_res ~ D + ad_position_idx,\n weight = triangular_kernel(x = ad_position_idx, c = 0, h = 10),\u00a0 # this is h\n data = df_listing_level)<\/code><\/pre>\nInterpreting RDD results<\/h3>\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/04\/Screenshot-2025-04-22-at-09.49.18.png?ssl=1\")
<\/figure>\nDifferent choices, different models<\/h3>\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/04\/rdd_models-1024x576.png?resize=1024%2C576&ssl=1\")
<\/figure>\nPlacebo testing<\/h3>\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/04\/AD_4nXe-rSOTvzK0PzlKdvoSvAlirM8tA2MAJd4qlkrGNrskuQJgaQEO9I-35tY2GzmGrMxS9i_bB4_Qajl15vWYue52e1I5eLbokeJ4G1cTsE8JYwtAMZgDnpIc4LcTC2caJGvWgCO6eg.png?ssl=1\")
<\/figure>\nNo-manipulation and the density continuity test<\/h3>\n
\n