How to Use Incrementality Testing to Improve Attribution

RMI Contributor: Igor Vladimirovskiy

Attribution has long been the go-to approach for measuring marketing effectiveness. By tracking the touchpoints that lead to a conversion, attribution models aim to show which channels are driving sales. With the digital marketplace constantly in flux and external factors at play, it has become increasingly fleeting and less predictive. Many marketers are turning to incrementality testing to understand the true impact of their media spend.

In this article, we’ll explore why traditional attribution is falling short, the principles behind incrementality testing, and how you can use it to gain deep insight into enhancing your attribution model.

The Drawbacks of Attribution

Attribution models, whether first-click, last-click, or multi-touch, often reflect a biased view of which channels or touchpoints contribute to conversions. Why? Because each model is designed to favor specific touchpoints. For example, a last-click model will prioritize the last interaction before a sale, potentially disregarding the contributions of any earlier touchpoints that helped nurture the customer along the way. This creates an incomplete picture of which channels are genuinely driving sales.

Moreover, attribution models can be skewed by externalities, such as seasonality, competitive activity, and shifting consumer behavior. Because of this, you need to find models that cater to your business’ needs, otherwise this can lead to budget mishaps and underperforming campaigns. It’s also important to have a few different attribution views that have some predictive value to identify which marketing actions will likely result in a conversion.

What is Incrementality Testing?

Incrementality testing takes a different approach. Rather than focusing on which touchpoints are involved in the customer journey, it seeks to answer a more fundamental question: what is the true sales contribution of a specific marketing channel or media spend? By isolating and measuring the lift in sales caused by a channel, incrementality testing reveals the real value of each marketing effort.

Incrementality testing typically involves holdouts that help measure the effectiveness of a marketing action. A holdout is a control group that is intentionally excluded from a specific marketing treatment to serve as a baseline. By comparing the results between those exposed to the marketing effort and those not, marketers can determine the incremental impact—essentially, the lift in sales or conversions directly attributed to that marketing campaign.

3 Key Components of Incrementality Testing

To conduct incrementality testing effectively, there are several important elements to keep in mind:

1. Holdouts: Brands can gain invaluable insights by comparing holdouts to individuals who were exposed to a particular marketing effort. Holdouts can be based on location, products, or even specific customer segments. For instance, if you’re testing the impact of paid social media, you might exclude a particular region and then compare sales performance between that region and others where social media ads are running.
2. Non-Promotional Testing: Incrementality testing works best when the market is relatively quiet. During promotional periods, there is too much noise that can skew results, such as increased traffic due to discounts or holiday sales. Conducting tests during non-promotional periods helps minimize these external influences, leading to cleaner data and more reliable results.
3. Single Media Channel Testing: Isolating specific channels during your testing is also a good idea. If you run a test while multiple channels are actively driving traffic, it can be challenging to discern which channel is actually contributing to incremental sales. By focusing on one media channel at a time, you can better understand its individual impact on sales.

Why Incrementality Testing is More Reliable

The primary advantage of incrementality testing while utilizing attribution is its ability to provide a clearer view of marketing effectiveness. Since incrementality testing uses control groups, it measures the actual impact of a marketing action, independent of customer journey touchpoints or external factors.

Consider this example: you run an incrementality test on display ads, showing the ads to one segment of your audience while withholding them from a control group. Upon comparison, you discover a 20% lift in conversions within the exposed group. This 20% represents the real, incremental contribution of your display ads. Unlike attribution, which might assign varying credit based on touchpoints, incrementality testing shows a straightforward result: without the ads, those conversions wouldn’t have happened.

Another benefit is that incrementality testing is less vulnerable to the data biases that often plague attribution models. Attribution can sometimes skew in favor of certain channels due to inherent model biases (like last-click attribution favoring the final interaction). With its holdouts and real-world comparisons, incrementality testing avoids this pitfall, providing an apples-to-apples comparison of different marketing efforts.

How to Implement Incrementality Testing

Below we explain exactly how to get started with incrementality testing:

1. Define Your Objective: Decide which specific channel or marketing effort you want to test. This can be paid search, display ads, email marketing, or any other medium you use.
2. Select Your Holdout Group: Determine who you will exclude from the marketing exposure. Your holdout group should be representative of your overall audience to ensure accurate test results. Remember, holdouts can be based on various factors, including geography, product categories, or demographic segments.
3. Run the Test: Launch your marketing activity for the test group while keeping your holdout group unexposed. Be mindful of the duration; the test should run long enough to collect meaningful data but not so long that external factors influence results.
4. Analyze Your Results: Compare performance metrics between the test and holdout groups. Look for differences in sales lift, conversion rates, or other KPIs. This comparison will give you a clear indication of the incremental contribution from your marketing activity.
5. Refine and Repeat: Incrementality testing is not a one-time event. Continuously refine your approach and test different channels or strategies to maximize marketing effectiveness.

It’s Time to Add Incrementality to Attribution

While attribution has served as a valuable tool for understanding customer journeys, the reality is that incrementality testing provides deeper insights into marketing impact. Its focus on real-world comparisons and measuring the direct impact of each marketing effort provides a more accurate picture of what’s working and what’s not.

As technology advances, marketers must be willing to adapt their measurement strategies to stay ahead. If you’re still relying solely on attribution models to measure your marketing effectiveness, consider adding incrementality testing to your toolkit. With it, you’ll be able to understand the impacts of your tactics, channels, audience, etc. to make smarter decisions when setting up your attribution model. In addition, you’ll gain deep insight into your marketing performance and unlock the potential to inform, fine-tune, and improve your attribution model.