Scope

How to do a random offer test in Retention?

Summary

You can start out by creating a random offer test in Retention. We determine the number of experiences that can be tested based on the monthly cancel volume and desired time to achieve statistical significance. For more on how to determine the number of Offers/Pages you can test, visit our MDE Calculator.

1. Create your Default Page as your Holdout
2. Create 3 to 4 Different Offer Types
3. Create 3 to 4 Matching Pages and Assign Offers
4. Optional: Create Loss Aversion Offer
5. Configure your Targeting Tree to set up the Test

Solution

Step 1: Create your Default Page as your Holdout

When creating your Holdout page you will want to make sure that you are creating a generic page that generally represents your existing in-house cancel flow. This will help us monitor performance as the tests evolve and ensure we are not biased in the initial test control with too many improvements. You should take this time to configure your cancel reasons when setting up your Holdout as you will want each Page to have the same set of reasons for this first test. 

Step 2: Create 3 to 4 different Offer Types

Once you have a Holdout page, it is time to navigate to the Retention Offer Library and create three to four different Offer Types. 

At Retention, we break Offers down into categories and types based on what will best mitigate the customer's problem or solve their need. We have 6 categories and several types within each category. Generally, we recommend 3-4 different categories for an initial random offer test (For example, a mix of one Discount, one Pause, and one Plan Change). However, when circumstances require it is ok to have 2 offers within the same category in your test, so long as at least two categories are represented, for example, a mix of one Skip, one Pause Plan, one Discount, and one Plan Change. 

Step 3: Create 3 to 4 Matching Pages and Assign Offers

After creating your Offers, you should navigate back to the Pages tab. Here we will copy the Holdout experience and assign the relevant offer to it by editing its placement. 

Your placement tab for that page should look something like the image above with the Discount offer applying to every reason. When naming pages it is good to associate them with your test as this will help in reporting down the road. Here we named this Page the Discount Offer Test, which is easy to identify and connect to the Offer that lies within it. 

Your Pages tab will look something like the image above.

(Optional) Step 4: Create Loss Aversion Offer

One of the immediate opportunities to generate a lift in Retention comes by layering two offers within a single page. This allows you to present an initial offer (say a Skip) in the Loss Aversion card that everyone on the page sees. Then when they select a reason, they will be presented with a second often stronger offer such as a Discount of full Pause. Several of our customers leverage this by applying a Loss Aversion offer right from the beginning of their initial test. 

To create a Loss Aversion Offer, navigate to the Offer Library to create a new Loss Aversion Offer. Select the Goal you would like, in this case, we will choose a skip. Under the layout section scroll to the bottom and select the Loss Aversion, CTA Card. 

Once you have created this card, you will place it in the Card B slot for all of your Pages except for the holdout. Setting this test up with a Loss Aversion card on each Page beside the holdout allows us to isolate the impact of the Loss Aversion Offer against the Holdout, without distorting the Random Offer test by showing the same Loss Aversion Offer to every customer that sees a Random Offer.

Step 5: Configure your Targeting Tree to set up the Test

Once you have created all of your offers & pages, it is time to go to the Targeting tab and configure your test. 

We will start by adding the Holdout with 5% of our overall cancel volume and then will add our three Pages with each offer in the Remaining traffic section. 

You want to ensure that each of the three Pages is distributed evenly, which can easily be done by selecting 'Redistribute evenly'. Once this is configured you are ready to go ahead and publish your Targeting by selecting 'Publish all' in the top right corner. 

Once your test is live, please notify your CSM and we will set up tracking on our end to monitor it. If you have any further questions about how to set up a test in Chargebee Retention, please contact [email protected].