Recently at the CohereOne Marketing Summit during the pre-conference intensive, we reviewed direct mail testing ideas, how to properly set them up, and how to accurately evaluate the results. During a portion of the presentation revolving around how to properly read the results of a test, I noticed many people in the audience looking wide-eyed and bobbing their heads up and down like a light went off. With this type of reaction, it was clear that others, who were not in the room, could benefit from the same discussion.
The first important discussion in the presentation was the set up of the test audiences, so that on the back end a definitive comparison can be made to evaluate the success of the test.
- The most important thing to keep in mind when setting up a test is to make it clean, so that you are only accounting for the change of one variable between two audiences.
Examples of such variables might be: postal mailing vs. no mailing (holdout), email contact vs. no email contact, 20% off offer vs. 10% off offer, and the list goes on and on. Get creative with your testing ideas! This is the magic and “art” of direct marketing. But remember, in each case, only one variable should change and the questions to ask yourself are: what is the overall performance of each audience? And what is the difference between those two audiences?
The next discussion topic in the presentation was how to properly evaluate the results at the end of the test period.
- In order to get an accurate read of the performance of each audience, it’s imperative to run a “matchback” on each audience to identify all orders/sales associated with that audience.
It is not sufficient to rely on internal reporting on just the promotion code that was captured during the order process. As an example in the analysis below the mailed audience was being evaluated by the orders and sales that were captured with the promotion code that was printed on the piece. Clearly the mailed audience would be the loser in this test if only recording the captured codes.
In actuality, after running a matchback on the mailed group and applying all orders and sales that were placed by that audience, it changed the overall results of the test as can be seen below.
The reason for the large difference is the redemption rate (also referred to as breakage) is generally only in the 50% range for those customers driven by an offer. In this case, the redemption rate was closer to 25% which boosted the results of the mailed file even more.
This then dovetails into how to properly calculate the results of the test and the total performance of the two audiences.
- It is imperative to calculate each audience down to the contribution level and not evaluate solely on the traditional KPI’s of response rate and dollar per book.
- It is also imperative to only apply the actual cost of goods experienced for each audience.
The example below is the evaluation of post-matchback data and applying the marketing cost for the mailed group in addition to the discount that was offered, in this case 20% off, applied to all orders received from the mailed audience. If evaluating with a full discount burden, then the mailed audience clearly is the less productive of the two.
To correctly evaluate the contribution by accounting for true cost of goods realized because of the lower redemption rate (25%), changes the overall results as can be seen below.
Identifying the actual redemption rate of an offer is not only important in accurately evaluating the results of a test, but it also offers flexibility in creating offers for your customers or prospects. If you know that if you put out an offer of 20% but you know you will only take a hit of 10% of less to margin, it opens up the door for creating/testing different levels of offers which can mitigate the loss of margin but maximize performance.
At the end of the day, the most important aspects of testing are:
- Set up separate audiences where only one variable is changing
- Run matchback to identify total orders and sales placed by each audience
- Calculate each audience to the contribution level using true COGS accounting for redemption
If you want any help setting up a test with an experienced circulation expert, feel free to contact me at tseaton@cohereone.com