The challenge
To promote the latest book of Dan Brown, Bol.com, the largest media store of The Netherlands, sent out an email campaign. The challenge was to maximize the sales of the book by adapting the content of the email to the recipient's preferences.
The optimization campaign
Bol.com created different header images with different calls to action and wondered which header would maximize the response rate of the campaign. Would this be an ordinary promotional offer (free shipping) or would future readers be triggered by a feeling of exclusivity (be the first one to read the book)?
Different headers were tested:
8Seconds Optimizer showed different header images to the recipients that first opened the email and measured statistically and in real time which header was most successful in terms of conversion. As soon as the image that generated the most click-through was found, 8Seconds Optimizer showed this optimal email to all next recipients.
Results
And the winner was (with an increase in conversion of +59%) .....

Conclusion
- Although the offer and the image were exactly the same, the fact that the highlighted text in the blue banner was different had a massive impact on the click-through rate of the recipients.
- Highlighting FREE SHIPPING appeared to be more powerful than giving the recipient a feeling of exclusivity (READ ON THE DAY OF PUBLICATION or BE THE FIRST TO HAVE THE BOOK AT HOME).
- Testing offers increases CTR (in this case by +59%).
- Real time multivariate testing of email campaigns allows adapting the content to the recipients' preferences and maximizes the response rate of email campaigns.
Comments
By Marnik D'Hoore - bSeen 14/10/09 (9 months ago)
re: Bol.com tested how to sell the latest Dan Brown and shares the results
Dat zijn mooie inzichten. We hebben laatst hetzelfde gedaan voor een site die gifts verkoopt (www.cowparade.be), echter -voorlopig- zonder MVT testing. Binnen paar weken weten we het verschil.
By Jordy van der Heide 20/10/09 (9 months ago)
re: Bol.com tested how to sell the latest Dan Brown and shares the results
I'm curious how the benchmark of this offer is determined (an +59% increase of CTR against what offer?). And is it an coincidence that the increase of CTR is the same percentage as the conversion?
By Maite Maes - Marketing Manager 20/10/09 (9 months ago)
re: Bol.com tested how to sell the latest Dan Brown and shares the results
Hi, testing the variables is done against an accepted standard. This standard of comparison in named the control. With 8Seconds Optimizer, you can choose this control. In this case, the control was the offer 'READ ON THE DAY OF PUBLICATION'. The increase in CTR (59%) is compared to this offer.
'Conversion' was in this email campaign defined as 'CTR'. So the percentages are indeed the same!
By Jordy van der Heide 20/10/09 (9 months ago)
re: Bol.com tested how to sell the latest Dan Brown and shares the results
Thank you for the explanation!
Did you (besides the increase in CTR) measure an increase in books that were actually sold? CTR is an important KPI, but an increase in sales (what I would define as an conversion) would be the really interesting part!
By Maite Maes - Marketing Manager 20/10/09 (9 months ago)
re: Bol.com tested how to sell the latest Dan Brown and shares the results
Of course, this was also measured by our client Bol.com, but this remains confidential information. So, unfortunately, we can't share this.
By Peter 23/10/09 (9 months ago)
re: Bol.com tested how to sell the latest Dan Brown and shares the results
I really like your blog and i respect your work. I'll be a frequent visitor.
By Damian Davila Rojas 28/10/09 (9 months ago)
re: Bol.com tested how to sell the latest Dan Brown and shares the results
Great post! A/B testing in real time that sounds amazing.
I was wondering if you could provide more details about the bounce rate.
Which of the 3 ads had the highest bounce rate?
By Maite Maes - Marketing Manager 29/10/09 (9 months ago)
re: Bol.com tested how to sell the latest Dan Brown and shares the results
Hi, thanks for the comment. 8Seconds measures the open rate, the unique views and the click-through rate. The bounce rate is measured by the email broadcasting platform of the client. So, we don't have information regarding the bounce rate.
By Kouba 01/11/09 (8 months ago)
re: Bol.com tested how to sell the latest Dan Brown and shares the results
Interesting and informative. But will you write about this one more?
By Peter 04/11/09 (8 months ago)
re: Bol.com tested how to sell the latest Dan Brown and shares the results
I read a few topics. I respect your work and added blog to favorites.
By Dave Westerhof 04/11/09 (8 months ago)
re: Bol.com tested how to sell the latest Dan Brown and shares the results
The biggest question of course is whether this has true consequences for the actual conversion (in my view, buying/viewer) and delivers ultimately sales. This is a hard one, and I can imagine why 8 seconds uses conversion interchangibly with clickthrough, but this is how I evaluated my campaign. Since every industry or company has it's more or less pre-fixed conversion-rate, we can derive an eventual premium from increasing the clickthrough.
Imagine a funnel where a customer goes from opener to clicker to buyer. Averages are mostly known per business for every step, say for instance from opener to clicker is 50% and from clicker to buyer is 40%. By increasing the initial 50% to 51%, we can derive an increase in profit at the end, ceteris paribus (40%). This is how I derived the actual increase in profit during the tests.
Usage of this program is recomended due to its power to deliver new insights, on a real-time basis. Therefore, no more arguments or discussions to choose what's the better option, just let the customer decide what's evaluated best!
By Software 07/11/09 (8 months ago)
re: Bol.com tested how to sell the latest Dan Brown and shares the results
Another great post.
Thank you for the information, Its good to see such quality posts.
Im subscribing to your blog.
Keep them comming.
http://www.rain-clouds.com" rel="nofollow">Software for craigslist