The best and wurst of pre-testing: why marketers need to visit my sausage factory

They all shared a mixture of pros and cons for pre-testing. I’m here to tell you that as a bit of a pre-tester myself, I agree with ALL of them. I’ve helped brands and agencies pre-test work for years at System1. And that was after working at top FMCGs who saw tremendous success with pre-testing work. I have seen it all. And let me tell you. There’s a good, a bad, and an ugly side to pre-testing.

So to understand what works, we all need to get on our white overalls and visit the sausage factory.

See how the sausage is made before eating it

A lot of the arguments against pre-testing are because people don’t understand there are many different types.

Many arguments attack the archaic idea of the dodgy printed storyboard being shown to a disinterested focus group in a carpeted basement office block. The complaints vary: “Too small a group,” “useless stim,” and “unrealistic environment.”

As Riston pointed out, with modern tech, we can now easily create early-stage stims that are rather accurate to the final ad. And now, most testing is now done online, at scale, on the devices people consume ads on. But still, to pre-test effectively, you really need to understand how the sausage is made before eating it.

Go through the process your consumer will, which means taking a survey hunched over the toilet with your pants around your ankles (that is how most ads are actually consumed, and you know it).

Once you understand the ‘how’ behind the metrics you are going to be placing your million-pound bets on, create research objectives from your marketing strategy. Don’t red-light, green-light work.

Have specific outcomes that you need to see. This might even mean we all agree failing on some of the metrics in the testing is OK. That’s fine, just align with your team and agency partners first.

Every marketer is a food critic, they’ve eaten too many sausages

My greatest fear in life is being “normal.”

Something irks me about it. Maybe it’s the small town I came from where regression to the mean was encouraged, or maybe I caught Gen-Z fever from TikTok. Either way – I’m blessed. I thought being queer made me rather different, but, boy-oh-boy, being a marketer makes me a FREAK.

There’s brilliant research out there showing that marketers don’t really watch TV, spend too much time on their phones (mostly using LinkedIn four times more than consumers), and are far more trusting. They are, thankfully, also totally obsessed with ads.

I tested a campaign with some Normos (consumers) and with some Marketers to see if this differences held water.

Not only did the marketers care far too much about the ad, but they felt significantly different emotionally, and they spotted the brand easier and earlier compared to your average sausage muncher. They got far too caught up in the story too. Marketers changed their emotions twice as much compared with Normos during the ad.

Marketers who still think their opinion of an ad matters are the issue, not pre-testers.

It’s like giving Gordon Ramsay a plate of frozen Wall’s sausages. Of course, he’s going to tell you where to stick them.

But I personally, a normal sausage consumer, love a midnight-Wall’s after too many Cosmos in Soho’s most questionable establishments. It’s a guilty pleasure, which is also the name of the questionable establishment.

Stop telling me creative agencies hate the sausage factory

My weekly ‘The Marketing Scientists’ cartoon with the brilliant Dhananjay Khanna is below. I caught myself being toxic as hell while thinking of it yesterday.

It was going to be a ‘creative’ with a fag hanging out his mouth. Ignoring that “82% of consumers think the ad was boring” and “52% thought it was for the wrong brand” with a swift “They just don’t get the creative leap” line. But then I remembered most creatives I’ve met love creative research and testing.

I found this out when I used to run System1’s partnerships with European creative agencies. I thought I’d get to Chancery Lane and be pelted with rotten veg but that narrative did creative agencies an injustice. A lot of them actually wanted to bring the consumer into the room. They know the value of a sausage-tasting session.

What is true though, is some marketers and creatives get a bit swept up in trying to pass the taste-test.

It’s obvious when we fall into this trap.

We need to focus on our marketing strategy and test it. We can’t focus on “passing testing” and then fitting the marketing strategy to it. Or, as above, set research objectives for the brand so you aren’t trying to “pass a test.”

Don’t send a chipolata to do a Cumberland job

I agree that creative research can encourage a “paint by numbers” approach in the wrong sausage fingers.

And it’s why, despite there being literally years of research from various sources that dogs in advertising makes them more effective, I wouldn’t tell every client to “put a dog in the ad”.

All research has to take into account the usual marketing rules we play by. And how effective marketing works. And your brand’s strategy. A water bottle brand should not throw in a pink-bow-wearing Shih Tzu into their campaign out of nowhere. That’s mincing madness!

But maybe your brand is in a very boring insurance category. That lacks consistency, distinctiveness, and humanity. Maybe you choose a dog as your mascot, and you consistently use it over many years. And that dog becomes a fluent device that you deliver all yourcampaigns through. Basically, exactly what Vitality did – a marketing success story. Now that’s what I call a hot dog.

Not all sausages are equal

Finally, let’s address the weird elephant in the room.

There is a good way and a bad way to pre-test. Some people have clearly been on the receiving end of a rotten sausage and the resulting mess that follows. I understand your pain.

If your research is too system two, you’re going to struggle predicting real-world effects. Just look at all the work from the late great Daniel Kahneman, author of Thinking Fast and Slow to understand more about that.

System 2-based pre-testing involves you strapping a consumer to a chair (in true James Bond, Casino Royale style) and grilling them to within an inch of their life about a bunch of stuff they’d have never even considered. “Did you understand that this sausage is made from 54% British pork?” in the intonation of someone asking for the location of the “rebel base.”

System 1-based approaches, as we adopt at System 1 (hence the name), leave the Clockwork Orange act at the door and ask the consumer, a representative target audience, to watch the ad alone in their spare time. We measure how they feel. They share what comes to mind and what brand they thought the ad was for. We interpret the data to predict real-world effects. And it works.

As Ritson shared in his article, Sky uses our testing and it almost perfectly fits with its VERY fancy econometric modeling. And we’ve even shown (with the IPA) that if you know your extra share of voice and our star rating, you can predict about half of the advertising effects.

So, don’t get swept up in blanket statements about pre-testing. First, ask: “What do you mean by pre-testing?”. We must make more of an effort to understand the research we use. And I want to personally help. This is an open call. Any marketer that wants to make more effective advertising by using pre-testing properly. Please, come and visit my sausage factory.

More from Andrew:

• Are ‘fake ads’ helping or hindering the advertising industry?

• It was adland’s worst Super Bowl in years, but everyone is too afraid to tell you

• 7 out of 10 OOH ads don’t work. Here’s why and how to fix them