AI-assisted copywriting more effective than human-penned ads, study finds

The experimental study found that AI-generated copy that is lightly edited by humans was 26% more effective at increasing click-through rate (CTR) than human copywriters on their own, unassisted by AI.

Pencil co-founder Will Hanschell says that the experiment proves the efficacy of AI-assisted copywriting for advertisers.

“It’s a relatively small study structured as an A/B test, not a massive piece of research, but it was highly practical and it was a real campaign,” he says.

Brandtech Group, the parent company behind agencies Oliver and Jellyfish, acquired Pencil last year. The firm uses a combination of generative AI tools from OpenAI and Stability to create copy, images and video content for use in performance channels such as Facebook.

The experiment was carried out in the fourth quarter of last year with a major beauty client on Facebook dynamic product ads (DPA) and targeting Singaporean consumers. “The question was, how big a difference would it make if you change the copy?” says Hanschell.

He says his team tested four categories of copy with the ads: copy written solely by the brand’s in-house team, copy generated by GPT-4, AI-generated copy that was edited for style and grammar by the copywriting team and ad copy written by the brand’s team in response to AI-generated copy.

“The control box was whatever they had before, not expertly crafted or tested within an inch of its life; it was Facebook ad copy. Then, we had the AI generate some options.”

Each category included between five to seven different ads, representing up to $20,000 in media spend, while each ad contained a single sentence of copy. The experiment was only run on English-language ads. More complicated tests could be run on different types of digital ad inventory, he adds.

Copy “inspired by AI” saw an 11% uplift in CTR, AI-only written copy resulted in a 19% increase, while human-edited AI copy saw a 26% increase.

Though the test suggests AI assistance could be more effective than humans left to their own devices, Hanschell suggests the results should not terrify career copywriters.

“We want people to feel like AI was something they could do, something they could hold in their hand, and that wasn’t threatening – that isn’t preparing to replace you. To be able to see, in this case, that working with it, as opposed to letting it do the job, was *chef’s kiss*.”

The fact that a form of human collaboration with an AI tool saw the best results proved AI was not to be feared, he says.

“It’s really important for people who are writing copy to feel safe enough with the technology to use it, to learn it, to learn its capabilities and limitations so that, when it becomes extremely good, they are taken along for the ride on that. I think there’s a danger that folks distance from it – and, before they know it, it’ll be really good and you’ve missed an opportunity. You have to want it to help you.”