We Put ChatGPT To The Test – And Here’s What We Found

Copywriting tools – there’s a ton of them. Artificial intelligence chatbots – there’s quite a few of them too. But what happens when the two come together? Open AI’s ChatGPT, that’s what. 

We’ve seen plenty of these tools over the years, most simply not making the cut when it comes to reliability and usability. Yet ChatGPT has made experts in both tech and marketing stand to attention. Described as a ‘trained model that specialises in dialogue’, it uses human feedback to constantly improve when responding to questions and providing information through a conversational interface. Its applications are huge – including its potential to write marketing copy. 

Naturally, we wanted to make our own judgement. So, we had a little play around – testing the AI platform on four written formats that MYC delivers day in, day out…

Social posts

It’s worth mentioning just how promptly the tool responds. Once we made our request, ChatGPT produced this social post in under a minute. 

Request:

Result: 

On the surface, it’s more than half decent too. It flows well, and it even features a CTA and hashtags. But there are a few snags. Firstly, it’s in US English (although you could always refine your request and ask for it in another language!). Most importantly, though, the post itself lacks creativity – while it captures the tone of a social post and incorporates some common phrases, we wouldn’t call this the finished article by any means. 

This could easily be a post from just about any law firm out there – there’s nothing that sets it apart in terms of a tone of voice or even the mention of a few USPs. Of course, you can devote a significant amount of time to feeding an AI model with information on your services as well as examples of your way of writing (simply telling ChatGPT you want to sound ‘friendly’, ‘disruptive’ or ‘straight-talking’ won’t be enough – certain words are subjective, after all). But if you’re only looking to use a basic tool, you’ll struggle to capture your brand persona within the copy. 

To craft a social post that will actually feel on brand and engage a particular audience, you’re still going to need an expert to cast an eye over it before you press that publish button. 

Emails

So far, so good. But how does it handle a slightly meatier piece of content? We decided to task ChatGPT with an email for a B2B training company. 

Request:

Result: 

If you remove the rather formal opener, there’s definitely some valuable content within this result. The sentence structure and vocabulary is surprisingly good in places, yet there’s a lot of repetition too – the word ‘business’ was used six times, where we might have expected to see words like ‘company’ or ‘organisation’.

What’s more, while the copy includes some logical points, it’s not exactly compelling email copywriting. It lacks the basic components of a direct response email – as well as any specific points that might relate to the reader’s target market or how the organisation can add value. To get a result that helps overcome the audience’s sales objections or persuades them to act quickly, you again would need to elaborate on your brief to the AI tool. 

Overall, this is a good foundation to build better copy on, but it would need more work before we’d approve it for our clients! 

Blogs

Next, we tested ChatGPT to see how capable it is of explaining a topic that might sit in a blog post. And sure enough, it delivered. 

Request: 

Result: 

On the face of it, this content is a great starting point for a writer. In terms of answering the question and providing detailed information, the chatbot has hit the nail on the head. And while we didn’t specify whether this article would be for SEO purposes, it has used the term ‘cloud-based accounting’ frequently enough that it would have a good shot at ranking (providing hundreds of other firms don’t use a similar output from ChatGPT!). 

However, just like the previous results, it still needs a creative flair to improve the flow and engage the reader. The copy is repetitive and lacks any means to aid scannability – things like bullet points and subheaders, which help people quickly digest a longer piece of content. 

Again, the copy also feels very generic. It wouldn’t hurt to throw a couple of examples into the mix, or add a few industry-specific statistics. It could even go a few steps further, looking at how cloud-based accounting differs for freelancers and large organisations. There’s no emotion either – losing data in a fire or flood can be devastating for a business.

It’s no surprise that tone of voice comes into play here too. Your blog is one of the best places to convey your brand’s message and image more fully – which you can only do so much using an AI writing tool. To craft a blog post that will genuinely inform and engage your audience, you still need a little magic from a wordsmith.

Case studies 

This is an area of copywriting where we were anticipating ChatGPT to fall short. But we were amazed by what it produced given the little information it had to hand. Let’s take a look… 

Request: 

Result: 

What struck us straight away is how ChatGPT has carved out a structure for the case study that many writers follow. But when it comes to outlining the challenges, solutions and results, the AI tool can only do this superficially. You could keep asking it to add in the specifics, but this is likely to take just as long (if not longer) as writing the piece yourself.

And, without wanting to state the obvious, those who create case studies on a daily basis know how to make one work harder for you – adding in extras such as testimonials that can nudge a prospect over the line. On top of this, they ask the right questions to the client in the first place, extracting useful and impactful information that can be brought into this type of content. Ultimately, ChatGPT isn’t going to make case studies any less hands-on if you want your copy to take a people-first approach. 

The MYC verdict 

ChatGPT is a valuable addition to any writer’s toolkit. It’s highly intelligent and, as we’ve seen, capable of generating high-quality text quickly and efficiently. However, like any tool, it requires some fine-tuning and can’t fully replace the expertise and creativity of a trained professional.

Okay, okay… we got ChatGPT to help us write that too! But it has a point. It’s clear that Open AI has managed to significantly level up AI writing tools. But giving it the correct prompts and refining the results remains a skill for humans to hone. In the right hands, each of the examples above could be turned into compelling copy that makes a real impact – even if it’s simply a case of giving it a good edit.

Looking for someone to take a creative and unique approach to your content marketing? With vast experience across a huge range of sectors, our wordsmiths are always going the extra mile to make copy pack a punch. Let’s talk.