AI model benchmarking isn’t currently very useful for marketing teams.

We’re making it useful.

Meet Definition’s AI Leaderboard.

Want to know which model’s best to help you create a particular type of marketing content?

Now, you can.

See the results

Why have we launched the AI Leaderboard?

Benchmarking means comparing how good different AI models are at doing things.

The thing is, there are benchmarks out there, but they’re not designed to help marketing teams. Anthropic (the creators of the Claude model) tests AI on things like:

  • Graduate level reasoning
  • Coding
  • Multilingual math
  • Math problem-solving

But that’s not much use to marketers. Does a model’s ability to do multilingual math help you decide whether it’s the best AI to write the emails for your next campaign?

Marketers need a benchmark that’s relevant to what they do. One that shows which AI model is best suited to help with the marketing content they need to create.

See the AI Leaderboard

How does it work?

Note to reader: This first AI Leaderboard covers four types of written marketing content. We’ll be expanding to include other content types like video, design, and research, as well as updating scores as models evolve and are released.

The methodology was built by our AI and marketing teams and advisory board. Put simply, we ask AI models to create marketing content, and our panel of experts ranks them.

1

We asked seven AI models to write some common types of marketing copy – a blog, a marketing email, an organic LinkedIn post and an opinion-led article.

The initial models we’ve tested are:

  • OpenAI’s
    • GPT-4 T
    • GPT-4o
  • Anthropic’s
    • Claude 3 Sonnet
    • Claude 3.5 Sonnet
  • Mistral’s
    • Large 2
    • NeMo
  • Google’s
    • Gemini 1.5 Flash

Results for additional models – such as Claude 3.5 Haiku and Gemini 1.5 Pro – will be added on an ongoing basis.

2

Our panel of language, PR and social experts (meet them below) scored the writing against a set of criteria. Things like clarity of language, structure, persuasiveness and creativity.

3

We used the scores to rank each model and each type of marketing content, by the individual criteria (so you can see the best model if you’re most interested in depth and detail, for example) and by overall score (so you can see the best model generally for a given type of content).

Here's the full methodology

The panel

 

Nick Padmore, Head of Language

Nick Padmore

Padders heads up the plucky band of writer-consultants we call our Language team. He’s been writing for businesses for nearly 20 years, working with the likes of Monzo, Specsavers and Disney+. Along with his team, he leads Definition’s projects on tone of voice, naming, brand storytelling and, unsurprisingly, writing – as well as working with our AI team to craft prompts that actually work. He studied English Language at university, and is a co-founder of greeting card company Deadpan cards, which you’ll probably find in a shop near you, and which still somehow manages to remain resolutely unprofitable.

 

Hannah Moffatt, Creative Director

Hannah Moffatt

Hannah Moffatt is one of our Creative Directors, a children’s author and a word nerd through and through. After studying French and Spanish at Cambridge University and Creative Advertising in Falmouth she’s spent the last 16 years helping global businesses find their brand voice and training them to use it. She’s written everything from charity appeals to whitepapers. And when Hannah’s not writing for clients, she’s writing for children too. Her debut, SMALL! was a Sunday Times Children’s Book of the Week and shortlisted for the 2023 Waterstones Children’s Book Prize.

 

Nick Banks, Senior Writer

Nick Banks

Nick writes for our language and PR teams. From tone of voice and naming to thought leadership for niche industry publications and nationals, he’s done it all. He’s also a key part of our AI team, specialising in prompt engineering and AI training for big-name brands like PepsiCo. Nick studied English Literature and History at Goldsmiths College, University of London, before earning an NCT Seniority Certificate in Journalism.

 

Tom Pallot, Head of Marketing

Tom Pallot

Tom leads our marketing team. He’s spent the last decade building brand awareness and generating leads (with a focus on PR, SEO and content) for B2B brands. He’s certified in data journalism by Google, generative AI by Microsoft and prompt engineering by Vanderbilt University.

 

Louise Watson-Dowell, Head of Digital PR and Social Media

Louise Watson Dowell

Lou’s been making digital PR and social media magic for ten years. She’s the boss of our digital PR and social team. Lou cut her teeth studying film and photography at the University of Leeds. That’s where she first learned to tell digital stories that grab people’s attention. Before she stepped into the comms world she spent a while writing scripts and assistant producing on shoots. Lou takes new and established brands to market with clever content and channel strategies and runs PR campaigns that make people sit up and take notice. And she’s done it for some massive names like GE, EY, KPMG and Mastercard.

 

Isabel Pitts, Social Media and Content Consultant

Isabel Pitts

With 12 years under her belt in charity, sustainability, tech, and education, Izzy’s our go-to for all things social. She studied broadcast journalism at the University of Huddersfield, where she wrote a dissertation on the power of social media and user-generated content in journalism – clearly a sign of things to come.

Since then, she’s produced for major TV stations and led social campaigns that have generated millions in revenue. If it’s social, Izzy’s all over it: organic and paid strategies, advertising campaigns, crisis comms, community management, personal branding – you name it. She proves daily that emotion-driven content with a strong narrative is the best way to genuinely engage an audience.

 

Katie Chodosh, Head of Media Relations

Katie Chodosh

Katie’s our Head of Media Relations. She’s worked in PR and communications for more than 10 years. After studying speech and language therapy, she started her career working exclusively with cyber security companies (at Eskenzi PR) and has since stayed in B2B.

Katie loves working on CEO and company profile pitching because it gives her the opportunity to interview really interesting people. Over the course of her career, her approach has landed interviews for clients with BBC, BBC Click, Wired, The Next Web, The Times, The FT, and more. She also spends a lot of time working on the exact wording of survey questions, article pitches and press releases, which have landed with nationals and a whole host of trade publications over the years.

The advisory board

The two members of our AI Leaderboard advisory board:

  • Consult on the methodology, to ensure that the benchmark results are consistent, unbiased and useful.
  • Ratify the results.
  • Provide analysis and insight on the performance of the different models.

 

Neil Taylor, Chief of Brand

Neil Taylor

Neil looks after our team of clever researchers, strategists, writers, designers, film-makers, trainers, and CX and UX experts.

Since studying linguistics and French at the University of Cambridge he’s gone from coming up with names (his tombstone will say ‘He named Ocado’), to writing whole sentences, to defining tones of voice and whole brands. He wrote Brilliant Business Writing, and has helped everyone from Cabinet ministers to call-centre workers think about how they express their organisation’s brand. And, he co-founded Deadpan Cards with our Head of Language, Padders.

 

Luke Budka, AI Director

Luke Budka

Luke builds, experiments with, and shapes our AI solutions (including our award-nominated private AI environment). His impactful work for us and our clients led to him being named one of the UK’s top AI Innovators in 2024. He brings 16 years of linguistic, content and tech agency experience to the table. He’s just as comfortable speaking at conferences as he is judging tech awards (which he’s done for over a decade) and he helps our clients use AI to get stuff done.

See the marketer's AI Leaderboard