The private AI environment we launched in September 2023 has evolved quickly.

We started out building a secure AI platform our team could log in to and use in a safe and secure way, mitigating the danger of data being leaked to various AI vendors, desperate for new content to train their models on.

We started adding bits. Like a prompt repository featuring prompts specifically written for clients.

And then we began to introduce new AI models, because even in the early days we realised OpenAI’s GPT-[insert number here] wasn’t going to be best at everything.

We now have a combination of speech to text, large language and text to image models, from OpenAI, Google and Anthropic, all under one roof.

Last week though, things ramped up a notch.

We’ve connected our private environment to OpenAI’s Code Interpreter (data analysis) and File Search (file augmentation) tools and added Anthropic’s vision capability.

These updates are important.

Code Interpreter analyses data.

Could be SEO data from Google’s Search Console, could be PR data from the latest survey research, could be quantitative data generated by our customer experience team, you get it, the list goes on. We can now interrogate data using natural language. This enables all of our employees to do meaningful analysis, even if they’ve never done it before.

File Search enables us to augment the OpenAI models with data from our clients.

We can now create AI versions of every client within Definition AI. Why do we care? Large language models are famous for scraping terabytes of data from around the web. But ask them where the coffee’s kept in your office and no dice. ‘Ground’ them (not a coffee pun) in company specific data and suddenly it’s a different ballgame.

Now we can include the AI in client brainstorms, you can ask it how your client compares to every other company in its sector and areas they could improve upon. You can get the AI to write company specific marketing material based on the data its grounded in. You can use it to teach new team members about the client.

It’s not a big leap to see how this will form the foundations of autonomous agents – after all, all the steps required in an autonomous agent set up will likely be governed by a large language model.

The addition of vision means we can understand and analyse images in our private environment.

We do a lot of workshops where we manually transcribe loads of writing from whiteboards and post sticks and flip charts. Automating this will save us hundreds of hours. We have also started to use it to reverse engineer images we like to help us write better prompts for image generation going forward.

Here’s an example of this in action from an early version of the vision capability:

 

We chose Anthropic for this capability* because it scores higher than OpenAI on a few benchmarks – particularly the one we cared about when assessing which vendor to use: Document visual Q&A. Yes Gemini scored higher but Google is awkward to deal with in Europe atm.

*This is the benefit of having your own private AI environment – you get to pick and choose your models.

Sonnet vision benchmarks

Vision, combined with File Search also means we can analyse a whole host of documents in a variety of formats instantly e.g. RFPs, research reports, etc. Estimates suggest that up to 80% of a business’s data may be unstructured and Anthropic acknowledges this by stating its enterprise customers are excited about the product because some of them “…have up to 50% of their knowledge bases encoded in various formats such as PDFs, flowcharts, or presentation slides”.

And these are just a few of the ways we’re using these features.

Truth be told, every day our colleagues come up with new and novel use cases that continue to surprise me.

What’s incredible, is how much we’ve been able to do and how quickly we’ve been able to do it.

In the words of Leopold Aschenbrenner, former OpenAI superalignment staffer and co-author of ‘Weak-to-strong generalization: eliciting strong capabilities with weak supervision’ (a paper on manging super AI with rudimentary AI):

“One year since GPT-4 release. Hope you all enjoyed some time to relax; it’ll have been the slowest 12 months of AI progress for quite some time to come.”

 

Written by Luke Budka, AI Director at Definition.

Drop us a line for a chat about AI training, prompting or building.