GPT-based meeting summaries, such as those based on OpenAI’s Large Language Models (LLMs), cost about 1.6p (GBP) to securely generate on Microsoft Azure. However, the GPT summaries must be thoroughly verified by a human (for example, an adviser or a paraplanner) before taking any client action based on their content.
Kit Ruparel, Recordsure’s CTO, demystifies this new wave of capabilities and allows us to peer through the clouds of hype to understand the practicalities, limitations and regulations around using such generative AI technology. Recordsure has now released a complete AI meeting note generator as an open-source for organisations that want to experiment with generating their own conversation summaries before committing to a vendor selection or to using GPT-based meeting notes as part of their business practices.
The innovation
In 2011, Nick D’Aloisio became the youngest ever recipient of Venture Capital funding for an app that could turn verbose reams of text into concise summaries. Less than 2 years later, he sold his business, Summly, to Yahoo for an estimated $30m – all before his 18th birthday.
Later, companies such as Otter.ai and Notiv (later acquired by Dubber in 2021) saw the potential to use generative AI to summarise Microsoft Teams meeting recordings into a concise set of notes and actions – a feature that Microsoft Teams now bundles with its premium edition.
What luxury: to sit back and talk without having to take meeting notes – and all thanks to companies with clever data-scientists and computational linguists who had the skills and resources to boldly build products that could make at least one aspect of the increasingly frequent online meetings less tedious.
In 2020, OpenAI first announced API access to their early GPT LLM AI models – unnoticed by the market until late 2022 when ChatGPT was released with accompanying GPT improvements – which was quickly followed by competing APIs and models from the likes of Google, Anthropic, Mistral, Baidu and many others.
The game was forever changed.
Suddenly, building tools such as AI meeting summarisers customised to your conversation needs didn’t need a proprietary model, painstakingly trained by expert data-scientists, using transcript labels carefully crafted over many years by teams of human annotators.
A junior developer with a couple of spare hours could find the ‘how-to’ code online, adapt it to fit an industry sector… and hey presto… a new industry was born. And a new wave of hype to go with it.
However, as with any new technology, excitement usually leads to many false promises before finally realising practical application. In other words, recognising Gartner’s ‘Hype Cycle’ registered trademark, the ‘Peak of Inflated Expectations’ has to go through the ‘Trough of Disillusionment’ long before ultimately reaching the ‘Plateau of Productivity’.
Responsible hype
In their latest (July 2023) Hype Cycle chart for Generative AI, Gartner places the technology right at the peak of the ‘Peak of Inflated Expectations’, and estimates five to ten years before it matures into the ‘Plateau of Productivity’ – and whilst it will be interesting to see how much their view has changed in the intervening year, it’s not a status quo nor a timeline quantum that most AI experts would vehemently disagree with.
But how many vendors, touting their GPT based wares, are honest about the pros and cons of the technology?
Hype Cycle Image Credit: By Olga Tarkovskiy – Own work, CC BY-SA 3.0
And how many times have you answered a call from a company with an AI-based meeting note summariser, and within their first breaths, they’d explain the limitations in the current state of the art, alongside the responsible use of the early-stage products they’re hyping?
Rarely, if ever.
We’re a responsible AI provider, and we talk to current and potential clients daily, taking the time to explain what the technology can achieve. Yet the experiences shared with us are of early trials of GPT-based AI products that led nowhere after six months – expensive and time-consuming experiments in bleeding-edge, oversold technology that fail to live up to the promises made. One bank was brave enough to admit that they’re experimenting with their third vendor – and, predictably, are on repeat.
This is no surprise to Recordsure. In an earlier article, I delved into the strengths and weaknesses of Generative AI, and when it comes to AI-generated meeting notes, the rules are the same as for any other application:
Yes, Generative AI is good at generating text – because that’s what it’s designed for.
No, today’s Generative AI is not good at finding or understanding content – and forcing it into such a use case will be error-prone.
To reliably find and categorise written content, whether from transcripts or documents, there is not yet a shortcut to painstakingly labelling large volumes of text and then having expert data scientists design and test deep neural networks to create a ‘Predictive AI’ product that can be authoritatively measured and qualified as to its accuracy.
Is there value to a GPT created meeting summary?
The industry desire for accurate AI meeting summaries is valid – and we know that the solution is ‘out there’. And whilst only partially accurate meeting summaries can be produced today, by Recordsure or by anyone else, we don’t dismiss the value of having one.
What we’re here to emphasise is the need for us all and the industry to recognise the limitations of GPT-created meeting summaries, be realistic about their utility, and urge adequate and responsible controls to be put in place around their use.
It’s essential to know that the GPT generated meeting summary, from any vendor, is not going to be accurate. It’s going to miss important stuff, it’s going to misinterpret some of what it finds, and it’s going to make stuff up (‘hallucinate’).
If you’re planning to hand the AI meeting note in draft form to a meeting participant for correction, say a financial adviser or a government inspector, who can then correct the meeting note from memory, refer back to the recording or transcript where memory falters, then that should ultimately give you a high quality outcome, and save you time (and money) as the notes don’t have to be created from scratch.
Yet if you’re going to pass the draft AI meeting note straight to the back-office, for an admin assistant or paraplanner who didn’t participate in the meeting to verify and correct, then it’s essential that they have access to the original meeting recording and transcript in order to generate an authoritative summary or to populate downstream documents or systems, such as a Fact Find, Practice Management System or CRM.
However, given that individual wasn’t present at the meeting, without additional tooling this can be inefficient, as they essentially need to review the entire conversation to check the meeting note. It would be nearly as fast to type it from scratch as they go.
Meeting notes with citations (references for each summary item to one or multiple ‘sources’ in the transcript or recording) can help in verifying the things the AI has found and how it has interpreted them. However they also give a false sense of security, as the way that GPT citations typically work is that they ask the question: “This is what I want to say, now find me the evidence to support it.” and often ignore conflicting information.
So, if a customer expresses they’d like to invest £50,000, then following a fruitful discussion on emergency funds they later state they’d like to invest £35,000, your GPT meeting summary may only point you randomly at one of those figures, and then perhaps only cite the one transcript source that supports its conclusion.
Citations of course also won’t tell you what things the AI has missed. For that, without additional tooling, you’re back to having to check the whole conversation record.
…and here is where traditional, data-scientist built, properly trained and tested, responsible, Predictive AI machine learning comes in. A tool like Recordsure’s ConversationAI will accurately and thematically mark up the original recording and transcript by topic, so you can rapidly find and check the stuff that is, or isn’t, in the AI meeting note; and safely ignore the rest of the conversation recording.
Will AI generated meeting notes improve?
Of course, although it’s unlikely you’ll see much in the way of improvements over the next few years from any purely Generative AI/GPT based solutions.
Instead, it will come from companies like Recordsure, who have taken the time to annotate many thousands of hours of recordings with millions of labels and who build, train and test deep learning models to find and interpret conversations the same way a human would. And keeping Generative AI in its place, simply turning the final AI findings into human-readable summary bullets.
How can I judge the quality of AI meeting notes today?
Having read this far – you might be interested in seeing some AI meeting notes for yourself, produced using your meetings and your conversations.
AI meeting notes, even given the current state of the art, have their place and can provide efficiencies if used correctly and responsibly.
The good news is that if you’re already a Recordsure client using our latest Capture solution, we’ll bundle GPT-based meeting notes for free along with your transcript.
If you are using our more advanced ConversationAI tools, then we’ll also be giving you higher quality outputs and the sophisticated tools to verify the meeting notes in the back-office more efficiently.
If you wish to stick with the very basic GPT notes for now, then there’s still good news: we’re pleased to open-source the code for generating meeting notes using the OpenAI models, securely within your own Microsoft Azure tenancy. Now you can try it out with some of your Microsoft Teams or Zoom transcripts, without any worries about data security or transcript redaction.
Remember that all GPT-based meeting note vendors are using the same technology (albeit they might choose to use one of the other GPT models), so this code we’re giving away for free is likely representative of the methods and outputs you get from anyone.
We’ve even made it easy for you to customise the prompts used to generate the summary and topics – essentially the same playground that any vendor would be playing in to tailor the meeting note to your needs.
And the cost of generating such an AI meeting note, securely, in your own Azure tenancy?
About 1.6p GBP (2¢ USD) for a 1-hour meeting.
So, there should be plenty of pennies left in your exploratory budget to contract a software developer for a few hours, if you need a hand with getting started.
Find the code on our GitHub repository here.
Other Generative AI Product Visions
And then there are vendors offering GPT notes with a few extra (future!) capabilities, such as automatically completing Fact Find documentation, populating PMS or CRM fields directly, or even drafting Suitability Letters and other client documentation for you. Given that everything the AI produces needs to be human-verified, look carefully at how any such ‘pre-population’ of documents or records fits into your qualification workflow.
And when it comes to automatically creating Suitability Letters (or SOAs for our Australian friends), don’t forget why you’re in business. There is a reason you have reams of specialist calculators, tools to examine the market for appropriate products, and that you hire suitably qualified and experienced financial advisers and paraplanners.
As one head of a wealth management organisation shared when approached by another vendor with a product vision that would, in the future, go straight from client meeting recording to suitability report preparation: “However good the tech might become, it’s just not feasible in practice. They’ve forgotten about advice. We might as well be providing robo-advice.”
Laws and regulation to know
Having covered the do’s and don’ts of AI meeting summaries, and the essential need to verify each and every line generated (and missed) from an accuracy perspective, we also impose on you a duty to give proper consideration to Responsible AI, and the emerging regulation and law in your territory.
In short, a GPT-created meeting note summary is an AI output, and you should thoroughly verify its accuracy, using a real human, before providing any client advice or taking any client decision based on it; with the more pertinent pieces of legislation being:
The UK GDPR ‘restricts you from making solely automated decisions … that have a legal or similarly significant effect on individuals’, whilst the UK government framework for AI regulation, adopted by the FCA, delves deeper into the need for public transparency and trustworthiness in AI, warning further about autonomous decisions, and mandating controls around (amongst other matters) safety, transparency and explainability, contestability and redress.
The EU AI Act, recently passed into European law, delves further into the need for human agency and oversight, categorising AI systems used to determine access to financial products (using credit-scoring as an example) as ‘high risk’, so requiring both the product vendor and the product user to register the use of such systems with the European Commission and to monitor and report on their controlled use of the technology, unless (in the case of responsibly used generative AI meeting note technology), the AI output does ‘not materially influence the decision-making’ and ‘that the AI system is intended to perform a task that is only preparatory to an [human] assessment’.
In Summary (pun intended)
AI-generated meeting summaries are an emerging technology, and whilst those products using GPT LLMs at their core are only going to be partially accurate, they still have utility if you have the appropriate human verification processes, coupled with adequate additional tooling to make their review and correction efficient.
Make sure you try before you buy, especially with recordings of true customer meetings previously unseen by the product vendor; or experiment with the open-source code provided by Recordsure to explore how they work and their strengths and limitations.
Finally, and most importantly, ensure you establish the governance and controls around their use – and if you haven’t looked at it for a while, remember to update your own Responsible AI policy.
