The 2¢ AI summary meeting note

GPT-based meeting summaries, such as those based on OpenAI’s Large Language Models, cost about 1.6p GBP to generate securely on Microsoft Azure. However, GPT summaries must be thoroughly verified by a human, such as an adviser or paraplanner, before any client action is taken based on their content.

Kit Ruparel, Recordsure’s CTO, explains the practicalities, limits and regulation around generative AI technology. Recordsure has released a complete AI meeting note generator as open-source code for organisations that want to experiment with their own conversation summaries before selecting a vendor or using GPT-based meeting notes in business practice.

The innovation

In 2011, Nick D’Aloisio became the youngest recipient of venture capital funding for an app that could turn lengthy text into concise summaries. Less than 2 years later, he sold Summly to Yahoo for an estimated $30m before his 18th birthday.

Companies such as Otter.ai and Notiv, later acquired by Dubber in 2021, saw the potential to use generative AI to summarise Microsoft Teams meeting recordings into notes and actions. Microsoft Teams now bundles a similar feature with its premium edition.

The appeal is clear: participants can talk without having to take meeting notes, supported by companies that built products to reduce the burden of online meetings.

In 2020, OpenAI announced API access to early GPT LLM AI models. The market took little notice until late 2022, when ChatGPT and accompanying GPT improvements were released, followed by competing APIs and models from Google, Anthropic, Mistral, Baidu and others.

This changed the economics of building meeting summarisers. A vendor no longer needed a proprietary model trained over many years by expert data scientists using transcript labels crafted by human annotators. A junior developer could adapt online how-to code to an industry sector, creating a new wave of products and hype.

As with many new technologies, excitement can lead to false promises before practical use becomes clear. Using Gartner’s Hype Cycle terminology, the Peak of Inflated Expectations must pass through the Trough of Disillusionment before reaching the Plateau of Productivity.

Responsible hype

In Gartner’s July 2023 Hype Cycle chart for Generative AI, the technology was placed at the peak of inflated expectations and estimated to be five to ten years from the Plateau of Productivity. many AI experts would not strongly disagree with that status or timeline.

how many GPT-based vendors are honest about the technology’s pros and cons, or explain current limits and responsible use at the start of a sales call. The answer is rarely, if ever.

Recordsure describes itself as a responsible AI provider and says it talks daily to current and potential clients about what the technology can achieve. The experiences shared with Recordsure include early trials of GPT-based AI products that went nowhere after six months: expensive, time-consuming experiments in early-stage technology that did not meet promises. One bank was experimenting with its third vendor and repeating the cycle.

This was not a surprise to Recordsure. The rules for AI-generated meeting notes are the same as for other applications: generative AI is good at generating text because that is what it is designed for, but today’s generative AI is not good at finding or understanding content and is error-prone when forced into that use case.

To reliably find and categorise written content from transcripts or documents, there is not yet a shortcut to labelling large volumes of text and having expert data scientists design and test deep neural networks. That process creates a predictive AI product whose accuracy can be measured and qualified.

Hype Cycle image credit in the input is By Olga Tarkovskiy – Own work, CC BY-SA 3.0.

Is there value to a GPT-created meeting summary?

The industry desire for accurate AI meeting summaries is valid, and Recordsure does not dismiss the value of partially accurate meeting summaries. We need to recognise the limits of GPT-created summaries, be realistic about their utility and put adequate responsible controls around their use.

A GPT-generated meeting summary from any vendor will not be fully accurate. It may miss important points, misinterpret content it finds and make things up.

If a draft AI meeting note is given to a meeting participant, such as a financial adviser or government inspector, that person can correct it from memory and refer to the recording or transcript where memory is uncertain. This can produce a high-quality outcome and save time because the notes do not need to be created from scratch.

If the draft goes straight to the back office for an administrator or paraplanner who did not attend the meeting, that person needs access to the original recording and transcript to create an authoritative summary or populate documents and systems such as a Fact Find, practice management system or CRM.

Without additional tooling, this can be inefficient because the reviewer must effectively review the whole conversation to check the note. It may be nearly as fast to type it from scratch.

Meeting notes with citations can help verify what the AI found and how it interpreted the transcript or recording. However, citations can give a false sense of security. GPT citations often work by seeking evidence for what the system wants to say, and may ignore conflicting information.

For example, if a customer first says they want to invest £50,000 and later says they want to invest £35,000 after discussing emergency funds, a GPT meeting summary may point randomly to one figure and cite only the transcript source that supports its conclusion.

Citations also do not identify what the AI missed. Without other tools, the reviewer must still check the whole conversation record.

This is where traditional, data-scientist-built, properly trained and tested predictive AI has a role. Recordsure’s Conversation Intelligence can accurately and thematically mark up the original recording and transcript by topic so reviewers can find and check what is, or is not, in the AI meeting note and ignore the rest of the recording safely.

Will AI-generated meeting notes improve?

AI-generated meeting notes will improve, but it is unlikely that purely generative AI or GPT-based solutions will make major gains over the next few years.

Instead, improvement will come from companies that annotate many thousands of hours of recordings with millions of labels, then build, train and test deep learning models to find and interpret conversations in the same way a human would. In this model, generative AI stays in its place by turning final AI findings into human-readable summary bullets.

How can I judge the quality of AI meeting notes today?

AI meeting notes have a place and can provide efficiencies if used correctly and responsibly. Recordsure says clients using the latest Capture solution will receive GPT-based meeting notes free with their transcript.

Clients using more advanced Conversation Intelligence tools will also receive higher-quality outputs and tools to verify meeting notes in the back office more efficiently.

For organisations that want basic GPT notes, Recordsure has open-sourced code for generating meeting notes using OpenAI models within their own Microsoft Azure tenancy. Organisations can try it with Microsoft Teams or Zoom transcripts without concerns about data security or transcript redaction.

GPT-based meeting note vendors use the same underlying technology, even if they choose different GPT models. The free code is therefore likely to represent the methods and outputs available from others.

Recordsure has made it possible to customise prompts used to generate the summary and topics, similar to the playground vendors use to tailor meeting notes.

The cost of generating an AI meeting note securely in an organisation’s own Azure tenancy is about 1.6p GBP, or 2¢ USD, for a 1-hour meeting. This leaves budget to contract a software developer for a few hours if help is needed to get started.

The code is available on Recordsure’s GitHub repository here.

Other generative AI product visions

Some vendors offer GPT notes with future capabilities such as automatically completing Fact Find documentation, populating PMS or CRM fields directly, or drafting suitability letters and other client documents.

Because everything the AI produces needs human verification, firms should examine how any pre-population of documents or records fits into their qualification workflow.

For automatically creating suitability letters, this output warns firms not to forget the purpose of financial advice. They use specialist calculators, tools to examine appropriate products, and suitably qualified advisers and paraplanners for a reason.

One head of a wealth management organisation, approached by a vendor claiming a future route from meeting recording straight to suitability report preparation, said: “However good the tech might become, it’s just not feasible in practice. They’ve forgotten about advice. We might as well be providing robo-advice.”

Laws and regulation to know

After covering how AI meeting summaries should and should not be used, firms must consider Responsible AI and emerging regulation and law in their territory.

A GPT-created meeting note summary is an AI output and should be thoroughly verified by a human before any client advice is provided or any client decision is taken based on it.

The UK GDPR restricts solely automated decisions that have a legal or similarly significant effect on individuals. The UK government framework for AI regulation, adopted by the FCA, goes further on transparency and trustworthiness, warning about autonomous decisions and requiring controls for safety, transparency, explainability, contestability and redress.

The EU AI Act, recently passed into European law, addresses human agency and oversight. It categorises AI systems used to determine access to financial products, using credit-scoring as an example, as high risk. That requires both product vendor and product user to register such systems with the European Commission and monitor and report on their controlled use, unless the AI output does not materially influence decision-making and the system is intended only for a task preparatory to a human assessment.

In summary

AI-generated meeting summaries are an emerging technology. Products built around GPT LLMs are only partially accurate, but they can still be useful where firms have appropriate human verification processes and additional tooling to make review and correction efficient.

Firms should try before they buy, especially with recordings of true customer meetings that the vendor has not previously seen, or to experiment with Recordsure’s open-source code to understand strengths and limits.

Finally, firms should establish governance and controls for use of the technology and update their Responsible AI policy if they have not reviewed it recently.

the images accompanying it, except for the hype cycle, were created with generative AI, while all words were human.