Sixfold Resources

Companies of all sizes are actively exploring how emerging AI technologies can overcome longstanding business challenges. Inevitably, they run up against the reality that weathered AI pros like myself have long known: AI ain’t easy. 

Rather than going it alone, many businesses choose to partner with firms that specialize in building solutions with LLMs. The good news? There are a growing number of AI vendors to pick from, with more popping up all the time. The bad? Discerning if a vendor can deliver what you need isn’t always so straightforward.

It seems like everyone and their little cousin touts the ability to “wrap” custom applications around one of the big-name LLMs. If that’s all they bring to the table, they might help you address simple use cases, but probably won’t have the chops to build and manage complex solutions in heavily regulated industries like insurance. That’s a whole different thing.

So, how can you tell if a prospective vendor can meet your business's needs? In this blog post, I’ll explore some key areas along the AI value chain and propose some questions to ask so you can make an informed decision.

So, how can you tell if a prospective vendor can meet your business's needs? In this blog post, I’ll explore some key areas along the AI value chain and propose some questions to ask so you can make an informed decision.

Input preparation

What you put into your AI system is what you get out of it. Make sure a prospective vendor prioritizes clean data, stored & handled in a secure compliant manner.

You can think of data like a commodity that powered the previous century: oil. You don’t just dig some oil out of the ground and pour it into your gas tank. (Or, I guess you could, but you wouldn’t get far before your engines seized up.) Like oil, data requires multiple rounds of preparation before it can be used. 

The value of the output your AI produces is directly related to the quality of the input. Before moving forward with any prospective vendor, ensure they have the means—and indeed, the knowledge—to help you build compliant, secure data workflows from beginning to end. 

The value of the output your AI produces is directly related to the quality of the input. Before moving forward with any prospective vendor, ensure they have the means—and indeed, the knowledge—to help you build compliant, secure data workflows from beginning to end. 

Here are some points to consider to ensure this is a vendor for you.

Questions to consider: 

• ‍How will the data be collected?
  ‍Data must be carefully collected to protect privacy and prevent bias. Ensuring that data has been ethically obtained and correctly governed is a point of emphasis for regulators.‍
• How will the data be “cleaned”?
  ‍Data needs to be refined and structured in a way that an AI solution can use and interpret. Make sure a prospective vendor understands what types of data are appropriate for your use case and how to prepare it at scale.‍
• How will the data be transferred, stored, and secured?
  ‍When developing solutions for complex, highly regulated industries, proof of certification for things like SOC2 and HIPAA are table stakes. Additionally, you’ll want to verify that the vendor uses secure data transfer methods, such as encryption during transit and at rest, to prevent unauthorized access. Also, ensure they effectively track the status of the data over time via robust version control and data lineage systems.

Prompt development 

LLMs work best when you make it difficult for them to make mistakes. An AI vendor should understand how to craft prompts to generate business value. 

For an AI solution to generate value, it must surface useful information with as little human intermediation as possible. This is achieved by ensuring that every prompt to an LLM includes all guidance, data access, and guardrails necessary to generate a high-quality return. Things like: 

1. Industry-specific content to guide results
2. Phrasing that reflects informed insight into the domain 
3. Precise instructions on the structure of the result being sought 

Your vendor will need to demonstrate they understand the capabilities and limitations of AI and can provide insights on how to structure LLM conversations to extract maximum value. Here are some points to review with a prospective partner to ensure they have the means—and better yet, a history—of value-oriented prompt engineering.

Questions to consider: ‍

• ‍How do they build prompts, and what domain-specific knowledge do they have?
  ‍Technical acumen is one thing, but does the vendor understand the specific needs of your industry? It’s one thing to ask an LLM to plan out a fun afternoon at the beach, it’s another thing to have it understand if, for example, family-owned restaurants align with a home insurer's risk appetite, or not. ‍
• What methods are used to select material included in the context window?
  ‍You should understand the vendor’s criteria for selecting contextually relevant information and how they ensure this information is timely and accurate. Ask what processes they use to filter and prioritize the most pertinent data for inclusion in prompts.‍
• How often, and in what ways are prompts updated over time? Are these changes tracked?
  ‍Learn about their schedule for reviewing and updating prompts to keep them aligned with the latest industry trends and data. Ensure they have a system for tracking changes to prompts, including version history and impact analysis, to maintain transparency and continuous improvement.‍
• What methods are used to evaluate the results of prompts, and to compare the results to prior versions when changes are made?
  ‍Ask about their evaluation metrics and benchmarks for assessing prompt performance, including accuracy, relevance, and consistency. Understand their process for A/B testing new prompt versions and how they compare the results to previous versions to ensure improvements.

Output control

Non-deterministic AI systems act in unpredictable ways. A quality vendor should know how to measure misaligned behaviors, as well as how to address them.  
‍
The value of the output your AI produces is directly related to the quality of the input. Before moving forward with any prospective vendor, ensure they have the means—and indeed, the knowledge—to help you build compliant, secure data workflows from beginning to end.  Ask an LLM the same question 10 times and you might get 10 different responses. The goal is to generate 10 accurate, useful answers. Achieving this requires putting as much care into reviewing the system’s output as you do into preparing the input. 

Ask an LLM the same question 10 times and you might get 10 different responses. The goal is to generate 10 accurate, useful answers. Achieving this requires putting as much care into reviewing the system’s output as you do into preparing the input. 

Continuous monitoring and tweaking are necessary to adapt your system to accommodate new data and evolving requirements. Here are some questions to explore when evaluating a vendor’s approach to scaled output control.

Questions to consider: 

• ‍What evals will you run?
  ‍Inquire about their evaluation frameworks, including both automated and manual assessments, to ensure outputs meet quality standards. Learn about the specific metrics they use to evaluate outputs, such as precision, recall, and F1 score, as well as checks for hallucinations and biases.‍
• What role will human experts play in this process?
  ‍Verify that human subject matter experts are involved in reviewing and validating AI outputs to ensure they are contextually appropriate and accurate. Ask about their process for incorporating expert feedback into continuous improvement cycles for the AI system.‍
• How often will you review overall results, and what metrics will you use to guide refinement and improvement?
  ‍Get a handle on their schedule for regular reviews and audits of AI outputs to ensure ongoing quality and relevance. Inquire about the key performance indicators (KPIs) and metrics they use to monitor and refine the AI system, such as user satisfaction scores, error rates, and feedback loops.

Transparency 

Not only does visibility allow you to properly evaluate an AI’s performance, it’s increasingly required by regulators as a means to address system bias.

Transparency is crucial for every step from data preparation to prompt development and output review. You cannot evaluate what you cannot see. To maintain the highest possible standards, every AI vendor should be prepared to provide a window into every step under their control. 

Questions to consider: 

• ‍Can you provide clear documentation of your processes and methods?
  ‍Ensure that the vendor offers comprehensive and understandable documentation covering all aspects of their AI processes, from data collection to output generation. Ask for examples of their documentation to assess its clarity and completeness.‍
• Can you demonstrate every point at which they interact with an LLM, and provide a complete trail of what information was exchanged?
  ‍Verify that the vendor maintains detailed logs and records of interactions with the LLM, including data inputs, prompts, and outputs. Ensure they can provide audit trails that detail the flow of information through their systems, which is crucial for regulatory compliance and troubleshooting.‍
• Will you provide a routine report about their evaluations and measuring for potential bias?
  ‍Inquire about their regular reporting practices, including how often they produce reports on AI performance, bias detection, and mitigation efforts. Ask to see examples of these reports to evaluate their thoroughness and transparency in addressing potential biases and other issues.

At Sixfold, we’ve created a Responsible AI framework for prospects and customers to showcase our ongoing transparency work.

In Summary

AI has the potential to overcome challenges that have been holding businesses back for decades. If you haven’t started your AI journey, now’s the time to get started. Partnering with an AI vendor can help you identify use cases ripe for transformation and provide the skills to get you there.

I hope that this checklist helped you identify which vendor has the right combination of technical know-how, industry expertise, and regulatory awareness to get your business where it needs to be.

This post was originally published on Linkedin

Today’s LLM-based AI solutions boast powerful capabilities that just three years ago were only found in science fiction. Modern AIs, driven by advances in machine learning & computational methods inspired by the human brain, continuously gain new capabilities from the data they encounter, enabling them with previously unattainable potential.

However, when it comes to operating within complex, highly regulated sectors like insurance, not any ol’ AI solution will do. In this post, I want to explore why carriers are turning to a new generation of vertical AIs purpose-built to address the industry’s unique needs and challenges.

Horizontal solutions only leverage the Internet’s surface

“Horizontal” LLM-based chatbots (e.g., Open AI’s ChatGPT or Anthropic’s Claude) are competent at a wide range of tasks, but you’d never trust them to execute a consequential insurance underwriting workflow.

Well, I mean, you could. But your underwriters would still need to engage in dozens (or even hundreds) of rounds of prompts, follow-ups, and clarifications to surface the information they need—all of which would require close review & scrutinization for accuracy, compliance, and hallucinations. They'd need to invest time in sorting through pages of answers to find important facts, correlate & de-dupe information, build timelines, and draw relevant connections. After which, they'd have to relate all of these processed facts back to their risk appetite to evaluate the quality of the risk.

Horizontal, multi-use AI solutions deliver little—if any—operational efficiencies for a complex enterprise use case like underwriting. The industry needs something more from its AI.

How vertical solutions overcome the data dilemma

One key area where general-purpose LLM chatbots and wrappers crucially fall short is lack of access to specialized data. A LLM’s “knowledge” can only run as deep as the data it’s been trained on. With a horizontal solution, you’ve generally been limited by what’s publicly available online. To echo a common observation: these models offer “the average of the internet.” They might be perfectly helpful in, say, planning out a keto-friendly dinner for two, but much less so when it comes to assessing risk signals on insurance applications.

In order to be useful in underwriting, a generative AI solution must have been — as table stakes —trained on informative but isolated datasets such as loss histories. Even anonymized versions of these datasets aren’t available for AI training purposes (they can’t even be purchased).

To access this invaluable cloistered data, an AI vendor must cultivate relationships with specialized data gatekeepers and arrive at a precise alignment on use and security. It’d be impractical for a horizontal AI provider to address every possible enterprise niche. To pry these data doors open, you need highly specialized vendors with a singular industry focus.

Vertical solutions: a partnership of insurance nerds and tech geeks

Beyond special data access, a vertical AI solution is designed to address the highly specific needs of its sector. The complexity and regulations inherent to insurance underwriting require a team that is as well-versed in emerging tech as they are in long standing carrier challenges. 

A vertical AI solution likely incorporates a medley of intelligent tools under a single platform umbrella. A foundational LLM, for example, may be tapped for specific functions (e.g., summarization), but higher-level capabilities can only be achieved when the LLM is partnered with purpose-built functionality dedicated to specific tasks (e.g., external data APIs, vector stores, etc.) The solution’s precise structure must be guided by experts with an intimate understanding of today’s industry challenges—and an eye on the ones soon to be in effect.

Keep your eye to the verizon

Horizontal AI solutions are amazing, but they fall short in core underwriting tasks due to their shallow expansiveness; lack of access to specialized data; and ultimately the fact that they’re just one building block that must be partnered with industry-specific capabilities to deliver value to carriers. 

This article was originally posted on LinkedIn

Working in enterprise tech - an industry that has notoriously struggled to stay on the cutting edge - I never needed much convincing that AI would one day unlock massive opportunity. While we are nowhere close to meaningful adoption of AI, that’s where we’re headed.

The first wave of generative AI has been horizontal with the release of GPT-4, Anthropic, Bard, and others. But the next wave will be vertical, marrying the summarization and reasoning superpowers of the LLMs with specially-trained models and industry expertise.

This is why today, I’m excited to announce the launch of Sixfold, the first generative artificial intelligence trained to solve the hardest problems in the insurance industry.

With $6.5 million in seed backing from Bessemer Venture Partners and Crystal Venture Partners, and leadership with decades of insurance experience, Sixfold is poised to rapidly transform how the insurance industry thinks about and uses AI.

To start, Sixfold Gen AI will focus on one of the most intractable challenges in insurance: the inefficiency of underwriting.

Underwriting is an art form, requiring human-level pattern recognition to capture all of the things that go into understanding a complex risk. For years, insurers have struggled to extract clear rules and standards for assessing risk because there has simply been too much information. Insurance carriers have a “front-door problem”: because of how manual and time-intensive the underwriting process is, insurers can’t provide quotes on all of the opportunities that come their way.

A smart underwriting platform like you’ve never seen before.

Prior attempts at AI in insurance haven’t gained much traction because they took a “black box” approach and thought they knew better than the underwriters. These failed attempts need way too much data and haven’t produced consistent results. Instead, we’ve trained generative AI models to "understand" all of this information and assist humans with the manual assessment. With the Sixfold Assistant, underwriters will be able to quickly evaluate and rate all submissions, thus improving underwriters’ capacity as well as the accuracy and traceability of their decisions.

Today, the Sixfold Assistant is best positioned to eliminate a lot of the “grunt work” that underwriters deal with on a daily basis: tracking down information from third parties, poring through thousands of pages of documents, and making sense of unstructured data. Sixfold will serve as a co-pilot to underwriters, plugging into existing technology so insurers don’t need to overhaul legacy systems in order to take advantage of Sixfold’s capabilities.

We’re thrilled to be launching with our customer BTIS, the commercial insurance provider focused on the construction and building trades industry. Sixfold will be an important tool for BTIS underwriters to improve the speed and accuracy of their underwriting efforts. Sixfold will ultimately make BTIS more competitive by allowing its underwriters to spend less time poring over data and more time issuing policies to its customers.

Sixfold is partnering at launch with BuildZoom, the leading provider of contractor profile data, property building permit data and contractor sourcing services. Sixfold will also expand its partner ecosystem to include consulting and advisory firms, cloud providers, risk and prior loss providers and medical records providers.

Sixfold’s initial focus will be on the commercial property & casualty and life insurance sectors, with plans to expand across the entire insurance industry, going after the over $100 billion spent annually in the US to underwrite insurance.

I’m particularly excited to launch Sixfold because of who I’m doing it with.

Our founding team is made up of former founders and operators, with deep experience in highly-regulated industries, including decades in insurance. Jane Tran is our COO and co-founder. Jane and I spent years working together as founding team members at Unqork, the $2 billion enterprise no-code platform. Our CTO and co-founder Brian Moseley, joins us from American Express where he was Head of Developer Experience.

A huge thanks to Charles Birnbaum and Jeremy Levine at Bassemer Venture Partners, and Jonathan Crystal and Stephen McGovern at Crystal Venture Partners for supporting us in this journey, to our ambitious customers, and to our partners.

Let’s rewrite the rules of insurance together!

This announcement was originally posted on LinkedIn