Generative AI: Context Is All You Need
By Tara Stokes and Joe Pak
Generative AI: Context Is All You Need
In Dumbledore’s office, Harry Potter stumbles across a Pensieve – a magical stone basin allowing his Hogwarts Headmaster to transfer and store memories from his mind to be referenced and studied at a later moment.
Pensieve illustration created with Stable Diffusion
“One simply siphons the excess thoughts from one’s mind, pours them into the basin, and examines them at one’s leisure. It becomes easier to spot patterns and links, you understand, when they are in this form.”
In short, using the Pensieve allows Dumbledore to avoid information overload and identify insights that weren’t apparent to him when stuck in his head. We see many parallels between Dumbledore’s Pensieve and foundational models like BERT and GPT-3 – large pre-trained machine learning models trained over vast swaths of the Internet. Like a Pensieve, foundational models can store vast amounts of information that humans would struggle to hold in short-term memory and help uncover insights relevant to completing the task at hand.
Foundational models already power technology that we use every day. A few examples include (1) Google’s query completion feature which predicts a user’s search phrase based on the first few typed words and (2) its answer snippet feature which predicts and synthesizes an answer in response to the user’s search query.
1. Query prediction leveraging foundational models
2. Answer snippets leveraging foundational models
Google’s intelligent search features are great demonstrations of how models can serve as a sort of Pensieve for users – equipping them with terabytes of memory which can be leveraged just-in-time for any task. In the near-future, users may no longer need to click into and read webpages and documents to identify the answer to a question – by leveraging foundational model’s retrieval and synthesis capabilities, search will become single-page, and answer-first.
While Google was early to implement foundational models into its products, we anticipate that these technological advances will be impactful across enterprises where 80-90% of data is unstructured in the form of text, audio, images, and video.
The future of work
This technology will have widespread impacts across businesses by making workers more productive and more collaborative. First, we believe context-aware generation will drive productivity gains in the workplace – particularly where there is a large demand for throughput and compressed timelines.
Companies like Copy AI and Jasper have rapidly built large customer bases by helping content creators automate the rote, manual work of rephrasing and iterating on advertising copy – a time-consuming task at-scale. Shifting the burden of drafting and optimization to models creates time for human writers to focus on the nuances that require human intuition.
A second major benefit of this technology is enabling businesses to break down information siloes, making it easier for workers across the organization to easily access insights produced in different parts of the business.
For example, product marketers today may have multiple meetings with cross functional teams to collect and stitch together the information necessary to write a relevant and compelling case study. Imagine an AI application for long-form marketing documentation that leverages insights from every customer conversation, competitive intelligence briefings, and product strategy documentation to assist the writer in generating drafts and suggest supporting points.
By embracing context-aware workflows, business will be able to proactively push insights to their knowledge workers at the most opportune time. Enterprise knowledge has ephemerality – in today’s ever-changing environments, insights are most useful when they can inform a conversation, negotiation, decision, or debate. Imagine this paradigm spreading to design, learning and development, brand identity and more, unlocking new ways of working and creating value.
How we evaluate opportunities
Point72 Ventures has made several investments in context-aware, generative AI. We look forward to speaking with founders that are thinking about or already building businesses that leverage these advancements. When evaluating these investments, we consider factors such as:
Feasibility
- Domain-specificity: Is the context and information for the task general or public knowledge?
Desirability
- Throughput scale: How much rote, manual work is being done today?
Viability
- Value creation: How expensive is the time of workers who complete these tasks today? Could their time be shifted to tasks that require more human intuition?
In addition to these criteria, a few questions we ask for all AI application founders include:
- Making it Magical: How does the product weave AI into the fabric of workflows to make them more meaningful, productive, and even magical?
- Feeding the Beast: How does the product build feedback loops to make features and workflows iteratively better over time?
While foundational models unlock a vast set of compelling use cases, they come with a host of obstacles that founding teams will need to consider as they build more complex interfaces for enterprise adoption. These challenges include the potential for models to produce non-truths (hallucinate) or produce offensive or biased content. With current methods of pre-training, models can also quickly become outdated, unable to keep up with current events or changes in knowledge.
Despite these challenges, we are energized by the pace of innovation and believe that advancements in infrastructure and tooling will help entrepreneurs overcome these hurdles and amplify the benefits of these large pre-trained models.
We welcome founders who can bring their own experiences and views to help us “spot patterns and links” and we are committed to helping them strengthen their own conviction through close partnership and collaboration.
“I sometimes find, and I am sure you know the feeling, that I simply have too many thoughts and memories crammed into my mind.”