• Reddit's Strengths: Community-driven, interest-specific, and admin-moderated platform that fosters deep, vertical & anonymous conversations, attracting a younger, tech-savvy audience.

• Challenges to Growth: Low Daily to Weekly Active user ratio (DAU / WAU), low Average Revenue per User, and competition from AI-driven tools and platforms like Discord and X.

• Growth opportunities: Increasing awareness, enhancing rich media types, improving monetization efficiency, and diversifying revenue streams.

• Potential for Product Innovation with AI as a Tailwind:

  • Browser-Based Conversational Search Engine: Develop a native semantic answer engine to reduce dependence on external search engines.

  • Community Embedded AI Personas: Introduction of AI personas to drive cross-community engagement and curated content.

  • AI-Based Content Creation Tools: Facilitate efficient content generation with tools like thread summaries and text creation assistants.

  • Product & Services Marketplace: For niche products and services, leveraging community trust to boost transactions and ad targeting.

Reddit as a community of communities

Reddit has evolved into a unique digital community of communities, transcending traditional social media platforms. It serves as a global hub where users connect through shared interests and passions, effectively functioning as a "digital city." Unlike platforms that prioritize personal connections, GenZ on reddit create, edit, post, comment, like and follow in ways that may feel completely unfamiliar to older generations

The platform's recent IPO, raising $519 million at a $6.5 billion market cap, underscores its significant market position. However, despite high gross margins, Reddit reported a $90.8 million loss in 2023, highlighting the need for strategic monetization and network effect amplification of its young & growing & user base with a high potential long term value.

Community driven, Interest Specific, Admin Moderated

Despite the dominance of short-form video on TikTok, long-form video on YouTube, & images on Instagram, Reddit proves the enduring power of text-based media. Meta Threads' success, reaching 175 million MAUs by its first anniversary, underscores this trend. Reddit, however, differentiates in the following ways:

• Anonymity fosters authentic, long-form discussions, enabling users to express themselves freely without personal reputation concerns.

• Over 100,000 niche lightly moderated subreddits represent diverse vertical interest graphs, each with its own culture and norms with the top ones being focussed on college education, gaming, animation, humor, college sports, esports & sports, consumer electronics, life advise & music.

• A predominantly younger, tech-savvy audience creates a dynamic user base with potential for long-term engagement and network effects broadly categorized into:

  • Young Entertainment Users that are often transient, contribute sporadically, and access Reddit during breaks or commutes with shorter session times.

  • Pass-by Users: Arrive via search engines to read about specific topics without contributing to the community.

  • Highly Engaged Users who engage deeply by posting or moderating, returning regularly (every 7-30 days) to actively participate.

  • Hobbyists that use Reddit to explore and deepen their knowledge of specific hobbies, connecting with like-minded enthusiasts on specific subreddits.

• Vast repository of realtime user-generated content, including billions of posts and comments, makes Reddit a treasure trove of authentic information and perspectives, making it one of the most-visited sites in the U.S as of Dec 2023. 60% of Gen Z users say things are big on Reddit before they are big anywhere else.

• High-intent engagement from users seeking trustworthy answers and recommendations translates to valuable opportunities for advertisers. Reddit has served as an authentic place to find eCommerce product reviews with 82% Gen Zers trusting it almost as much as Amazon & Google.

• Light admin moderation strikes a balance between free expression and community health, allowing for open communication while maintaining order.

Engagement, Monetization & Competitive Risks

Let’s take a deeper look at some engagement & user metrics:

• DAU/WAU/MAU Ratio & Search Dependence While Reddit boasts a strong 77% WAU:MAU ratio, indicating consistent weekly value, its lower DAU:WAU ratio suggests a significant portion of transient users not returning frequently.

• Referral Traffic: A substantial portion of Reddit's traffic comes from users appending "reddit" to their search queries. This reliance on external search engines and potential changes to their result ranking poses a risk to organic traffic growth.

• Low Average Revenue Per user (ARPU): As of March 2023, Reddit's global ARPU was $3.42, with U.S. ARPU at $5. This is significantly lower than competitors like Meta, which boasts a U.S. ARPU of $68.5. The disparity is partly due to a large number of logged-out or non-U.S. users, limiting monetization potential.

• Multi-Faceted Competition: Reddit faces competition on multiple fronts:

  • Information sharing: AI-powered search tools and chatbots that summarize rather than route traffic may threaten Reddit's position as a go-to source for answers and discussions.

  • Community engagement: Platforms like Discord and X (formerly Twitter) compete for user attention, especially among younger demographics.

Core Drivers for Reddit to Innovate

To compete, differentiate and improve its user, creator & advertiser engagement, the following are some core pillars of focus for Reddit:

• Awareness Expansion: Reddit needs to broaden its reach beyond its current low ARPU user base by developing products and partnerships that make its vast knowledge base more accessible to a wider audience with higher willingness to pay. Integration of Reddit's vertical interest graphs into other platforms (such as eCommerce) could enhance engagement and visibility.

• Rich Media Expansion: With a 35% increase in video engagement in 2023, Reddit can capitalize on the growing demand for multimedia content. Combining video with its threaded text format can create a richer, multi-modal community experience, similar to YouTube's comment section but with its unique community thread dynamics.

• Improving Ads Effectiveness: Reddit's advantage lies in its ability to offer contextual and interest-based advertising within niche, highly-engaged communities. By focusing on moments of high user intent, delivering relevant and contextually-placed ads, it can differentiate itself from competitors platforms like Meta and Google.

• Creator Revenue Share: While exploring data licensing deals for AI training, Reddit must also develop new products and revenue streams that align with the community-centric model, including creator revenue sharing programs to maintain network effects and incentivize quality content production.

Potential Bets on Search Integrations, AI Personas, Creator Tools and Niche Marketplace Offerings

Improving Semantic Search with Deeper Browser & Integration

In Dec 2023, Reddit saw over 35 million search queries, with a 30% increase in “7-day user retention” for those searching directly on the platform. However, the vulnerability due to dependance of traffic from external search engines can be mitigated by:

• Building a vertical semantic answer engine that capitalizes on its vast corpus of human conversations, providing answers validated by community thread references, deep linked within search results, allowing users to expand, read, and engage directly with content.

• Developing browser plug-ins for Chrome, Safari & Edge and Android / IoS widgets to surface Reddit’s native search as closer to the start of the user session, circumventing traditional search engines.

• Providing users the capability to search within specific subreddits, search across, providing references & related queries for users to organically dive deeper.

Embedding Authentic Avatars within Community Threads

As explained by Marc Andreessen in this podcast, the future of internet is going to be conversations between AI & humans. With over 20 mn global users & 16 mn chatbots created in 2 years, character.ai has gained a lot of prominence by challenging the uncanny valley of people talking to custom-built AI avatars. Meta AI’s recent implementation of AI avatars across its family of apps such as Whatsapp, Messenger & IG DMs showcases how chat assistants can be distributed across social products. Reddit can capitalize on this behavioral user trend shift by:

• Introducing AI personas natively within community threads to drive cross-community engagement and curate fresh content to bootstrap content.

• Fine-tuning the persona to write in specific styles and curate content tailored to particular chains of thought for specific threads, maintaining consistency.

• Enabling cross-pollination of insights from similar or orthogonal communities, bringing diverse perspectives to existing threads & amplifying cross community network effects by deep linking other relevant threads.

• Enabling AI personas to engage with users over DMs, to facilitate deeper and more contextual conversation catering to user needs, but harnessing from the vast community corpus of data.

Discord has already seen an uptick in the use of AI characters within group chats while Character.AI rolled out a similar functionality late last year.

Empowering Users and Moderators with AI-Infused Creator Tools

Given Reddit's user-generated content (UGC) play and large text based expression, the platform is uniquely positioned to offer creator tools that can enable the following:

• Thread summaries, generated on-demand, would benefit transient users from search engines seeking quick answers without sifting through long conversations. While this might reduce time spent on threads, strategic use of citations to specific discussions could encourage deeper engagement. Additionally, in-line ad placements within these summaries offer new monetization avenues.

• For moderators, Text-based content assistants could significantly reduce the time spent bootstrapping and engaging communities. These AI tools, fine-tuned on Reddit's unique data corpus, would help craft content tailored to each community's style and interests. This approach maintains authenticity while fostering diversity through the cross-pollination of ideas from related subreddits.

• Multi-modal content creation features would allow users to seamlessly augment their posts with images, creating richer expressions. This is particularly valuable for younger users in entertainment-focused subreddits, empowering them to create "meme-o-graphic" content that can drive viral engagement loops. Offering these tools to both moderators and regular participants can enhance content quality and user engagement, potentially increasing DAU/MAU engagement.

  

The Future of Reddit as a Product & Services Marketplace

Reddit's potential as a marketplace leverages its strong network effects and community trust, similar to how Facebook Marketplace evolved on top of the flagship platform . Reddit is uniquely positioned in this space because of:

• Differentiated supply from traditional marketplaces through various products and services currently transacted on Reddit:

  • Digital services such as graphic design, writing, programming and tutoring are often exchanged via Reddit through specific communities.

  • Artisanal & bespoke products such as handmade craft collections, jewelry, collectibles (r/marketplace), bespoke watches ( 20K watches sold on r/WatchExchange), mechanical keyboards (r/mechmarket) paves a path for Reddit to find a niche similar to marketplaces such as Etsy.

• Existing Trust established through the detailed community threads that can enlist historical transactions, comments & reviews from other users regardless of the anonymous nature of users makes it easier for users to transact.

By formalizing these exchanges through dedicated marketplace pages and streamlined buyer/seller interactions, Reddit can enhance user experience and engagement. While marketplace fees may not be a primary revenue driver, the real value lies in the deep understanding of user intent gained through these transactions which can significantly boost advertising effectiveness and grow into new Ads verticals. The potential vertical search and discovery engine can be extended to interest-based e-commerce, bringing a segment of buyers closer to transactions.

Is Community + Data + AI = Higher Engagement + Profitability?

The Bull Case

Reddit's younger, tech-savvy user base, segmented across vertical interest graphs, has a higher propensity to engage with the proposed vertical search & embedded AI functionalities. This segment is also more open to transacting via such community focused platforms and engaging with niche brands which provides Reddit an opportunity to create a more nuanced advertising business. This, coupled with potential marketplace for bespoke products and services, brings the buyer much closer to the commercial and casual sellers. Browser & device level partnerships and semantic answer engine capabilities, if executed well, gives Reddit the chance of become a dedicated vertical search tool & eCommerce marketplace for an emerging user segment.

The Bear Case

Despite promising growth avenues listed above, the platform's ARPU may be constrained by the relatively low willingness to pay among its users, especially within a highly competitive ad market. Browser integrations for vertical search may not significantly boost user engagement compared to established LLM search providers such as OpenAI, Claude & Perplexity. While creative AI content could enhance community engagement, it introduces risks related to content quality, safety, and latency. Deploying such functionality at scale also requires substantial infrastructure investments, which must be justified by positive unit economics, especially for a loss making, yet emerging public company.

Find me on X

Find me on LinkedIn

Email Me

Today, we’re going beyond the cloud, to the device, removing those fundamental constraints of power and space, reducing latency, ensuring privacy. We believe AI will be distributed. The richest AI experiences will harness the power of cloud and the edge, working together, in concert. This in turn will lead to a new category of devices that turn the world itself into a prompt. Devices that can instantly see us, hear us, reason about our intent and our surroundings

The above is the opening line from Satya Nadella's recent keynote for the launch of a Windows PC tailored to AI. The new and improved Windows OS and laptop design claims to be 58% more performant and have 20% more battery life than the latest MacBook Air while processing state-of-the-art AI workloads. The event was perfectly timed two weeks prior to Apple's WWDC, where Apple is expected to unveil native AI features embedded within iOS 18 through Siri, Search, and Safari. The following trends are very clear:

• Small Language Models: The advent of smaller language models is intended to act as a local semantic kernel that can run on the device, understand and process the user’s screen context, and handle queries to a certain extent while delegating complex user queries to the cloud model.

• Cloud to the Edge: Part of the AI community is shifting from a pure cloud-based Model-as-a-Service approach to a hybrid approach including local multi-modal AI workloads interfacing with cloud hosted models. This approach leverages an ensemble of open-source and closed models based on the nature of input and complexity of the task at hand.

• Specialized Chips for Local AI Workloads: While companies such as Groq provide lightning-fast inference speeds on the cloud, the need for specialized chips for on-device processing is gaining momentum. Big tech and major chip manufacturers are partnering to bring state-of-the-art neural chip architectures to run local AI workflows in a low-latency, battery-efficient, and secure manner.

• Omni-Modality & Device Ambience: Model architectures are transitioning to uniformly tokenize multiple modalities of input, including text, images, voice, and video. They process them using an ensemble or sometimes a single model architecture while exposing such services across laptops, phones, tablets, and smart glasses. This leverages all sources of the user’s environmental context, creating ambient services that users can take anywhere and deeply interact with.

Multi-Modal SLMs deployed locally on devices

The core benefits that SLMs claim to offer over LLMs are reduced latency and higher data security due to on-device processing. However, this comes with trade-offs in battery life, compute constraints, and intelligence constraints. Notable recent model releases include:

• Phi 3-Silica from Microsoft: With 3.3 billion parameters, this latest release from the Phi 3 small language model series is optimized to work with Microsoft NPUs designed specifically for co-pilots and PCs. Phi 3 claims to outperform GPT-3.5 with far fewer parameters, has a token latency of 650 tokens/s, and leverages 1.5 W of power. It signals progress in deploying SLMs locally on optimized end-user hardware.

• Apple ReaLM: This hybrid architecture deploys a smaller language model fine-tuned on reference resolution tasks such as a user’s active screen, conversational history, and background processes that serve as multiple signals of context. It creates diverse context representations sent to OpenAI’s models for comprehensive responses. This is targeted at maximizing user experience by deeply understanding user’s screen context and getting better responses, possibly from cloud hosted LLMs to create a hybrid approach towards building useful AI functionalities within Apple devices.

• Google Gemini Nano: This multi-modal on-device foundational model aims to make the end-user experience faster and more secure. It is currently catered towards locally processed audio calls for spam detection, improving accessibility via TalkBack, etc. that make for useful functionalities for the Google Pixel.

• Stable Diffusion 3 Partnership with Argmax: Deployed via DiffusionKit for on-device inference on Mac, this model is optimized for memory consumption and latency for both MLX and Core ML (frameworks for running machine learning on Apple silicon) and showcases Stability AI’s investment in on-device model execution frameworks.

On-device end user apps deployed across multiple devices

Now that we’ve discussed the underlying models powering the end-user context layer, it’s important to understand how this context is gathered via various device sensors. Local agents view what the user views by capturing their screen context alongside the camera and microphone feed for external audio and video when prompted. They process multiple modalities of inputs in real-time, running inference quickly to provide a low latency user experience. Here are a few latest announcements:

• ChatGPT-4o: The latest launch announced a local desktop client that can watch a user screen and help with various use cases such as summarizing code, analyzing complex charts, or assisting with math problems. One of the demos showcased ChatGPT’s capability to watch a live video feed from a tablet camera and explain the solution to the user in real time with minimal latency. This signals the power of an omni-modal model service interfacing via a local desktop client with low-latency video & user screen inputs. Creating useful desktop clients that can not only chat with the user, but also watch the user’s screen, gather video via device cameras, while sending high bandwidth information in real time to the core model service facilitates seamless human-machine interaction.

• On-Screen Widgets: In the latest Android keynote, Google introduced the “AI at the Core” initiative for Android, that includes functionalities such as “Circle to Search” and context-aware features by overlaying the Gemini assistant on top of existing apps to seamlessly switch back and forth between the app and the assistant. Google also claims to release the capability for Gemini Nano to analyze the content of active applications on a user’s screen to provide relevant suggestions in the upcoming months. This is similar to the ChatGPT-4o desktop client, but implemented keeping mobile form factor in mind, making it easy for users to retrieve information with minimal context switching.

• Meta’s Ray-Bans’s latest multi-modal AI upgrade: This has turned out to be a surprising success for consumers, as the glasses can see exactly what the user sees through the embedded camera. While the AI assistant largely uses a Bluetooth connection to a mobile device, leveraging the mobile internet to interface with Meta’s hosted Llama models, the core value-add comes from the capability to summon the model intuitively via voice and interact with the assistant in real-time through audio and video feed. This sets the precedent for how AI agents can become much more ambient across our real and virtual world.

In the existing architectures leveraged for various device types, we’re noticing deeper OS level integrations for the SLMs exposing various on-device components while giving them the flexibility to interface with end users via lightweight client applications supported across multiple devices. However, we may see see more powerful on-device models that increasingly reduce the need for interfacing with cloud LLMs, processing complex user queries locally while keeping user data more secure and minimizing latency.

Specialized Chipsets

While Nvidia's data center revenue in Q2 2024 has continued to surge with increasing gross margins due to growing demand for AI training and a lot more inference, we’re starting to see several hyperscalers build out or advance their own chipset initiatives with Google Trillium, AWS Trainium v2, Meta MTIA. While lot of chip-level innovation seems to be shifting from training to inference, at the same time, we’re also observing various innovations for on-device chips optimized for local AI workloads.

• Microsoft’s Neural Processing Unit (NPU): This power-efficient local chipset can run voice recognition, image generation, and various other tasks on the edge while complementing CPUs and GPUs on the latest Microsoft Windows PCs, that claims to run a variety of small language models locally.

• Apple’s latest M4 chip: This release claims to have the fastest neural processing engine, paving the way for various on-device machine learning operations that Apple may announce for Siri and Search at WWDC 24. Apple currently uses this on the iPad Pro, but we may see variants of this chip leveraged for MacBooks and eventually the iPhone, which is where Apple may release a plethora of AI-native features across Siri, Search, and Safari.

• Google Tensor G3: Powers Pixel 8, claimed to locally execute the same text-to-speech model that Google has historically used in the cloud, and runs twice as many machine learning models on-device for various use cases.

• Several other manufacturers are competing to bring on-device chips to market for local AI workloads including AMD with its Ryzen 7040 series of notebook processors, as well as Samsung and LG for their consumer grade mobile devices. NVIDIA is also speculated to join the party by preparing a system-on-chip (SoC) that pairs Arm's Cortex-X5 core design with GPUs based on its own recently introduced Blackwell architecture. It signals a strategy shift from the Grace ARM CPU design focussing on data center applications to datacenter applications, to get a greater slice of the AI PC market, coalescing around CPUs with built-in AI acceleration.

From Multi-Modality to Omni-Modality

The latest release of ChatGPT-4o, which can reason across text, audio, and video in real-time in a fast and natively multi-modal way while detecting emotion in voice input, clearly marks an evolution in input encoder architectures, giving them a new name: "Multimodal Unified Token Transformers (MUTTs)". This signifies an evolution from single large monolithic models used separately based on prompt inputs to a micro-services style muli-modal model ensemble that can process various modalities

• Meta’s introduction of Chameleon, a mixed-modal early-fusion foundation model similar to GPT-4o, capable of interleaved text and image understanding and generation, also signals a shift in the research communities towards an early fusion multimodal approach.

Open Questions & Constraints

While the progress for on-device assistants looks very promising and is backed by real model releases, device rollouts, and meaningful demos, the following aspects are yet to be fully addressed:

• Intelligence Constraints: While small language models have shown promising performance on various benchmarks with a smaller number of parameters, they’re still far from the benchmarks of state-of-the-art large language models (LLMs) with medium to high parameter counts. Initially, the consensus was to scale data, compute, and parameters in a coordinated fashion in line with Chinchilla optimal scaling. However, more recently, it has become evident that for deployment efficiency, it’s more useful to focus on big compute and data for much smaller models. Meta's Llama-3 8b was almost as powerful as the biggest version of Llama 2. Phi-3 mini with 3.8 billion parameters was trained on 3.3 trillion tokens of high-quality data with extensive post-training, paving the path for future model evolution.

• Battery Constraints: While the models have become smaller and are running on optimized chipsets, battery usage remains a significant trade-off. Although Phi 3 has promising battery usage claims, long-term performance versus battery consumption is yet to be seen.

• Compute Constraints: While the inference speeds for Phi 3 are stated at 650 tokens/s, the small language model community as a whole is yet to see broader community-built end-user applications beyond those created by Google and Microsoft. This will help determine how latency problems are broadly solved by the open-source community on a broader set of small language models.

The Bull Case

From an optimistic perspective, the following factors may lead to large-scale adoption of SLMs (Small Language Models) locally deployed on devices:

1. Advancements in Research: As research progresses in finding high-quality data sets to train models with fewer parameters, we could potentially see smaller parameter models (< 1 billion) achieving accuracy benchmarks comparable to existing state-of-the-art (SOTA) large language models (LLMs).

2. Chipset Efficiency: Chipsets may become smaller and more efficient at handling higher magnitudes of computation. This could enable them to run even larger language models locally on consumer devices such as phones, tablets, and smart glasses.

3. Battery Improvements: Batteries in PCs, tablets, mobile phones, and smart glasses may become more powerful without changing form factors, allowing both small and large models to run with acceptable inference speeds and minimal power drain.

The Bear Case

Looking at future evolution from a different perspective:

1. Historical Trends in Cloud Computing: During the rise of cloud computing, many companies provided on-premises "cloud API proxies" or appliances for creating on-premises clouds. While infrastructure companies like Nutanix and Cisco had emerging products in this space, the value consolidated to the cloud hyper-scalers. A similar trend could apply to LLMs, potentially reducing the value of SLMs.

2. Increasing Power of Cloud-Hosted Models: Cloud-hosted models may become increasingly powerful compared to SLMs, making the trade-off for deploying SLMs less attractive. This could be further emphasized by significant improvements in network bandwidths and data center computing capabilities, reducing network latency and inference speeds, thereby decreasing the need for locally executed AI workloads.

Ultimately, the decision between LLMs and SLMs will depend on specific use cases, security requirements, latency, power usage, and user experience considerations.

Find me on X

Find me on LinkedIn

Email Me

• From Assistants to Agents: Initially designed as basic tools for answering tasks, AI chat assistants have advanced into autonomous agents capable of solving complex, multi-faceted problems.

• Limitations of Single-threaded Interfaces: These traditional chat interfaces, while effective for linear tasks, struggle with complex queries that require broader contextual understanding and multi-step reasoning.

• Open-Source Frameworks: Open-source frameworks have been instrumental in enhancing the capabilities of AI assistants, contributing to areas like software development, academic research, and creative content generation.

• Evolving Software Ecosystem: The growing ecosystem is currently burgeoning at the infrastructure layer through frameworks such as ChatDev, MetaGPT, AutoGen, with some releases for vertical-ized tools such as Devin and Multi-On, and promise of on-device multi-threaded agents for consumer grade devices

Base Large Language Models to Helpful Chat Products

The 2017 release of the transformer architecture created an inflection point for adoption of language models. Subsequent releases of GPT 3 in 2020 & GPT 3.5 in early 2022 led to the first helpful internet-scale answering engine (ChatGPT). Base models are NOT assistants, and they merely want to complete documents by predicting the next token. This first product iteration of a base model ended up being the model tricked into serving as an assistant tasked at answering questions, using the corpus of internet knowledge that it was trained on. This led to users being able to ask deeply complex questions and get answers as opposed to getting traditional search results which created a step function in productivity gains.

Limitations of Single Threaded Chat Assistants

The way we interact with LLMs is through single-threaded chat interfaces such as ChatGPT, Perplexity, Claude, Gemini etc. wherein every search query is attached to a single threaded chat session. While this model has yielded productivity gains thus far, here are some limitations that we’ve observed:

• Planning based on Broader User Intent: Queries often embody a user intent behind them, tied to a longer and much more complex task goal, that often requires planning an outline / identifying bodies of work, determining the necessary tools for each task and whether web searches are required, writing initial drafts & reflecting to iterate on multiple drafts etc.

• Context Segregated Search Queries: Based on complexity, users may typically spin up multiple search queries, associated with individual tasks which are often disjointed from a context perspective, but could benefit from cross-collaboration & abstraction into a broader goal. As of today, chat products enable users to spin up multiple queries without a means to group or link them together

• Context Switching within the same query: Users switch context all the time within these single threaded conversations wherein they may move from one phase of their complex goal / task to the other, to maintain continuity and ease of use. While Large Language Models offer context lengths to the order of 1 million tokens (Gemini 1.5), model accuracy reducse as the user adds more complex and varying context types as opposed to if the model is fed with a singular topic.

• Task Specialization by Model: While LLMs are typically adept at a wide variety of tasks within different verticals, models fine tuned / grounded on specific data may have the capability to perform better than general purpose models.

• Computational Inequality: A model spends the same amount of compute on predicting every token regardless of whether the task is complex or not, thus making it computationally very expensive. Only a portion of the model layers & nodes are leveraged for specific tasks, thus creating inefficiencies from an inference time and cost perspective.

• Reflection As Andrew Ng perfectly summarized, we use LLMs in zero-shot or a few-shot prompting mode to generate final output tokens. It’s like asking someone to write an essay from start to finish, without any pauses, backspaces and review. This often leads to hallucinations over a long range of predicted output tokens reducing overall reliability of responses.

Where do multi-agent frameworks add value?

From a use case standpoint, here are some key categories / users where such multi-agent frameworks may yield the highest benefit:

• Software Development Typically involves multiple activities including planning, requirements gathering, architectural design, systems design, coding, testing, bug detection and resolution done by a variety of roles such as product managers, architects, program managers, SW engineers and QA engineers. This makes it a great fit for a multi-agent style framework to solve each task separately while collaborating on a broader end goal. Github Co-Pilot Workspace is a great first example of a broader project based, task oriented & multi-agent assisted software development workflow.

• Corporate & Academic Research: Research across various industries involves engaging in information search based on multiple hypotheses, driving multiple internet searches to analyze market trends, leveraging tools to build scenarios and validating external search with internal knowledge bases. This is followed by strategy creation & iteration through an iterative process of documentation (tooling). This process is multi-faceted, involves multiple steps, tools and often delegated roles. Part of OpenAI’s 600K sign-ups for ChatGPT for enterprise showcases the growing value of this specific use case.

• Content Creators leverage an iterative process that involves discovering latest trends for novel content ideas, leveraging existing audience analytics to understand demographics and preferences, and experimentation to create an end product via potential trailers. Considering each step requires research, amalgamation of ideas and content creation via creative tools, multi-threaded agent workflows can help orchestrate the multi-modal content creation piece as well as augmentation with the creators’ existing audience analytics and trends via tooling integrations. Meta during its latest earnings call, talked about dedicated AI content creation tools for its advertisers as well as businesses leveraging their suite of services by creating specific AI tools for each aspect of their workflow.

Application Co-Pilots vs Unified Chat Interfaces

The most evident trade-off while designing multi-agent frameworks is whether companies building in this space should focus on building co-pilots that are deeply integrated with their existing business apps or building a completely new interface that integrates with an existing productivity suite while aggregating information for the user in one single place. This can be viewed from the following lens:

• Accretive Innovation: This is currently seen within incumbents wherein multiple co-pilots have been built per underlying application. This design decision makes sense for large productivity suite companies such as Microsoft or Google, wherein the business model is based on the number of seats per tool sold to customers. These co-pilots may or may not collaborate, reflect on each other responses or help with broader user intent to solve for an overarching problem.

• Disruptive Innovation: This interface may make more sense for newer companies such as OpenAI, Anthropic, Cohere and various other end user AI application builders wherein integrating across a wide variety of productivity tools, while keeping users on a centralized dashboard may make sense to justify the subscription costs (the most common business model) and provide the best user experience, while keeping it simple.

Product Architecture

A typical multi-agent architecture can include the following nuances:

• Multiple model architectures to solve for multiple modalities (diffusion driven image / video vs text based transformer models).

• Multiple orchestration types such as smaller models being hosted locally on a user device for the conversational component while the larger models orchestrated on the cloud via API calls for more complex tasks.

• Same models orchestrated with different contexts & prompts to generate different responses based on the decomposed sub-tasks.

• Models of different sizes / fine tuned states, based on nature of task, to leverage specialized knowledge and achieve higher accuracy.

The above nuances are representative and designs can largely vary based on the use case, product constraints and accuracy requirements. Below is a high level representative architecture of what this multi-agent workflow may look like:

Ecosystem

The multi-agent framework has an existing footprint and potential across these respective layers:

• Infrasturcture & Tooling: Several open source tools have gained traction recently such as AutoGen, Crew AI, ChatDev & MetaGPT that provide tools for collaborative multi-agent workflows for a broad set of, or specific use cases.

• End User Applications: This is the most interesting and eventually most valuable layer where:

  • Verticalized applications such as Devin, MultiOn, Glean leverage multi-agent workflows for consumer and enterprise use cases while solving complex tasks from coding, workflow automation to enterprise productivity.

  • Large novel consumer chat interfaces interfaces such as Cohere, Gemini, Claude, ChatGPT, Meta AI etc. are largely single threaded as of today and may achieve higher user stickiness by evolving beyond the existing single threaded workflows to such multi-agent architectures. While they may be using multiple models in the background, surfacing that framework for solving complex end user problems is yet to be seen. OpenAI’s custom GPT mention seems to be a step in this direction but the chats are still largely single threaded

  • On-Device Agents such as the one that Apple is rumored to release may involve leveraging multiple models under the hood including OpenAI, Gemini and their own OpenELM model, deeply integrated with Spotlight, Siri and Safari as the multi-agent assistant. Device manufacturers such as Google Pixel and dedicated AI devices / pins may follow similar product designs.

Looking Ahead

In the next section of this topic, we will focus on understanding how these multi-agent frameworks can be built from a product and design lens, the technical and cost / efficiency trade offs that are common while building them and various tooling considerations that need to be kept in mind.

Find me on X

Find me on LinkedIn

Email Me

References & Further Reading

• Communicative agents for software development - ArXiv

• MetaGPT: Meta Programming for a Multi-agent Collaboration Framework - ArXiv

• AutoGen: Enabling Next Generation LLM Applications via a Multi-Agent Conversation

• Microsoft - Multi-Agent Collaboration Framework

• Self Organized Agents

• LLMs as Juries

From a “low-key research preview” to 180 million monthly active users as of Jan 2024, OpenAI’s ChatGPT marked the explosion of a sprawling AI community across open source and proprietary models. In the week before the date of writing this article, there were 140 open source models released on Hugging Face Open Model Leaderboard. Here are a few trends that explain the growth curve and its shift from accuracy and breadth to efficiency and depth:

• NVIDIA sold a whopping 550K H100s across big and small AI vendors, creating immense revenue growth.

• With the current computational loads, AI is forecasted to consume 85-134 Terawatt hours by 2027, similar to what Argentina, Sweden and Netherlands use in a year

• Investment in AI followed a pre-ZIRP (Zero Interest Rate Policy) anomaly of growth at all costs in 2023, while the higher cost of capital demanded early & late stage private business models to generate cash flow and profitability

• In late 2023, research shifted towards smaller models like Mistral 7B, focusing on training with high-quality, smaller datasets, reducing human supervision for fine-tuning, and developing model compression and low-cost deployment techniques.

• Going from proof-of-concept to production, to a hyper scale business requires sustainable unit economics that has to stem from the ground up model architecture and design to efficient deployment.

Let’s dive deeper into how research in the last few months is paving the path via various creative solutions that accelerate this trend. For anyone looking for a quick primer on fundamental concepts on LLM training, I would highly recommend State of GPT by Andrej Karpathy or this article.

Dissecting existing unit economics of AI models

The human brain is an amazingly energy-efficient device. In computing terms, it can perform the equivalent of an exaflop — a billion-billion (1 followed by 18 zeros) mathematical operations per second — with just 20 watts of power. On the other hand, large language models still have a long way to go, in terms of computational and energy efficiency, as they use a significant amount of GPU compute and energy to be trained from scratch and make inferences.

• Capital Expenditure: Meta's acquisition of 350,000 NVIDIA GPUs for about $10 billion highlights the significant capital required for AI infrastructure. Meta reportedly spent $20 million training Llama 2, while OpenAI's daily expenses were about $700,000, or 36 cents per query, as of April 2023.

• Training Time: Meta’s Llama 1 took approximately 21 days to train on 1.4 trillion tokens on 2048 NVIDIA A100 GPUs which is a substantial amount of time to train models on base data sets that can expand and evolve.

• Gross Margin: Anthropic on the other hand is estimated to have gross margin of 50-55% as opposed to a typical 80% gross margin profile often boasted by SaaS & PaaS businesses. A large part of it comes from the cost of investing in cloud compute that runs AI models at scale.

• Mega Venture Funding Rounds: MANG (Microsoft, Amazon, NVIDIA, Google) recently represented $25+ billion of venture capital investment that accounted for 8% of the capital raised in North America, out of which a whopping $23 billion + was invested in Data and AI. A vast portion of this investment came in the form of cloud credits for leveraging the pool of compute that’s owned and run by the hyper scalers allocated towards model training.

While OpenAI was incredibly successful in creating a first mover’s advantage for its information retrieval & infrastructure services, to realize sustainable software unit economics, with higher gross margins & operating leverage has to evolve from the model training phase all the way up until the inference phase.

Learning from History: Data Centers in 2000s

In the late 1990s and early 2000s, the tech boom spurred a sharp increase in demand for data storage, web hosting, and internet services, escalating the need for data center space and infrastructure. This surge pushed up the costs of building and operating data centers due to significant investments in construction, power, cooling systems, and internet connectivity. At the time, less advanced cooling and energy technologies led to higher operational expenses. Additionally, real estate and energy costs soared, while supply chain issues caused by increased demand for servers and networking equipment further inflated prices. Companies heavily invested in IT infrastructure, aiming for growth but not always with profitability in sight. The dot-com bubble's burst prompted a reevaluation of such investments, revealing an oversupply in data center space and network capacity that took years to adjust. The fallout included numerous internet company failures and a shift towards more efficient data center designs and operations, paving the way for cost management improvements, thus creating sustainable gross margin behemoths like AWS.

Business Implications for AI Infrastructure

Learning from history, the key pillars for success in the coming months and years would require an amalgamation of finding efficiencies in the e2e model training & deployment process with accurate demand mapping to build profitable businesses with healthy gross margins:

• Cheaper model training through efficient data collection, labelling and training methods potentially reduces recurring CapEx (for retraining models with GPU compute) with exponentially expanding public and private data, which directly impacts Free Cash Flow (FCF), thus freeing up capital for more strategic investments in R&D.

• Efficient fine tuning can help reduce investments in labor intensive fine tuning and feedback techniques, managing incremental OpEx investment for improved model accuracy, and maintaining operational efficiency while improving customer retention and increasing topline.

• Efficient model designs has the most direct impact on COGS (Cost of Goods Sold) and thus gross margins, through reduction in infrastructure and operational costs related to model deployment and maintenance.

In an AI economy where open source foundation models are abundant, pricing wars around providing the cheapest, fastest and best service will create margin compression which further accentuates the need for efficiency in all aspects.

How is the research community working towards building efficiencies?

Let’s dive deeper into the ongoing research in each of the phases to better understand how this may be achieved.

Cheaper and Faster Model Training

While the corpus of internet scale data for training large language models is extremely diverse, it can be unstructured, noisy, and of poor quality, and thus the training can often be compute intensive and long, or may require significant manual & computational effort in cleaning the data set. A couple of areas of research involve models creating their own training data sets or representing data sets in high quality formats.

• Recent research introduces a novel method called Self Alignment using Instruction Backtranslation to enhance language model instruction-following. This technique generates training examples from vast, unlabeled datasets, like web documents, and selects the most effective ones for further training. By self-training, it significantly improves the model's ability to understand and execute instructions without extensive human-annotated data. The technique was experimented on pretrained LlaMa model, with different parameter sizes (7B, 33B, and 65B) and was proved to maintain similar levels of accuracy, while consuming less computational power

• Web Rephrase Augmented Pre-training (WRAP) was recently introduced as a method to enhance the efficiency of training large language models (LLMs) by using a mix of real and synthetic (rephrased) data. Think of WRAP like a smart filter that cleans and diversifies the data before feeding it to the learning model, similar to refining raw ingredients in a gourmet meal. WRAP uses an off-the-shelf, instruction-tuned model to rephrase web documents into styles like "Wikipedia" or "question-answer" format. This approach aims to create high-quality synthetic data that complements the real, noisy web data. Experiments demonstrate that WRAP can 3X the speed of pre-training while improving model perplexity and zero-shot learning capabilities across a range of tasks.

Efficient Fine Tuning

Advanced techniques like Supervised Fine Tuning, RLHF, and Reward Modelling enhance AI performance by incorporating human feedback, improving model responses to mimic human behaviors more closely. By evaluating multiple responses from the model, human preferences shape a reward model that guides AI training, optimizing accuracy without extensive computational resources. Techniques like LoRa and QLoRa offer efficient, minimal model adjustments, facilitating large model fine-tuning on limited hardware. Some more promising research in this area includes:

• Direct Preference Optimization was recently introduced as a technique that fine-tunes language models directly on preference data, bypassing traditional, more complex reinforcement learning approaches. It gets rid of is the iterative training of a separate policy model by cleverly re-parameterizing the reward model in a way that one can train a language model to optimize the reward. Results show that DPO can match or exceed the performance of existing methods in making models produce preferred outcomes for sentiment control and summarization tasks, with less computational effort. Part of what makes this exciting is that there is now an active ecosystem of small “alignment” datasets and so many people are producing DPO instruction tuned models relatively cheaply (compared to training from scratch) with different characteristics. Andrew Ng summarizes this very well in his tweet:

• Self Play Fine-tuning (SPIN) was recently introduced as a training method, enhancing the performance of language models by leveraging self-play, a method where the model generates data by interacting with itself. This self-generated data serves as new training material, helping the model learn more nuanced language patterns and improve over time. The approach demonstrates a notable improvement in language model performance across several benchmarks, showcasing the method's effectiveness in enhancing model capabilities in a cost-efficient manner.

• Weighted Average Reward Model (WARM) as a method proposes averaging of parameters of different reward models. This technique is akin to taking the collective wisdom from multiple expert systems to guide the decision-making of large language models more effectively. The research primarily aims to enhance RLHF alignment step for LLMs. Taking advantage of multiple LLMs created during training is particularly lightweight, and is claimed to cost less than a single model during inference time.

Efficient Model Design

The classic scaling laws of LLMs generally state that an increase in model size, dataset size, and computational power tends to have a significant impact on the model performance, but with diminishing returns beyond a certain point. For a while the consensus was to scale all three in a coordinated fashion in accordance with Chinchilla optimal scaling, whereas more recently it has become manifest that the useful thing for deployment efficiency (rather than training efficiency) is to go big on compute and data for a much smaller model. Hence we see things like GPT-3 with 175B params trained on 300B tokens but Phi-2 with 2.7B params trained on 1.4T tokens. The race for the biggest model eventually seemed to converge around similar metrics / numbers on various benchmarks.

• The most famous advancement in building efficient models has been the Mixtral 8x7b release that signifies the computational efficiency of sparse mixture of experts where an ensemble model combines several smaller “expert” sub-networks, where each subnetwork is responsible for handling different types of tasks. By using multiple smaller sub-networks, MoE aims to allocate computational resources more efficiently. With 47B parameters, it is significantly smaller, yet comparable in performance to the Llama 2 70B model.

• Jeff Dean (Chief Scientist at Google) talked about the concept of Pathways in 2021 which stated that the activation of a whole dense neural network to accomplish a simple task is unlike our brains, where different parts of the brain specialize on different tasks. He proposed that “sparsely activated” models can dynamically learn to activate parts of the neural network that are the most efficient for that task, which not just gives it the capability to learn a variety of tasks but also makes it faster and energy efficient. A recent paper on Improving transformers with irrelevant data from other modalities was an interesting approach that seemed to mimic the concept of interdisciplinary thinking by leveraging unrelated data sets for model training.

• Matroyoshka Representation Learning is a notable method that proposes a nested structure of data representations where each layer can serve a different computational or accuracy requirement. This technique helps machine learning models adjust their complexity based on what's needed for a task. It's like having a tool that can switch between being a hammer and a screwdriver based on the job, helping balance accuracy and computational needs efficiently.

• Slice GPT is an approach that can be imagined as a highly detailed and expansive library (representing a large language model) where you're trying to make space and improve efficiency without losing much information. The approach involves "slicing" - selectively removing parts of the model's weight matrices (akin to shelves/books in our library analogy). This is achieved through a novel method that ensures that the core functionality and outputs of the model remain largely unchanged despite the reduction. The method effectively reduces model size by up to 30%, with minimal impact on performance.

On-Device Multi-Modal Models

The final chasm that’s currently being crossed is the capability to run model inference locally, on existing mobile phones, laptops and potential AI native devices of the future. While desktops and powerful laptops may still have the capability to run inference for Small Language Models (SLMs) such as Mistral 7B & Phi 2 via Ollama, recent advancements through Core ML in iOS and MediaPipe on Android set some great foundations for running many on-device models. We’re already seeing work done on top of Phi 2 such as TinyGPT (2.8B parameters) and more. Here are some examples:

• TinyLlama, with its 1.1 billion parameters is an accessible and affordable Small Language Model, that is cheaper to develop and pre-train, can typically be fine tuned using a single GPU and is more energy efficient which is crucial for preserving battery power on a smart phone or other edge devices. While it’s far from outperforming major LLMs, it’s intended to present better results when fine tuned for specific tasks.

• Google recently released Mobile Diffusion that can run on both iOS and Android to generate 512x512 images in less than a second, with a mere 520M parameters, leveraging key techniques such as latent diffusion. It can be run on both iOS and Android, and seems to be a great step

• After the release of Ferret late last year Apple seems to be making updates to its iOS 17.4 release, wherein it plans to roll out updates to Siri and messages, by rolling out the Siri summarization framework. Apple seems to be using multiple versions of its AjaxGPT models including one that is processed on device and one processed on the cloud. It is also benchmarking the model results with GPT 3.5 and FLAN-T5.

While the above research is promising, running models locally on devices such as laptops & mobile phones without consuming significant memory, compute and battery power is yet to be seen at a large scale and based on Apple’s and Google’s announcements, there’s optimism that this can be brought to production scale, at-least for specific tasks & features. Moreover, small to medium sized models may have weaker on general knowledge and for acting as a dialog partners.

Find me on X

Find me on LinkedIn

Email Me

Subscribe now

After writing a series on explaining the current state of traditional autonomous software workflow agents through part 1 focussed on the impact of multi-modal large language models on existing tools, followed by part 2 & part 3 focussing on how such agents can be effectively designed keeping various factors in mind such as product, UX, data, systems, cost of ownership, I wanted to sum up by discussing a vision for what the next generation of workflow agents look like, from a technical lens based on all the latest research on multi-modal language models, tying the existing gaps and user journey to viable solutions that address the problem statement and pain points. The article takes a real life example of an autonomous agent for a financial services use case in context to better explain how such agent systems can be built & architected with multi-modal models.

Moving from analysis to action

2023 has seen a massive advancement in the quality and efficiency of language models of all sizes being implemented at scale with cloud based models such as GPT 4 from OpenAI, Claude 2 from Anthropic, Command from Cohere etc. as well as open source models such as Llama 2 (Meta), Mistral 8x7b and many more listed on the Open LLM leaderboard on Huggingface. Language models seem to have the capability to predict the next best token to conduct sentence completion, hold a chat conversation, summarize large text chunks, generate and understand images and perform basic reasoning from the context given. While doing so, they can also perform basic actions such as a web search with Bing, spin up a Python environment for code execution etc. However, the ability of agents to interact with the real-world by calling existing apps and services gives them "arms and hands" to automate various business processes that make users far more productive. Action is an area for advancement that has the capability to create value for end users.

Understanding the evolution in the stack

Andrej Karpathy recently discussed the idea of an LLM OS on X, which is a much finer picture to the Semantic Kernel Architecture built by Microsoft, that gives a model implementation “arms & legs” to perform user actions. The evolution from cloud infrastructure, app frameworks, DevOps, SaaS applications and APIs / integration tools has found its AI equivalent with an updated architecture drawn out by Microsoft, as showcased below. This then creates a pathway for a new set of co-pilots with updated capabilities that enable them to interface with existing and new apps and services to improve user productivity.

Action driven agents for enterprise knowledge workers

Let’s dive deeper into how action driven agents add value to enterprise knowledge workers. Knowledge workers leverage software orchestrated on computing devices, commonly the desktop and mobile which constitutes the majority of manual work today, thus creating an opportunity for the most value creation. The recent release of AutoGPT paved a great first stepping stone for the community to envision how agents can break down complex tasks into simple steps and then interface with apps & services hosted on the internet to complete those tasks autonomously. While this is a great start, we have come a long way in building better ingredients through research in various core functionalities that can power the new generation of workflow agents.

Before we dive in, let’s understand the user journey for a workflow agent in its capability to observe & identify an existing process, build the agent flow / logic and finally deploy it, while also fine tuning it based on user interaction.

Taking a real life knowledge worker example…

To put the above in context, let’s take an example of an actual process that occurs within a sell side or asset management vertical at a large bulge bracket bank or a hedge fund, engaging in foreign exchange trades wherein a foreign exchange trader sells $100,000 USD in exchange of $140,000 CAD, at a specific rate of 1.4 CAD / USD and sends an email to a trade settlements desk analyst to get compensated for the trade, as well as ensures that the trade is settled. This is a lengthy and arduous process that has the following steps:

Let’s use this example to understand the AI agent capability needed to fully accomplish this task and the technologies available to power this.

Observing via Multi-Modality & Comprehension

Taking the example above in context, an autonomous workflow agent powered by a multi-modal model should be able to:

• Watch Email Queues and identify emails that reference to a ForEx trader executing a specific trade from the queue.

• Read Unstructured Emails to identify the currency sold, bought, executed rate & transaction ID.

• Identifying various apps and services that the settlement desk interfaces with such as email, Bloomberg terminal, internal terminal / mainframe, sales credit payout system, externally available trade settlement system such as Broadridge.

• Reasoning to identify the underlying formula for sales credit calculation.

Adept AI recently released a multi-modal model for knowledge workers called Fuyu 8-B, which constitutes a simpler architecture compared to other image understanding models, and is able to answer UI-based questions on the user screen. Let’s consider a great example stated in the release notes wherein a specific workflow may involve questioning whether the second email has a star agains it or not. The above steps, applications and logic can be reasoned by multi-modal models by closely watching the user screen, sometimes over several iterations to chalk out simpler sub-tasks, identify applications and user actions.

Building & Iterating on the Workflow

Once the agent has observed the user actions across several attempts, the agent ideally would perform the following actions:

• Summarize the key steps / sub-tasks in the process, applications interfaced, data exchanged from defined input & output sources, key triggers etc.

• Identify and present the initial skeleton with built in triggers, actions, logical constructs and branches with different outcomes while capturing all the edge cases & exceptions.

• Test run various scenarios in a guided way with controller user inputs to test robustness and functionality

• Present the user with specific areas where the agent requires more context to clearly articulate the process steps, branches & exceptions.

• Identify SLAs, runtime constraints and deploy multiple instances at specific speeds as needed to meet actual volumes

• Allow configuration of channels for the user to interface with the agent such as a guided forms based app / widget interface, web form, chat or voice based interfaces. The capability to structure and present this information in a guided & easy to understand manner is key to building a robust and scalable agent alongside the user through automated workflow discovery & development.

Interfacing w/ Various Apps to Perform Actions

Putting the example into context, modern workflow agents powered by Large Language Models have the capability to interface with applications and services of various types in the following ways:

• Interaction with applications via REST APIs can be achieved for applications such as Bloomberg and Broadridge built on a micro-services architecture, exposing APIs that can be harnessed via built in plug-ins or auto generated code by the agents.

• Interaction with legacy applications may require emulation / mimicking of human actions due to lack of integration capabilities. It’s more common for traditional industries to use such applications which are slow to migrate due to the tech debt and the scale at which they operate, which prohibits them to move to modern application stacks for long periods of time.

• Interaction with AI native third party autonomous systems which may involve exchange of data on a lower level semantic layer to facilitate lower latency or efficiency. In this case, specific trading systems may be fully autonomous in nature, built on AI native algorithms. This is a growing area of research as model architectures evolve.

Here is a fun post on X with an example of how Adept interfaces with various applications w/o API capability natively on a browser using Fuyu 8-B.

Role of Human-In-The-Loop driven Fine Tuning

The key problems / hindrances for agents to work consistently over long periods of time with a variety of complex apps can be largely bucketed into:

• Reconfiguring actions to adjust to frequent changes in user interfaces including changing button positions, UI styles , for applications with no API capability which otherwise requires a ton of refactoring, thus added more time to develop & maintain agents.

• Seamlessly flagging errors and absorbing user input during execution to accomodate for any unaccounted edge cases, error scenarios, build robust rollback and handling mechanisms over time which creates a true human-in-the-loop / continually learning agent as the workflow evolves.

Supervised Fine Turning and Reward modelling have a great role to play but can be expensive and time consuming with shorter feedback loops. This is another area of growing research, and methods such as Direct Preference Optimization may pave the stepping stones for efficient HITL workflow agents with reduced human touch over long periods of usage.

Agent Deployment / Orchestration

Putting the use case / example into context, the following considerations need to be kept in mind:

• The implementation of such agents may often require the agent to run locally on an on-premise container or the end-user machine, based on the sensitivity of information handled by the agents. There have been various advancements in locally executed multi-modal language models such as Tiny GPT-V, Apple Ferret and most notably, Phi 2 by Microsoft that are able to run locally on a user machine and perform at considerably high levels of accuracy with text & images. The key advancement to watch would be the availability of chipsets that can power such expensive computations on user devices.

• Maximization of the user experience involves splitting the end to end process into sub-tasks that can be done asynchronously in the background and tasks that require a user input for guidance. Putting the example into context, this can be explained using the following visual:

Notable Open Source Projects

While researching on workflow agents, I came across some great open source projects that are bringing the above vision to action:

Open Interpreter

Open Interpreter, a project that facilitates running code on a local machines recently released an update that enables a user to interface with a computer screen using vision models and general interaction APIs. This is indicative of the movement towards local execution, which is especially beneficial for high volume enterprise workflows tasks done in secure environments.

App Agents

While modern day workflow agents have typically been limited to desktop screens, a lot of enterprise applications are now native to mobile with crucial tasks also being done on mobile. Agent frameworks such as AppAgents provide the capability to run such automations natively on mobile, thus extending capability beyond desktop.

Wrap up

Thinking of this more philosophically, a case can be made that a Cambrian explosion and adoption of AI native applications and services could possibly bring in a new paradigm of user interfaces beyond the “knobs-and-dials” interfaces of today, thus introducing new types of data exchange mechanisms where underlying models exchange data & information differently, hence creating a new paradigm of APIs. This may yield the capability of mimicking human actions on existing software types largely archaic and less valuable. However, adoption curves, especially for knowledge workers are often very slow, due to the complexity of organizational structures and their processes, which creates technical debt. Information & application silos are a constant occurrence for knowledge workers and enterprises. Thus, applicability of such agents shouldn’t be viewed as a bandaid, but rather a stepping stone for organizations to transition to more modernized and consistent application interface. The nature of human machine interaction may evolve over time and we may be mimicking evolved human-machine interaction types than the current day actions such as “clicking on buttons, dragging & dropping, scrolling etc.”. This maintains defensibility & relevance for this area of technology, which will evolve along with the new technology paradigm.

Find me on X

Find me on LinkedIn

Email Me

Following up on my previous article about spatial computing in enterprises, this piece focuses on immersive learning as a key application of Mixed Reality and how existing developer ecosystem solves for this problem. Immersive learning stands out for its ability to transform traditional training methods due to several factors:

1. Enhanced Operational Efficiency: Immersive learning significantly increases information retention by introducing spatial awareness in a 3D setting, combining visual and auditory elements. It's particularly effective for roles requiring spatial understanding and physical interaction with objects. Research and industrial case studies show that immersive learning participants retain operational details better than those in 2D or conventional training.

2. Reduced Training Time and Costs: This modality is ideal for high-turnover jobs where traditional onboarding is costly and complex. Immersive learning provides quick, comprehensive yet compressed training scenarios, beneficial for roles like last-mile delivery, which often experience high attrition rates.

3. Cost-Effective Asynchronous Training: Like digital training, immersive learning supports asynchronous formats, cutting the need for facilitator-led sessions. This is valuable in decentralized settings, reducing costs related to facilitation and the downtime of employees who would otherwise be working.

Let’s continue to explore key applications of immersive learning in various sectors:

1. Frontline Productivity: Common in logistics, manufacturing, retail, and healthcare, immersive learning is used for tasks like warehouse box stacking, store shelf arranging, and last-mile delivery. These simulations enhance productivity in these crucial areas thus boosting throughput.

2. Health & Safety: Building on frontline productivity, immersive learning is vital for training in health and safety procedures. It's particularly useful for scenarios that are rare, dangerous, or costly to replicate in real life, like bank robberies, workplace active shooter situations, and fire safety. Traditional methods like 2D trainings or skits are less effective compared to the immersive simulations that can prevent hazards and have significant financial implications.

3. Soft Skills & Customer Service: Traditional employee onboarding for customer service often involves 2D trainings or role-plays with trainers, which can be ineffective or expensive. Immersive learning offers a more effective solution by realistically simulating customer or employee interactions and enabling practice of soft skills in a spatially aware environment.

Understanding the gaps between existing customer demand and immersive content supply

While the ecosystem of immersive applications focussed on training as a use case is ever-growing due to its tangible ROI, there are some fundamental characteristics of how learning & development organizations adopt immersive learning and the nature of experiences that are built by the ecosystem.

General Acclimation & Navigation

Existing Problem: Most employees subject to immersive learning aged from 18 to 60 show varied familiarity with mixed reality. Younger employees often have gaming experience with similar technology, while older groups struggle with the novelty of wearing a mixed reality headset. This disparity can lead to discomfort or disorientation, especially when first using virtual reality. Navigating through a 3D UI is not intuitive for many enterprise users, requiring guidance such as quick tips or ghost animations to interact with 3D elements effectively.

Developer Painpoints Developers often create immersive learning experiences without standardizing the acclimation or navigation processes. This inconsistency means users must learn different systems for each new application, creating a need for additional ramp up on every marginal individual application workflow, thus creating friction in onboarding employees to a diverse set of training experiences.

Learning & Instructional Design

Existing Problem: Traditional enterprise learning content adheres to an industry-wide framework, emphasizing aspects like creative design, content flow, narrative, instructional format (visual and audio), behavioral modeling, and real-time feedback. While current immersive learning materials may boast high-quality assets and animations, they often miss the mark on instructional design and structured flow, which are key to transforming a good experience into an effective training module.

Developer Painpoint: Many game developers, primarily involved in creating immersive learning content, lack expertise in educational frameworks necessary for effective learning experiences. The gap in their knowledge, coupled with the absence of suitable tools or templates and lack of learning and development (L&D) resources, hinders their ability to create properly structured learning experiences. While SDKs equip them to create interactive experiences, they don't provide the means to author educationally focused content.

Controller Interactions

Existing Problem: The complexity of interactions with immersive learning can vary from simple & guided trainings with a linear playback and simple point and click interactions, to more advanced 6 degrees of freedom (DoF) trainings that warrant advanced controller interactions, thus requiring the user to leverage controller buttons such as trigger and grab buttons for translation and object manipulation. Knowing when to hold and leave such button combinations is especially cumbersome for a relatively less savvy employee demographic. This is accentuated for more complicated trainings that requires such interactions frequently frontline productivity and safety trainings. The friction leads to learners getting lost within the experience due to the nature of complexity and not knowing what button combination achieves the specific task at hand, thus creating churn and lack of ease.

Painpoint from Developer Ecosystem: Developers often rely on headset SDKs for controller functionality, but there's no universal standard for button mapping across different head-mounted display (HMD) manufacturers. This inconsistency results in varied controller mappings for similar actions, sometimes even inverting button uses. Consequently, each new immersive learning experience may require users to relearn controller mappings, adding a layer of complexity and hindering smooth adoption.

Here’s a demo from Oculus on controller walkthroughs which is one of the best demos that I’ve seen, however, it still is complicated to navigate:

Spatial Data

Existing Problem: Traditional 2D trainings provide limited depth in performance data, typically through methods like MCQs or free-form questions. In contrast, spatial training can track head, hand, and click movements, offering a more detailed understanding of user engagement. For example, it can assess how accurately a user turns a knob in a frontline training scenario, including their gaze direction. However, many existing immersive learning experiences lack the capability to capture such detailed data consistently.

Developer Painpoint: While headset SDKs provide access to detailed tracking data, developers often lack the tools to effectively analyze and utilize it. This includes tracking head, hand, and click movements, as well as object manipulation events. The absence of a standardized data abstraction and analytics framework leads to inconsistent or underutilized data analytics in immersive learning experiences. Many developers either don't leverage this data due to its complexity or do so inconsistently, failing to fully exploit the potential of spatial training analytics.

Here’s an example of how head gaze data can be helpful in an immersive training:

Skills Abstraction & Customization of Experiences

Existing Problem: Immersive learning spans a range of institutions, from technical schools to large corporations. Developers typically create foundational, concept-driven modules for common skills like welding, electrician training, lock-out-tag-out, pallet stacking, fork lift training, fire safety etc.. While these "Off-the-Shelf" modules are useful, particularly for trade schools and larger enterprises, the true effectiveness of immersive learning at scale requires customization. Tailored experiences need to reflect the specific assets, environments, and characters of the customer's environment, as well as adhere to their standard operating procedures (SOPs). Custom content creation is a detailed and time-consuming process, involving asset mapping and editing workflows to meet customer goals.

Developer Painpoint: Developers use a variety of tools, like Solidworks and Maya 3D, for asset creation. While emerging tools like Polycam and Luma employ AI and LiDAR for asset generation, the overall development lifecycle remains lengthy. The application landscape is fragmented, leading to substantial time spent on asset customization. Smaller development teams, with 5 to 50 developers, may lack the comprehensive skill set required for such detailed customizations.

Application Packaging & Distribution

Existing Problem: Unlike wellness or gaming applications, which are often a single, unified experience, immersive learning applications typically follow a course → module → experience structure. A single course might include multiple sequential experiences, leading to large, cumbersome applications due to increased size, complicating distribution. Packaging multiple experiences into one application significantly increases its size, making distribution more challenging.

Developer Painpoint: On Android, developers either bundle multiple experiences in one app or divide the course library into several smaller apps. Both approaches can frustrate end-users due to the inconvenience of managing multiple apps or one big app. Although WebXR is emerging as a potential solution with its on-demand web app approach, it faces challenges in interoperability across browsers, limited functionality support, and latency issues. These factors contribute to slower adoption and effectiveness in delivering immersive learning experiences.

Types & Mixture of Content Types & Modalities

Existing Problem: As alluded to in my original article, there are multiple modes of 3D visual rendering including 3 DoF live action video & CG generated 6 DoF experiences. While certain trainings such as frontline productivity require a high level of interactivity such as pallet stacking wherein learning how to manipulate objects in a 3D space is crucial to the job, specific simulations / experiences require higher levels of consistency, linearity of training and may warrant a pre-recorded realistic 3D video that’s enough to do the job such as explaining a store manager on how to implement company’s value in a grocery store while serving customers. In such scenarios, realism of environments, characters and animations is key to information retention. Most trainings that exist today are largely CG 6 DoF simulations (post release of 6 DoF headsets such as Quest 2, Pico Neo 3 etc) which are either an overkill for the use case at hand, or under represent the realism of the environment, which thus reduces their overall effectiveness. Moreover, beyond just 3D, components such as spatial audio driven narration, guidance and re-iteration is an angle of “immersive simulation” that is largely missing from a lot of these trainings. Lack of an understanding of business goals and working backwards from those to define the right content type (Live action vs CG 6 DoF), as well as lack of appropriate narration audio tends to make experiences less effective.

Developer Painpoint: While developers might be capable of embedding high fidelity environments, characters & experiences, and even audio narrations using existing toolkits and asset stores, they lack the understanding of what type of content as well as modality of narration is best suited for the specific experience. Additionally, they might now be adept with methods for 360 video production and editing which is why most experiences that exist today are largely CG. We also generally observe for missing or slower feedback loops between developers and customers, to fine tune the simulations for achieving business goals.

Identity Management

Existing Problem: Most enterprise applications use employee organizational accounts, typically managed in Active Directory, for identification. Conventional training applications like Cornerstone and Workday employ Single Sign-On (SSO) methods facilitated by tools like Okta, PingFed, and Azure AD. However, many enterprise VR applications either lack SSO functionality or lack a standardized implementation method. Manual entry of email IDs and passwords, especially using VR controllers on QWERTY keypads, leads to user frustration and increased login failures.

Developer Painpoint: Developers have varying approaches to authentication in VR applications. Some use login codes generated through companion mobile or web apps, while others combine email and login codes. The absence of a unified SSO layer in VR applications creates user identification and security challenges. Collecting Personally Identifiable Information (PII) like emails and passwords necessitates compliance with various regulations, which developers may not always fulfill. Additionally, integrating SSO systems often requires a mid-tier authentication service to establish a connection between the VR device and the enterprise's active directory, complicating the process.

Bridging the demand and supply gap and developer pain points

As stipulated above, while there is a ton of “Off-the-shelf” immersive learning content out there which may or may not have the right level of quality to accurately meet the demand from the field, there still seems to be gaps in terms of what the market needs and what developers are building. This can be characterized from the following angles:

• Demand - Supply Mapping: There's a mismatch between market needs and available immersive learning content. The focus should be on creating simple, consistent, and linear training experiences that address the actual pain points and requirements of the immersive learning market. Aligning demand with supply involves effective demand generation strategies and incentives for developers to create experiences that meet these specific needs.

• Monetization Model: The common 30% revenue share in marketplaces poses a significant challenge for developers in realizing value from their immersive learning content without having the right toolkits to customize and continue to enhance it to meet evolving customer needs. A more favorable monetization model could encourage the development of higher-quality content.

• Developer - Customer Engagement: The capability to eventually directly engage with end developers to deliver feedback on off-the-shelf experiences and articulating requirements on what needs to be customized to reflects their specific organizational standards & procedures can eventually create long term value and customized content libraries for such organizations.

• Developer Tools: Providing developers with comprehensive toolkits can simplify the creation, customization, distribution and analytics of immersive experiences. These tools should address both software and hardware aspects, enabling developers to easily adapt content to specific customer requirements.

How can developer tools help bridge the gap?

• Default Passthrough: Enabling developers to use passthrough with spatial UI anchors as the default acclimation view upon entering an app enables users to view the environment, one’s body and hands upon wearing the headset, which creates a higher level of comfort for users, that facilitates them to easily pick up the controllers, adjust headset fit, set safe boundary etc. The better the mixed reality passthrough, the more welcoming the experience would be. The Meta Quest 3 for example, now defaults to passthrough mode and not immersive mode which improves comfort upon first wearing the headset.

• Standard Acclimations: Providing developers with standard UI elements / toolkits that they can embed within experiences for users to acclimate to the headset and controllers, set headset parameters creates consistency across experiences beyond immersive learning. A few of these walkthroughs include tweaking headset fit, inter-pupillary distance (IPD), controller calibration etc. Leveraging as many ghost animations, spatial anchors for annotating controllers, quick tips and examples create a very simple and guided acclimation experience.

• Simplifying Controller Interactions: Considering the varied ways developers map controller buttons to operations in VR environments, there's a need to simplify these interactions. The goal is to maintain the essence of the learning concept while making the experience more user-friendly. In specific applications like a pallet stacking simulation, it's more important for learners to understand the placement of objects (primary task) rather than the mechanics of grabbing (secondary task). Simplifying this process can enhance the learning experience. A practical solution is to abstract complex point-and-grab actions into a more intuitive drag-and-drop mechanism, utilizing a single trigger button. This approach mirrors familiar real-world interactions, like using a computer mouse, making it more accessible for learners. Creating such simpler abstractions for developers to use out-of-the-box can largely reduce friction.

• Hand Tracking & Advanced Gestures: The Vision Pro’s gaze and pinch gesture has opened up a completely new paradigm of human-machine interaction wherein one doesn’t need a controller & a beam to point & click but can rather interact with objects by simply looking at them and pinching. In context of enterprise learning, this makes navigation a lot more easier and accurate and such advanced hand tracking is especially helpful for object interactions and manipulations for experiences that require such interactions. Moreover, the capability to seamlessly switch between controllers and hand tracking also creates ease of use and flexibility between simple and more complex experiences which is also now available on the Meta Quest 3.

• Automated Walkthroughs / Quick Tips: Similar to how web application walkthroughs / guided toors exist in the 2D web era, walking a learner through complex interactions within the 3D experience through ghost animations or playback video such as picking a box & placing it in another spot, turning a knob or transposing to a specific location within the 3D environment creates clarity. While this may sound very trivial, superimposing this in an experience requires custom development for every single action. The capability to expose tools that auto-create & standardize such scenarios based on the specific interaction within custom experiences is beneficial for developers to quickly embed such walkthroughs w/o needing to design them for every action.

• Embedded Authoring & Script Creation Tools: The capability to provide developers with creative tools that enable them to design immersive learning experiences guided by script designs that follow industry wide instructional design frameworks largely eliminates the need for them to hire explicit instructional design resources. Given the latest advancements in generative AI, such authoring co-pilots powered by LLMs, either specifically trained on such frameworks or fed with context specific to such frameworks can assist developers with defining learning friendly scripts

• Standardized Analytics Pipelines: Equipping developers with standard data analytics pipelines & models that are able to harness head, hand & click data to build user engagement heat maps at stipulated time intervals, standard voice analytics models to analyze sentiment, tonality, speech rate, conduct summarization etc. out-of-the-box reduces the friction, effort and resources that developers would have to otherwise spend on building such analytics dashboards. Providing developers with custom events that they can embed within their interactions to track granular interaction data, and providing configurable dash-boarding services that they can either embed within their own apps, or expose to other applications to showcase real effectiveness of such trainings adds a level of depth to how such analytics are presented and consumed by the end user.

• Automated Asset Generation and Editing: Given the latest advancements in 3D asset generation with Neural Radiance Fields (NeRF), providing developer with toolkits that can map real customer environments and objects to generate high fidelity assets that are editable to meet customer needs has the potential to accelerate custom content development that requires high fidelity replication of 3D assets within the mixed reality experiences.

• Enabling Faster Mixed Reality Development: Mixed Reality development may involve building immersive learning experiences that leverages real life objects as a part of the augmented trainign experience. In such cases, facilitating developers with out-of-the-box computer vision models that enable them to automatically annotate a wide array of objects creates enormous time savings since it can potentially eliminate the need to train customized models for such scenarios. Segment Anything by Meta AI is a great advancement in this area which can possibly accelerate adoption of mixed reality training experiences.

• Web App Distribution: Equipping developer with toolkits that expose the capability to author and publish their experiences using protocols such as WebGL to render those experiences in a responsive way and WebXR to publish them as web apps rather than bundled APK (Android Application Package) can make consumption of such trainings much easier, provided that the headsets have adequate browser support for rendering such experiences with high fidelity and low latency.

• Types & Mixture of Content Types & Modalities: Enabling and empowering developers with instructional design frameworks that enable them to better understand the right modality for achieving a specific business goal for a customer, is key to facilitating development of more effective experiences. This can be accomplished by standard approaches such as developer discussion forums, webinars. Building developer toolkits that facilitate piecing together both live action and CG experiences, or piecing hybrid experiences (realistic environments with virtual characters), using a unified toolkit creates diversity of modality. Moreover, providing toolkits that merge both visual text & audio based narrations (Using OOTB text-to-speech models and APIs) enables developers to embed such multi-modal interactions within their experiences with minimal effort or custom development, thus making them more effective.

• Standardized Identification Templates: Providing standard toolkits to developers that facilitate them to leverage the existing user account on the headsets (such as a Meta account), as the source of user identity eliminates the need for them to explicitly plug-in a login flow within app experiences. Extending such functionalities to generate unique codes within the headset’s companion apps as a means to conduct MFA also makes this method more secure. Lastly, providing the capability to link the headset account to a customer’s active directory / in-house IdP facilitates enterprise single sign on functionality.

Wrapping Up…

Mixed reality has come a long way from tethered headsets powered by desktop compute to fully untethered headsets with better form factors that provide incredible passthrough, advanced hand tracking and smoother software experiences that facilitate easier user adoption. Finding high ROI use cases such as immersive learning both from a consumer and enterprise lens can greatly spearhead mass adoption of head mounted devices. Leveraging developer ecosystems to go-to-market faster with high quality content is key to the growth of the industry. This makes it quintessential to empower developers that are adept at building gam-ified experiences to contextualize their content for the end user persona, as well as the use case. While immersive learning is an example that has been dissected in this case, the power of such marketplaces powered by helpful developer toolkits released in such adjacent vertical markets has historically created inflection points in adoption for new hardware advancements and will likely play a major role for mixed reality devices.

Share

Leave a comment

Subscribe now

• Voice input/output features in ChatGPT's latest release have significantly enhanced user interaction, suggesting a transformative potential for voice search amidst the current bandwidth and computing capabilities.

• Historical progression of voice assistants, starting with ELIZA to the advent of Siri, Alexa, and others, has paved the way for voice interaction to become a mainstream method of engaging with technology.

• Voice chat offers critical advantages, including omnipresent access to information (ubiquity), enriched auditory experiences via spatial audio, efficient information exchange outpacing traditional typing speeds, and a nuanced understanding of context and continuity in conversations.

• In terms of practical application, voice-driven interfaces promise to revolutionize both consumer and enterprise sectors, with use cases ranging from on-demand specialized services to enhanced cross-device communication, mirroring natural human interactions with professionals.

• The current landscape is ripe for voice-driven interfaces, given the substantial improvements in computing power, network bandwidth, and the quality of speech input/output devices.

• Future developments in voice interaction are expected to focus on deep personalization, seamless integration across multiple devices, and the ability to interpret non-verbal cues, further enhancing the natural nature and empathy of human-machine communication.

Inspiration

I recently tried the voice input / output features from the latest ChatGPT release, focussed on having a continuous conversation instead of typing text on the mobile app or the desktop / laptop. It was an enthralling experiences for me and having spun up 100s of voice conversations on the mobile interface through voice and image mediums, I realized that this has facilitated an increase and improvement in my interactions, while opening up the neural pathways in my brain that were othewise unexplored. I wanted to dissect and articulate how voice search could be transformational, given where we are with bandwidth and compute availability in modern internet era.

Walking down the history lane

Since the internet's inception in 1983 and the rise of the World Wide Web with over 100,000 websites by 1995, text and images have dominated as the primary means of communication. The first voice assistant, ELIZA, was created by Joseph Weizenbaum at MIT in 1966, but internet-based voice assistants like Alexa, Siri, Cortana, and Google Assistant only gained prominence between 2014 and 2016. Initially used on laptops, these assistants transitioned to phones and smart home devices, commonly for setting reminders, sending texts, making calls, controlling smart devices, and performing basic web searches. Recent advancements in this field owe much to improved speech-to-text models, high-performance computing (GPUs & TPUs), and enhanced internet bandwidth.

Drawing an analogy

The analogy between indexed internet search and conversational AI-driven search, especially with the introduction of voice, can be summarized as follows: In pre-internet times, information was disseminated through libraries, the central knowledge repositories containing scrolls and books, akin to modern databases which were indexed by librarians and read by scholars & tutors. Personal tutors, knowledgeable in these texts, provided verbal instruction, crucial in societies with low literacy and a strong oral tradition. ChatGPT can be looked as resembling an ancient well-read scholar, uses its extensive training data to answer queries, similar to historical advisors. Voice interaction echoes this oral tradition, offering a natural way to access information, akin to learning from a tutor. However, historically, such resources were often limited to the privileged few, whereas now, having a virtual tutor accessible anytime and covering a vast array of topics democratizes knowledge, especially for the least privileged.

What’s interesting about voice chat?

The four key things that currently make or can make voice chat transformational are ubiquity, spatial audio, improved bandwidth of information exchange and contextual understanding & continuity. The sum of these parts exceeds their individual value:

• Ubiquity: The mobile revolution has made it possible to access information anytime and anywhere, enhancing internet interaction. I use mobile apps like Google Keep and Assistant for on-the-go information search and thought journaling. This often comes from thoughts sparked while walking, driving, or socializing. Outdoor activities, like walking, boost brain function and creativity due to increased blood flow and sensory stimulation. It also promotes neuroplasticity – the brain's ability to form and reorganize synaptic connections. People often walk to clear their thoughts and then record them on a device. It’s also common, albeit inconvenient for people to walk and read a physical book, Kindle, iPad or their phones to gain information. However, shifting attention to a device can disrupt the thought process and learning. Using voice interaction can reduce this disruption, by simplifying the communication to a natural conversation, which is very palpable to humans.

• Spatial Audio: Building on the previous topic, Apple's AirPods, is poised to generate over $14.5 billion in 2023, placing it greater than Twitter, Shopify or Spotify. They enhance auditory experiences with spatial audio, which stimulates brain areas linked to spatial and auditory processing. While we’re scratching the surface of voice search using ChatGPT, spatial audio has the capability to conduct a seamless back and forth conversation between the human and assistant, rendering & ingesting information using earbuds or other voice outputs, deeply integrating with one’s computing devices, mimicking how we experience sound in the real world by engaging the brain's auditory processing pathways more effectively than traditional stereo sound. Meta’s Rayban smart glasses is a fantastic example of this, wherein one can chat with Llama 2 powered Meta AI while wearing the glasses on the go. The triggers can be both user initiated, such as “Hey Meta!” for the Rayban glasses, or service initiated such as notifications on one’s iPhone casted to AirPods. While ChatGPT is solely user initiated at this point, there is a ton of exploration to be done, to find various triggers of voice initiated human machine interaction. Bill Gates also talks about this in his latest blogpost on AI agents along with various use cases. Bill Gurley & Phil Rosedale also talks about the importance of spatial audio in this Invest Like the Best episode with Patrick O’Shaughnessy, when talking about the audio metaverse and why applications like Clubhouse & Twitter spaces bring this concept to life.

• Improved bandwidth for information exchange: For someone with large fingers, typing on a QWERTY keyboard can be inefficient and error-prone, hindering activities like web searches or messaging. Conventional keyboards may not effectively convey speech nuances and can be difficult for those with disabilities. Voice interaction, a more natural method, can be faster and more accessible, allowing multitasking and reducing physical strain from typing. The average person speaks at approximately 150 words per minute, while the average typing speed is around 40 words per minute. Voice communication is a natural human ability, requiring no special training or adaptation. Furthermore, voice interaction greatly enhances accessibility. Combining ubiquity and efficiency, it enables multi-tasking efficiently wherein one is able to retrieve information while cooking, driving or exercising.

• Contextual understanding & continuity: While modern day search engines facilitate recommendations that get personalized based on one’s search history over a period of time, voice on the other hand is able to transform information retrieval into a continuous chain of thought, which takes into account previous conversations as almost perfect memory (commonly known as context window). For example, instead of starting a fresh search for a topic that I’ve previously searched on, I pick up a former conversation from my chat history to re-visit and build on the chain of thought. While this may reduce the diversity of thought in some cases, it helps retrieve historical CoTs and builds on them, fully utilizing the initial context, and maintaining continuity over multiple interactions. Essentially, one is continuing where they left off in a very coherent manner, that could potentially be better than raw web searches.

Why now?

To understand why now is the right time for voice driven interfaces, let’s think about this from a network bandwidth, compute and input / output quality lens:

• Compute: Running speech recognition, conversion to text, output inference and conversion from text to voice requires a significant amount of compute due to the resource intensive nature of models behind them and the latency constraints. Modern day GPUs can handle these tasks more efficiently and at a larger scale than ever before, making real-time voice processing feasible for widespread use.

• Network Bandwidth: Standard MP3-quality voice data transmits at about 128,000 bits per second which warrants internet speed of approximately 0.128 Mbps. Early internet connections, like dial-up, were painfully slow, operating at speeds around 56kbps. Downloading anything substantial, such as a song or a movie, could take hours​. In contrast, the average internet speed in the U.S. in 2023 is 171.30 Mbps, and globally, the average speed has reached 45.6 Mbps, thus facilitating faster and reliable audio streaming.

• Quality: Speech input (microphone) has taken massive leaps on sensitivity, noise cancellation and beam-forming which makes the microphone sensitive in the direction of speaker’s voice. For voice output, the quality and texture of sound from modern day consumer devices has significantly improved in terms of fidelity. Apple introduced spatial audio with dynamic 3D tracking for Airpods through a firmware update in September 2020, while Sony released the 360 Reality audio in late 2019. Such monumental changes in audio rendering open up the possibility for democratization of voice driven UIs.

• Ubiquitous devices: Voice driven interfaces cannot go through an inflection point on their own without a paradigm shift in hardware adoption. While, we’re more than a decade into the mobile wave, we’ve finally started seeing scale adoption for wireless earbuds, smart watches, smart home devices and may see a growing trend for smart glasses. Although, sales for gaming oriented virtual reality still sees a rather slow or flat growth curve, however, Apple’s step into mixed reality via the Vision Pro will be interesting to see. The graphs below showcase the last 5 - 6 years of active users / units sold for such devices which signifies a step adoption rate thus making a stronger case for apps that can now fully leverage voice capabilities.

  

The humane pin was launched recently by former Apple designers and pre-dominantly indexes on gesture (hand / palm pinches) and sound as a medium of interaction. While the product success is yet to be discovered, this is a great example of a device that introduces ubiquity and spatial audio while being powered with ChatGPT as it’s conversational interface that switches between a holographic style and a voice interface.

Enterprise Use Cases

The initial few iterations of voice driven UIs might very well be designed as co-pilots on top of existing visual interfaces as we’re seeing with most AI driven assistants today, however, over time, as the hardware adoption for ubiquitous devices becomes more mainstream, voice driven user interfaces may become more and more native to specific types of devices. Here’s a simple visual that illustrates the same for both external and internal use cases.

Consumer Use Cases

In the. consumer scenario, the easiest way to think about use cases is to think about any particular service that you seek from a professional such as a teacher / tutor, health practitioner, financial advisor, travel agent etc. The fundamental use case can involve the use of a conversational interfaces, since voice is the most natural medium of interaction that mimics how we interact with such personas in our present day life. The core benefits of having such specialized agents include on-demand access, which means that you would be able to talk to your AI therapist at 4 AM in the morning. Each of these patients will be incredibly knowledgeable, infinitely patient and empathetic which is the highest standard of what one could expect from this persona.

Underlying architecture

Given that large language models are very flexible, with the ability to solve a wide range of generative tasks, we’ve recently seen some great advancements in large language models focussed on speech understanding & generation. For eg. Google’s AudioPaLM fuses text-based and speech-based language models, PaLM-2 and AudioLM, into a unified multimodal architecture that can process and generate text and speech. The following is a representative architecture diagram of AudioPaLM. This may significantly streamline input & inference and based on recent research, automated speech recognition (ASR) models built by augmenting audio encoders in the input architecture seem to have a higher accuracy over text based models, especially for multi-lingual data which is a crucial aspect of design given the diversity in language spoken.

Prominent Companies

As of the writing of this article, the core innovation seems to be happening at the hardware & infrastructure layer while we’re yet to see more innovation on the end user experience layer:

• Hardware layer: There almost seems to be a commoditization wave for hardware supporting spatial audio including earbuds, smart home devices, watches and smart glasses including the incumbents such as Apple, Google, Meta and Amazon. This is what is currently setting the precedent for a possible future wave of audio native AI services.

• Software Infrastructure: This layer empowers the end user applications through voice transcription (speech to text), generation (text to speech), overlay & editing. This also seems to be a commoditized space but a lot of innovation is shifting from accuracy of output (which has been largely optimized for already), to personalization, building natural voice interfaces and multi-lingual support that helps improve efficacy of such interactions.

• Broad AI Assistants: As of the date of this writing, Google Assistant, Alexa and Samsung’s Bixby are some of the most prominent AI assistants that have been around for a while. ChatGPT’s latest functionality for speech ingestion and generation is state of the art, especially with the efficiency, depth, verbosity and human like nature of the conversation that leverages GPT 4.

• Enterprise Voice Assistants: While voice analytics tools for sales & support such as Gong, Cresta, Clari, Obersve.ai, kore.ai have been around for some time, we’re seeing new tools for complex meeting summarization such as Otter.ai & Fireflies as well as competition from incumbents such as the Zooms AI companion & Teams assistant.

• Personal Voice Assistants: This area is a massive greenfield and we’re just starting to see more traction with AI doctors & speaking coaches such as Elsa AI

Thinking about the greenfield

While we might be at the inflection point for voice based interfaces to be more prominent, the following advancements are yet to be seen, and may bring tremendous value add:

• Personalization: As of the writing of this article one can choose from 5 different tones within ChatGPT, however, deeper personalization may involve training models on a user defined tone and voice through historical data. Personalization in content can also stem from elongated context windows from past conversations and not merely the two custom instruction questions that ChatGPT allows a user to pre-fill today. This can be similar to how a personal tutor gets acquainted with a student’s style of listening or ingesting information by practicing visual learning methods, drawing common analogies to specific concepts, re-iteration etc. Moreover, Custom GPTs open up the possibilities to contextualized conversations via uploading pre-determined set of knowledge bases to achieve responses specific to the topic at hand.

• Understanding non-verbal queues: Not all aspects of one’s dialogue can be expressed in verbal cues. Emotional queues, tonal variations and speech patterns can speak an additional 1000 words (metaphorically), and thus a plain speech to text conversation fed as model input for next token prediction has limited expression. Future voice models may include the ability of such interfaces to alter the response based on such cues to make responses more empathetic by adapting its tone, or altering the content to create a more natural and human-like conversation.

• Continuous Ubiquity & Deep Integration: Similar to how Apple allows a user to switch Airpods between a Macbook and iPhone and use various apps (For eg. messaging) across devices, ubiquity in voice conversations can also be facilitated across several devices including wearables such as smart glasses, smart watches, mobile phones and laptops, easily allowing switching between text and voice based mediums, thus maintaining continuity of conversation / CoT. Ubiquity can also be achieved between different AI services where exposing context from one service may be beneficial for others. For eg. what you personal AI nutritionist recommends might be important for your AI doctor. However, this warrants a strong security framework to ensure user privacy and this broadly applies to all AI applications.

References

• AI is about to completely change how you use computers - Bill Gates

• Bill Gurley & Phil Rosedale - Back to the Future [Invest Like the Best, EP. 254]

• AudioPaLM - A large language model that can speak and listen

• Prompting Large Language Models with Speech Recognition Abilities

Share

Leave a comment

In continuation to part 1 of the three part series, this part delves into the components of the Total Cost of Ownership (TCO) for AI co-pilots, including licensing, setup, data, infrastructure, change management, performance monitoring, security, and compliance costs and examines the factors that contribute to the accrual of business value for AI co-pilots through various phases.

A re-usable tabular version of this part of the framework can be found on this Coda page.

Executive Summary

• The initial setup and integration phase is crucial, considering the diverse infrastructure and security standards in enterprise environments. Key considerations include deployment architecture, activation, installation, distribution, integration with existing systems, interoperability, private cloud/on-premise/hybrid support, and user management.

• Configuration and change management are ongoing processes, requiring the ability to fine-tune input and output formats, incorporate human feedback, and maintain brand consistency & tonality. Scalability is essential, with attention to increased transaction volume, intensity, types of transactions, and concurrent users. Monitoring and management tools play a vital role in overseeing AI co-pilots at enterprise scale.

• Security and compliance are paramount, with measures such as securing model parameters, federated learning, input and output validation, data storage, and adherence to industry standards. Transparency and accountability are necessary to understand the decision-making processes of AI co-pilots.

• The ecosystem and network effects are highlighted, emphasizing the benefits of sharing privately hosted models, federated learning, data and integrations marketplaces, and the importance of aligning customer demand with available supply.

Total Cost of Ownership (TCO) for AI Co-Pilots

Total cost of ownership is a commonly used metric for characterizing the value of adopting and using a SaaS solution over its entire lifecycle. TCO takes into account cost components beyond the subscription or licensing fees, helping organizations make informed decisions about the true financial implications of adopting a specific software service. Putting AI co-pilot in perspective, here’s how this cost can be broken down

Licensing Subscription which has typically followed a per user model, based on what we know from Microsoft co-pilot and OpenAI. This cost however may not may not linearly increase with incremental users or transactions due to volume discounts.

Initial setup / implementation costs cover the initial configuration, integration, data migration, and training required to get the co-pilot solution up and running. This may include consulting fees, customization, and the time spent by a customer’s IT team. In context of AI co-pilots, the following costs may apply, especially for scenarios where private models are developed and deployed within customer environments:

• Data Acquisition, Storage and Processing costs for obtaining high-quality data for training and validation, storing, pre-processing and cleaning.

• Data labelling costs as a pre-requisite for training high quality models.

• Model development costs

• Deployment and testing costs

Infrastructure Costs includes cost of any internal infrastructure that may be needed by the customer to maintain / manage the solution. This may include:

• On premise / private cloud compute costs for hosting private model inference.

• Cost of GPUs for localized training / private model deployment and management

• Costs to integrate the AI-driven software into existing systems and workflows.

Change Management Costs are incurred for maintaining the solution on an ongoing basis. Putting AI driven co-pilots in context, this includes:

• Costs for training employees and users on how to interact with and benefit from the AI-driven software

• Additional configuration costs required to make changes to the AI co-pilot as the business process evolves and to fine tune the co-pilots.

• Performance & monitoring costs for tools and services tracking the performance and accuracy of AI co-pilots

Security & Compliance Costs include any costs needed to adhere to enterprise security and compliance standards.

• Security related expenses for implementing security measures to protect AI models and data.

• Compliance adherence costs related to complying with data protection and privacy regulations, such as GDPR or HIPAA.

The more compliant the solution is, the less costs are incurred. The other side of these costs includes costs incurred in the event of a non-compliance or security incident, adjusted for probability of such events.

Initial Setup & Integration

While the modern SaaS stack has made the introduction of a new tool or service considerably simpler, enterprise environments still adhere to varying infrastructure and security standards. The effectiveness of a product feature is not merely a function of the value that it’s creating to the end user but also the ease with which it can be implemented / introduced. This can be driven by the following factors:

Deployment Architecture: Co-pilots can utilize publicly available APIs from OpenAI, Anthropic, Cohere or opt for an in-house model. Efficiency in building a SaaS or PaaS architecture is crucial for rapid adoption. This encompasses specifics like selecting cloud services for co-pilot interaction and optimizing traffic flow through internal firewalls and proxies. Customers usually adhere to a strict zero trust firewall policy, potentially blocking traffic to public LLM services by default, necessitating an enhanced networking design for seamless connectivity.

Activation, Installation & Distribution: For AI agents installed locally on user workstations or browsers, creating a seamless distribution and installation channel streamlines adoption. Additionally, when these agents execute tasks locally, it's essential to assess and assign the necessary privileges for specific functions. This consideration becomes more relevant when the agents overlay existing applications rather than being deeply integrated into them. In the latter case, configuring the source applications to exchange data with the agent may be required, making a user-friendly activation process is essential for boosting usage.

Integration with customer’s business systems: Integration with existing enterprise systems is often a pre-requisite to adoption of any SaaS software, not just a co-pilot. Some of the most common pre-requisites are:

• Integration with existing customer databases, which may be hosted on premise or on private cloud instance.

• Integrations with systems of record, reference, action and insight.

• Integrations with customer’s credentials management system such as Cyberark.

In these cases, an agent's capability to quickly integrate is typically achieved by:

• Using pre-built connectors based on target system APIs that reduce cost of building custom integrations

• Allowing customers access to agent APIs and event data for custom connectors.

• Configuring systems for data scraping when no integration is available. The capability to fine tune and configure such tools also streamlines integration.

• Implementing mid tier agents for scenarios where customer systems are locked down in on premise environments

Interoperability: For enterprise co-pilots, supporting legacy browsers and operating systems, like Edge, IE, and older systems, is vital for sectors like finance and healthcare, where technical debt is common. This broad compatibility accelerates adoption, especially in cases where customers have strict security settings, such as disabled JavaScript on browsers.

Private Cloud / On-premise / Hybrid support: In traditional and highly regulated enterprises, there's often a reluctance to use public cloud-based co-pilots, particularly when dealing with sensitive data. To accommodate these concerns, customers may opt for co-pilot installations in their private cloud or choose a hybrid setup, keeping data sources internal while processing in the public cloud. Google Cloud's Vertex AI, for instance, offers support for such deployment models.

User Management: Most enterprise applications require users authentication to integrate with their identity providers (IdP) such as Okta, Azure AD, Auth0 etc. The capability of a co-pilot to quickly integrate with customer’s Single-Sign-On (SSO) systems also contributes to initial setup costs.

Licensing: In a seat / user based licensing paradigm, specific customers may allocate co-pilot licenses on a rolling basis to facilitate moving around licenses between different teams based on the timing of use. A great example of this is a business process outsourcing provider for outsourced support services that may leverage co-pilot licenses for a team in the Pacific region and would then want to re-allocate those licenses to the team in the EU region. While this floating model can play against software providers, such enterprise wise licensing agreements can be largely priced based on business cases and thus moving around seats may not affect the total contract value. Such flexibility in licensing creates an ease of adoption.

Configuration & Change Management

The co-pilot may require ongoing calibration / tuning to achieve higher levels of accuracy on the existing use cases or to accomodate a broader variety of use cases. Moreover, business processes are always bound to change, thus creating the need for users to accomodate those changes to maintain accuracy & compliance.

Tweaking Input & Output Formats: AI Co-pilots may leverage a standard set of user configured prompt templates as inputs and generate outputs in a pre-configured format which eventually feeds into downstream systems that expect them to be in a that format. Since business processes and systems are always subject to change, exposing the capability for user to quickly configure and eventually change input and output templates streamlines the time and cost for building and managing such co-pilots. Examples of input formats include the capability for users to inject only questions, or additional documents along with questions. Examples of output formats can include JSON, XML, CSV etc.

Fine Tuning via Human-in-the-loop: This applies to AI co-pilots that may use multiple methods of fine tuning an output such as supervised fine tuning or reinforcement learning. Fine-tuning in AI co-pilots, especially those employing methods like supervised fine-tuning or reinforcement learning, hinges on key factors for effectiveness:

1. Ad-Hoc Training Data: The ability for customers to upload tailored training sets, such as use-case-specific documents or chat transcripts, and seamlessly fine-tune out-of-the-box models to enhance accuracy for specific tasks.

2. Human Feedback Loop: Incorporating human feedback into ongoing outputs is crucial. For instance, a co-pilot aiding an accounts payable specialist can present uncertain invoices for human review within the source system (e.g., NetSuite). Changes made are tracked, with possible follow-up questions for context, all seamlessly integrated with existing workflows. This minimizes friction and enriches the context required for optimal performance.

Brand, Tonality & Moderation: For co-pilots generating content for widespread consumption, maintaining brand compliance is crucial. This involves adhering to specific design elements, language guidelines, and desired tonality, both for internal and customer-facing materials. Effective co-pilots offer customizable settings and utilize contextual brand data to seamlessly integrate these requirements. Additionally, they should allow easy editing and adaptation to changes in brand and tonal preferences, ultimately reducing maintenance costs.

Scalability: As co-pilot software scales across various users, use cases and diverse data sets, scalability encompasses several aspects such as:

1. Increased transaction volume per user, for eg. processing higher number of invoices per AP co-pilot.

2. Enhanced transaction intensity, for eg. as processing more pages per invoice by an AP co-pilot.

3. Expanding the types of transactions handled, for eg. an accounts payable co-pilot managing purchase order and invoice document types.

4. Growing number of concurrent users leveraging the co-pilot.

Scaling these transactions can lead to substantial compute demands unless optimized. OpenAI employs distributed computing, load balancing, auto-scaling, resource pooling, caching, prioritization, and algorithmic resource management to handle concurrent requests efficiently. Scalability profoundly affects pricing, as discussed in part 3. In PaaS scenarios requiring customer-side infrastructure, the incremental cost of customer-side compute infrastructure raises total ownership costs. The rate of cost increase with linear scalability impacts ROI, making solutions less cost-effective at scale.

Monitoring & Management: The capability to monitor the usage and activities conducted by AI co-pilots across various users, bodies of work, systems and functions, supplemented with the capability to initiate, terminate or edit the existing workflows is key to effectively managing co-pilots at enterprise scale.

Security & Compliance

Heavily regulated customers such as financial institutions have some of the highest standards for enterprise security. Given the architecture of modern day co-pilots that relies heavily on data for training and fine tuning, having stronger security controls becomes more evident. Here are some key pillars to keep in mind to minimize the cost of compliance and rather the financial & reputational risk from non-compliance

Securing Model Parameters: For co-pilot software with custom / open source model deployment, some commonly used approaches to maintain high levels of security are:

• Encrypting model parameters to protect learned knowledge.

• Restricting access for model parameters to authorized users/systems.

• Using HSMs/key management for secure key storage.

• Utilizing techniques like zero-knowledge proofs for secure computation on parameters without revealing them.

Training on Sensitive Data: Co-pilot software can leverage techniques such as federated learning, where the model is trained collaboratively without sharing sensitive data, reducing the exposure of confidential information. I’ve covered this in more detail in the ecosystem section below.

Input & Output Validation & Sanitization: To mitigate injection attacks and prevent inappropriate or harmful content in LLMs trained on a broad dataset, specific techniques include:

1. Thoroughly validating input data by checking data type, length, format, and range.

2. Developing custom filters and rules to flag unacceptable content, such as hate speech or personal attacks.

3. Implementing human review for sensitive or high-risk applications, leveraging human judgment that automated filters may overlook.

4. Creating feedback loops where user-reported issues and false positives/negatives from content moderation enhance filtering mechanisms and train the model to produce more suitable content.

Information Storage & Retention: For SaaS software, secure storage and retention of PII & PFI data, if collected, are paramount. This entails securing data both at rest and in transit using industry-standard protocols. Additionally, clearly communicating data retention policies and empowering users to manage their data and retention preferences are essential for compliance and adoption. While not a new concern, efficient implementation within the modern LLM stack significantly impacts usage and compliance costs.

Industry Standard Compliance: In continuation to the point above, adherence to standard certifications / data security frameworks such as GDPR and CCPA are key pre-requisites to enterprise adoption.

Transparency and Accountability: LLM outputs combine probabilistic elements with deterministic rules and customer system knowledge, which can yield inconsistent outcomes for the same input. To address this, most enterprises seek accountability and require an audit trail to understand the rationale behind a co-pilot's decisions based on the given data. Exposing the co-pilot's workflow and methodology, including how inputs were processed by multiple models or deterministic systems, is crucial for internal and external accountability. This can be presented as audit trails or decision flows, similar to how a service manager for example audits support ticket transcripts. Understanding how an LLM handles customer requests, classifies them, accesses knowledge bases, and provides solutions or recommendations is vital for security and performance monitoring.

Ecosystem & Network Effects

In my article on Business Process Automation and the Generative AI Turbocharge, I highlighted that co-pilots can be implemented in very similar ways across end users from different organizations which creates redundant implementation & fine tuned work, that can be streamlined to create quicker adoption and reduce cost of ownership. In context of modern day AI co-pilots, such commonalities can be attributed to fine tuning of models, data used and systems that the co-pilots integrate to create quicker adoption:

Model Ecosystem: A model ecosystem can be characterized by:

• Sharing Privately Hosted Models: This involves sharing fine-tuned models for specific use cases within a customer ecosystem, reducing the need for individual fine-tuning. HuggingFace provides an example, though it's limited to models that can be imported into private cloud instances for custom co-pilot development.

• Federated Learning: In this approach, an initial global model is created and shared among participating entities. Each entity then trains the model on their local data iteratively without sharing the data itself. Model updates are computed after each round of training and sent, not the data. A central server aggregates these updates to create a global model update, which is then distributed to participating entities. Federated learning has the potential of creating true network effects within co-pilot ecosystems while managing and maintaining customer compliance and securing data.

Data Marketplace: While enterprises are less likely to share proprietary datasets with each other, systems integrators and service providers, who offer initial implementation services, may have a stronger incentive to host and potentially monetize their collected datasets as part of their business strategy.

Integrations Marketplace: In addition to the initial set of integrations offered, there's value in a community-driven approach. Allowing customers and developers to create and share integrations fosters implementation synergies and encourages network effects. An excellent example is Zapier, which has successfully built its ecosystem around this concept.

In the modern AI landscape, the success of an enterprise ecosystem hinges on the speed at which models, parameters, datasets, and integrations can seamlessly integrate into existing co-pilot workflows and adapt to specific use case requirements. Aligning customer demand with the availability of these resources has consistently been the driving force behind the creation of thriving marketplaces.

Bringing it home

Beyond just the design of the co-pilot, product market fit, user experience and data types, understanding how easily the co-pilot can be introduced and implemented within complex enterprise environments, managed and maintained with the ever changing business language in a secure and compliant manner contributes significantly to the long term success of the software maker and the customer. Finally, the capability for such co-pilots to create an ecosystem that maps the demand for latest and greatest fine tuned models, data and systems integration to reduce the redundant pockets of implementation across the customer base, can turbocharge adoption and reduce the time to execution. The final part of this three part series will focus on the mechanisms of realization of the business value from a cost-benefit perspective, how such co-pilot software will be sold as work and not software.

References

• Thick wrapper is the moat

• Scaling Intelligence and Generative Apps

• Business Process Automation & the Generative AI Turbocharge

• Assessing the real business value of AI co-pilots

In continuation to my last post on Business Process Automation and the Generative AI Turbocharge, where I break down the concept of enterprise corporate micro-processes, spanning various levels of an enterprise (front, middle & back offices) and various types of systems (engagement, action, reference and insight), this article is an amalgamation of several evaluation frameworks that I’ve used in the past to assess the efficacy of co-pilots focussed on enterprise productivity, and the extent to which they can add business value. While the technology is rapidly evolving from rules based automation tools, to more enhanced AI powered tools that can harness unstructured data, leverage conversational interfaces, support multi-modal interactions, a lot of the business value ultimately come down to how the core problem is being solved. Adoption of enterprise AI-copilots boils down to three things:

• Does it make a business user more productive via the end-user experience over a long period of time?

• Does it make IT user incrementally march towards a goal of an integrated and a clean enterprise?

• Does it make sense from a business case perspective, from a hard and soft dollars lens, for an executive sponsor including the buyer (COO, CRO, CHRO) & approver (CFO) to pilot and expand?

This is the first of a three part series, that focusses specifically on product market fit, synergies, risks, user experience, data and systems integration. A re-usable version of the framework explained below can be found on this Coda page. Here is part 2 of this 3-part series

Key Highlights

Here are some key highlights / a shorter version to this part of the framework, however most of this is discussed in much depth below:

• Product Market Fit: AI co-pilots verticalized to specific personas, functions or industries may be more effective due to their increased capability to handle nuances and attain depth of use case. If co-pilots are broader in nature, there have to be some system, process or user level synergies / commonalities that create the capability to increase breadth while also going deep into the use case. Co-pilots have high levels of ROI when approaching simple use cases with high frequency or a long complicated use case split across several steps conducted by several FTEs in collaboration.

• Modality: Co-pilots have the capability to be more effective when they accomodate multiple modalities of input & output, specific to the use case at hand. Text / Chat may not always be the best modality by default.

• Risks: The efficacy of a co-pilot is also dictated by the financial and regulatory risks associated to the process that is in scope and may largely dictate need for accuracy as well as human-in-the-loop design.

• User Experience: Co-pilots are the most effective when they’re guided / step-by-step in their workflow, while being consistent across multiple users. Their capability to augment information, remember from memory and make suggestion driven insights makes them more compelling for enterprise use cases. Finally, editability plays a major role in ensuring that change management is seamless.

• Data Types: Diversity of data types (structured vs unstructured) as well as capacity to manage missing / lower quality data is what truly governs efficacy of co-pilots to solve use cases with high levels of accuracy.

• Systems: Capability of co-pilots to integrate with a broad depth of systems segregated by type (record, insight, action etc.), function (HR, F&A), market segments (mid-market, enterprise), as well as depth of integration via traditional API based approaches, and non-traditional data scraping approaches contributes to its effectiveness.

Product-Market-Fit, Synergies & Risks

Vertical-ization & Synergies: For AI co-pilots to perform a task in an effective way, they can either target a very specific type of use case vertical-ized by industry (eg. FSI, healthcare), function (eg. HR, F&A), or a can target a specific type of persona (eg. developer, contact center associate) or a system (eg. system of engagement). While an argument can be made that with general purpose LLMs, co-pilots have the capability to maintain depth, but also have the breadth of use cases, building for one specific use case provides long term value since it instills focus and allows the product (or the model) to capture variations and complexities of the industry, persona, function or systems involved. If a co-pilot goes across multiple use cases, this is typically effective when there are synergies across those use cases which can create depth and breadth at the same time.

Re-Engineering the Process: Often times, co-pilots may be looking to solve for processes that are fundamentally broken due to the prevalence of redundancies. For example, a review step may have been conducted twice in the current state, to account for human errors, however, the future state may only need one review step instead of two, since it could have less errors than the manual process. It’s important to assess if the co-pilot is also re-defining the target state process, while optimizing for the benefit that the underlying models bring, rather than simply mimicking the process as-is.

Depth of Process: While typical enterprise operations involve corporate micro-processing, the true value is only realized when a process is automated end to end. The solution can be architected in a way such that there are individual co-pilots for each owner of the micro-process, collaborating and communicating on across each other. For eg. A securities trade settlement co-pilot at an investment bank should be able to operate across the front office (traders), middle office (responsible for processing trade settlements) and back office (responsible for other regulatory reporting), integrating with relevant systems and relaying various information between the desks. The true valued is accrued when each of the desks have sub-co-pilots assisting them to eventually streamline the end to end process, however, to an operating leader, what matters more is cycle time & throughput of processing trades.

Modes of User Inputs: While a lot of co-pilots have largely indexed on chat based conversations as sources of inputs, depending on the type of use cases, the efficacy of the co-pilot depends on the breadth in channels of engagement. A classic example is a support co-pilot wherein customers can engage via phone, text or email. Prioritizing based on the channel of engagement with the highest volume can define how quickly a co-pilot can add business value.

Frequency & Volume: The efficacy of a co-pilot to solve for high frequency and volume use cases with the potential to translate to a high ROI can be bucketed into two categories:

• A small, specific but repeatable business process conducted by several FTEs, at a high cadence. The capability of the co-pilot to handle the same linear workflow consistently across multiple FTEs, multiple times a day, creating a highly accurate output with little variance.

• A complex process conducted across a high number of FTEs in chunks / smaller steps with multiple hand offs, reviews and high frequency of records. Effective copilots can seamlessly assign and triage these workloads across multiple FTEs in a consistent and organized way.

Regulatory Risks: Specific use cases may have regulatory requirements that may warrant forced manual steps such as a human signature on a document prior to submission. In such cases, it is important to assess how co-pilots address and work around such regulatory risks, and establish controls.

Financial Risks: Some co-pilots may be working in functions such as FP&A, wherein an inaccuracy of the processed transaction output may create a potential financial loss, thus adversely affecting the enterprise’s financial situation. In such cases, it is important to understand whether the co-pilot addresses such financial risks via mechanisms that minimize them. Moreover, understanding whether the co-pilot is consistently churning outputs with minimal variance, and ensuring that the error rate is very minimal is crucial for minimizing financial losses.

User Experience

Guided / Step-by-Step Setup: Effective co-pilots are able to breakdown a specific process into multiple steps of human-in-the-loop which allow for continuous and refined context from the user to drive the co-pilot towards a more accurate answer that’s attuned to the exact ask from the user. This may also allow for seamless step-by-step augmentation of information from source systems about the question in scope to minimize redundancy in gathering user input and induce effective reasoning prior to coming to an answer

Workflow Consistency and Collaboration: A co-pilot designed for a single process that is repeatably done by multiple team members needs to be designed to ensure linearity of user experience. On the other hand, a co-pilot designed for a process split between multiple members of a team needs to be designed to seamless intake, route and triage work items as per the stipulated process flow. This requires seamless back and forth of work and the corresponding output between processors and reviewers. This is crucial for ensuring efficient collaboration and ensuring accountability of work items. Most service lines ideally set standards for consistency, governed by SOPs for a task to be done. Thus, having high levels of consistency and controls on the probabilistic output of a co-pilot is crucial to ensuring that the team is operating within a finite margin of error.

Memory, Suggestion & Augmentation: Language models are stateless which implies that they may not remember context from past conversations. Thus, for enterprise applications of co-pilots, it’s important to augment a workflow with historical memory which can include past transactions and corresponding inputs / outputs from those transactions. Moreover, augmenting information from source systems is crucial to make the system as stateful as possible. Finally, making suggestions based on past information can add tremendous value to proactively identity any outliers throughout the operation. For eg. A co-pilot identifying a higher than usual payable amount on an invoice for a specific vendor, and triaging that to the user instead of auto processing adds a certain level of control to the AP process.

Edit-ability: Business processes are often bound to changes, which thus requires the business to edit the co-pilot workflow. Thus, providing the right product capability to make such edits is crucial for ensuring that change management is cost efficient and easy to execute on. The business vertical leveraging the co-pilot may have to edit the input / output data format / transformations, integrated source systems, order of steps, positioning of human review step, regulatory changes, configuration / thresholds etc. Edit-ability creates ease of implementation and change management, which I will cover in part two of this series.

Data Type & Quality

Diversity of Data Types: Business processes may have varied complexity of data inputs ranging from fixed values, key-value pairs or tables, to more complicated unstructured data types embedded within small to huge documents as free form texts, complex tables or highly variable key value pairs. While language models are powerful at extracting unstructured data, it’s important to understand the percentage of data points extracted and the corresponding accuracy to dictate the ROI.

Not so clean or missing data: Regardless of the co-pilots capability to extract structured or unstructured data, specific verticals / use cases have low quality data, which may involve inaccurate / incorrect data or completely missing data. For eg. A co-pilot working on automating online orders may have customer contact information that can be outdated / have missing values which can then create downstream implications on the capability to fulfill customer orders. In such cases, it is important to understand a co-pilot’s capability to handle such data quality issues and building controls to either extrapolate missing data, reference with source systems or initiate triaging workflows to human-in-the-loop interfaces complete the missing data.

System Integrations

Breadth by system Type: Enterprises leverage system of record, engagement, insight, reference and action. Each of these systems have different ways in which one can transact with the data stored within them. The efficacy of AI co-pilots depends on how the information from such diverse types of systems in scope can be retrieved efficiently across various parts of a user’s workflow to create a stateful and augmented conversation. An organization typically builds several silos across these system types and a bigger breadth creates the ability to bridge such silos. For eg. an operations team may have an important mapping pinned in a Slack thread instead of a central database which creates the need to integrate deeply with Slack.

Breadth by business function: Business processes within a specific function can be spread across a wide variety of systems such as CRMs, ERPs, HR & Payroll systems. For eg. To automate critical tasks such as a customer refund, it’s important to understand the nature of the customer via the CRM, but it is also important to understand whether the team running the process has the right privileges to issue refunds, which may from an HR system.. Each of these system types have different schemas and integration capabilities and the richness of augmenting context across such system from multiple functions is crucial for responsible decision making.

Depth of integration: Such systems can also have varying level of integration capabilities, including REST / SOAP APIs, or no integrations at all. The efficacy of a co-pilot is governed by the breadth and the depth of such integrations to systems specific to the use case(s) in scope. The trickiest integrations are the ones where legacy systems have no API capabilities which may involve screen scraping or interfacing with mid tier data warehouses. The more regulated and big the industry becomes, the less integration friendly can the systems in scope be.

Capability to augment insights and generate intelligence: Beyond what’s stipulated in the user workflow, the co-pilot may integrate with additional systems of insights such as databases to store historical records from transactions, to generate unique insights over the course of the usage of the product. For eg. A sales co-pilot summarizing meetings across a customer lifecycle also has the capability to augment insights across customers regarding common pain points as it pertains to product usage and product features requested across the customer base which then becomes a massively helpful data point for product development teams.

Addressing fragmentation across market segments: Certain co-pilots may operate in verticals / use cases that have a highly fragmented source tools used by the customer across market segments. For example, HR tools may largely vary from Gusto in SMBs to Workday & ADP in enterprise customers. Depending on the target market for the co-pilot, the fragmentation of system creates a requirement to build a large amount of integrations, thus adding complexity. Effective co-pilots are the ones who are able to address such fragmentation, if it exists and prioritize the right set of systems for their target market segment.

Conclusion & Looking Forward

The above can be leveraged as a great starting point for potential customers, investors, or product companies developing the co-pilot to deeply understand its product market fit, and rate the planned design based on the above focus areas by answering the set of questions in the Coda document. In part 2, I focus on areas such as implementation, security, compliance, customization and ecosystems which will further strengthen the assessment from a total cost of ownership perspective. In my final article, I will purely focus on the business case lens including hard and soft dollar metrics, which are majorly responsible for governing buying decisions for such enterprise co-pilots.

References

• Generative AI Moats in B2B with Emergence Capital’s Jake Saper

• Unleashing the power of Unstructured Data - Madrona

• Business Process Automation & the Generative AI Turbocharge

• Generative AI is neither hype, nor reality (yet) in the enterprise

As someone who started his career as a management consultant, ended up joining an enterprise SaaS software company as an implementation engineer, transitioned to pre-sales / sales engineering to get closer to the company’s revenue function, became a product manager at LinkedIn, and moved back to being a sales engineering director at an early stage enterprise software company, I’ve observed various parallels and overlaps between the roles of a product manager and a sales engineer. While both roles are equally crucial in making a company’s day to day operations and its customers successful, there are a ton of synergies and relevance between the two roles that eventually help in building meaningful value for end customers. I aim to dissect this by various focus areas and skillsets of a product manager that tag back to the core competence of a sales engineering function.

Who are Sales Engineers and what do they do?

Sales Engineers operate within the “Go To Market (GTM)” division of a typical enterprise software company wherein they bridge the gap between product development and sales. SEs use their applied technical knowledge of the product (typically software & hardware) to help account teams persuade prospects to choose a company’s products. Here are some key components of the role:

Technical Product Sales: Sales engineers work alongside sales reps to mobilize, architect and close enterprise deals from a technical standpoint. They are product experts who have a deep understanding of the platform from a technical and use case standpoint. They get to listen to the problem that the customers are facing and trying to solve, ideate how the product can solve it and the features that could potentially add value to the customer. While Account Executives are great at prospecting into accounts, building relationships and closing deals from a commercial lens, Sales engineers work alongside reps to provide contextualized product demos, build business cases, help customer teams understand the technical and business implications of the product. They work cross functionally across various enterprise stakeholders such as the core business vertical, IT, Security & legal to talk through different aspects of the product.

Customer Onboarding & Hand Off: While most sales engineers may primarily be involved during the sales cycle from discovery to SoW / Close, some of them are also involved in setting up customers as per their requirements. The simplest example of this would be an SE setting up a Salesforce SaaS instance for a customer. Additionally, Sales Engineers can also be tasked at handing off the onboarded customer to post sales, customer success and support teams, who then onboard and ensure that the customer uses the product in a way that maximizes adoption. This involves communication and documentation of information specific to the customer’s environment that is crucial for post-sales execution.

Internal Sales Alignment & Deal Reviews: While sales & GTM teams are typically aligned in operational units / pods typically split by geography, industry or product; sales engineering functions follow a similar model of alignment to such sales pods / units to best align with account executives. In each of these pods, SEs are responsible for communicating the health of the deal as it pertains to the customer’s product and use case fit, thus giving important signals to account team on the overall propensity of the user / buyer towards the product, which eventually helps revenue teams forecast the percentage probability of closing the deal accurately.

Product Management Partners: Given the vast number of customers that an SE may end up working with, they develop a deep understanding of customer use cases, product aspects that customers relate to the most, gaps to adoption and enhancements that can address those gaps. In this way, they act as great partners to product teams during:

• Product Development: Help provide early feedback on the initial versions of the product based on deep expertise on user journey

• GTM Adoption: Work cross functionally with marketing, sales and product to highlight the positioning and value of the product to pipelined customers

• Recurring, Adoption & Maintenance: Post-sales engineers (and sometimes pre-sales engineers) help in pitching incremental features for deeper adoption.

• Customer feedback: They are responsible for distilling this customer specific feedback for product teams in ways that helps them prioritize what needs to be built to meet incremental market needs

Sales Enablement: Sales Engineering teams also collaborate closely with sales enablement, revenue leadership and product management to enable internal sales teams on new product releases, technical playbooks / frameworks and are responsible for cross pollinating learnings from one deal to the other, to build a broader sales strategy and achieve more efficient execution across the board. Since sales engineers support multiple account executives, it may not always be possible for an SE to be on a customer call. Hence, they enable account teams on a set of questions that they should be asking early on, to drive informed product execution based on customer nuances.

Channel Partner Alignment: In specific product & GTM functions, where companies may rely on specific partners selling, hardware, integrations or other professional services bundled with the core platform to solve a customer problem, sales engineers can be responsible for identifying, filtering and bringing in the right partners on the table alongside account teams to solve for specific technical facets of the solution. This requires them to understand types of partnerships that the company holds, their value add and respective GTM motions.

How SEs exhibit product management super powers

To explain this, I took a lot of inspiration from The Skip podcast: Six Superpowers for Product Managers by Nikhyl Singhal, VP of Product at Meta who summarizes various superpowers that product managers can exhibit to create a high performing product teams. I merged it with my understanding and experience of working in sales engineering across each of these six focus areas:

Market expertise: The best product managers hold a strong expertise in various markets and segments of the value chain, understand the cyclicality of an industry, fragmentation, consumer behavior, buying power and various use cases. While product managers conduct user research for crafting a product on specific focus groups that involve existing customers and sometimes beta customers, sales engineers have the added advantage and access to existing customers and prospects / leads that are both qualified and disqualified per the sales team’s framework giving them a larger breadth of insights. Over time, they develop an understanding of the user personas and the buyers, various use cases that each of those personas have, the tools that they currently use and why these tools work or don’t work. They get to closely watch the part of the market that the product currently serves and the segments that it doesn’t serve. These insights can be translated to customer use-case dashboards that segregate customers by use case focus areas, industry, forecasted revenue and the core product features that define that customer’s propensity to buy or retain the purchase, thus creating a data driven pipeline on what customers really need. These insights, coupled with their intuition about the customer landscape helps product organizations make the right product investments.

Product Crafting: Product managers are experts are crafting a product with the right innovative features that connect with a customer and their needs. Some of these skillsets can arise from product design and marketing roles in consumer businesses and from sales engineering function in enterprise businesses. Sales Engineers spend 60 - 80% of their time talking to customers, understanding their problems through customer visits and looking over a user’s shoulder to understand the current user journey. They’re the first receivers of feedback when a product is showcased or used by a customer and thus hold an added advantage of directly interfacing with more customers than typical product managers. Given their engineering bent, they also possess a deep understanding of the technology that backs the product and can clearly connect the dots to create products that focus that optimize value and fit the right mix of technologies. They can seamlessly integrate their understanding of the market (the “who”), with “what” problems customers possess, prioritize ones that are the most painful or create the most value, and can chalk out “how” a product can be crafter from a user experience and technology perspective.

A great example of this would be a product team building a roadmap and design decisions on a business process automation platform, aimed at automating generic corporate workflows across several industries, focussed on mundane user tasks of copying data between systems. Identifying the high priority industries, understanding the types of systems that are used across specific verticals within the customer base, dissecting the problem into overlapping systems which are the most frequently used, putting them in the right order to build a short term and long term roadmap, finding the right techniques to build integrations with those systems, packaging them in ways that is the most easy to use for end users and finding the right repeatable framework that can be used by sales team to take the product to market paves a path towards successful execution. Sales engineers typically work in lock-step with product teams at each step of the above lifecycle, thus maximizing the company’s product and revenue success.

Growth Expertise: Product managers can be maniacally focussed on scaling product market fit for products that are just starting to take off. Typically, data science and growth marketing professionals are a great fit for such cases. However, growth is best achieved when one has a deep understanding of the market, the agility to gather feedback and iterate and the intuition to understand what could work. In enterprise businesses, growth ideally creates an increasing base of the sales funnel. Sales engineers hold the right level of intuition and an understanding of what customers (and buyers) really want versus what they say they want and the type of customers that may greatly benefit from as aspect of the product. Spending hours looking over the shoulder of various customers from a diverse set of industries and functions provides SEs the unique capability to match patterns and find the common pain points that when implemented as a product feature, can create organic product led growth and viral-ity. Their capability to ask the right questions, go multiple levels down and take a first principles led approach towards solving the problem can help achieve product market fit for a niche problem, while casting the net on a wide array of users. Being in a sales team also requires them to be tactical, nimble and agile in closing deals. The same attitude may work well for early stages where experimentation plays a major role in achieving product growth.

A great example of this came from my experience of working on an AI driven document understanding product that the company was selling to customers across utilities, financial services, healthcare and several other industries. The initial version of the tool was focussed on extracting information from documents, however, conducting 50+ proof of concepts across several customers required us to first segregate bulk documents into individual ones, before extracting information. We initially did so using scrappy product extensions and eventually realized that customers were actually spending more time sorting and scanning documents than extracting information. We concluded that this can be a product line in itself which then pushed us to build a document classification feature as a pre-requisite to document extraction. This ended up improving the overall efficacy of the product, while generating a bigger sales pipeline due to the high ROI nature of the feature. The nimbleness and agility in implementation helped us in finding the right solution over a period of time.

Domain expertise: A portion of product managers may specialize in specific domains such as Artificial Intelligence, Security or Hardware. This domain expertise can come from either an engineering function, or specifically a sales engineering function that’s laster focussed on specific product areas. Given how Sales Engineering teams are aligned to operational units / pods, depending on the nature of alignment to a market segment (enterprise, mid-market, SMB), an industry (such as financial services, healthcare etc.), a technology focus area (AI, mixed reality) or a specific product line (databases, business applications etc.), SEs can bring tremendous value with their laser focus on one of these specializations. Use cases and user behavior can greatly vary by market segments and industries which dictates nuances in user experiences and product workflows. Expertise in a specific technology domain can be incredibly helpful in understanding how workflows can be nuanced by customer type.

A great example of this from my experience involved going to market with a product focussed on immersive experiences in mixed reality for enterprise learning, wherein the core users comprised of knowledge and white collar workers working in a corporate or industrial setting. While most Mixed Reality headsets are designed, keeping gamers at the forefront focussed on highly interactive experiences that involve use of controllers to manipulate objects to transpose in 3D space, the same concept when applied to learning use cases created a rather uneasy user experience, primarily because the average learner at a target customer may not be a gamer and thus may not understand how to use a controller or its buttons. However such learners are accustomed to a simple, linear and guided learning experiences that are pre-recorded with minimal interactivity. Creating simple point-and-click, placard driven, yet spatially rich experiences created high levels of engagement, but this insight only came to life, as we gained domain knowledge about traditional e-learnings, what works with existing designs and what doesn’t work with lack of spatial aspects for specific use cases and lack of interactivity.

Organizational expertise (Internal & External Alignment): This typically involves expertise at aligning various teams across the organization on a set of problems that require involvement & buy-in from multiple product and engineering teams to create end customer value. In a world of more connected products and user workflows, a change on one part of the product often implies an upstream and downstream change to another feature. Teams with individual roadmaps can often have competing priorities due to their own interpretation of value that their product features offer to the customer. Sales engineering is an extremely cross functional role that sits between teams with somewhat overlapping but different incentive structures. Sales & revenue teams are incentivized to meet revenue goals and close customers quickly to create revenue growth while product & engineering teams are incentivized / motivated to roll out features that they hypothesize to add value to the customer. Such features may sometimes take time to implement. Sales engineers are responsible for chalking out an “implementation roadmap” of how a customer can adopt a software solution in the near term to maximize speed to execution, but have a vision to bundle on various features that incrementally add value in the long term to create retention. This may involve suggesting tactical workarounds / alternatives to implement the products in the short term to accomodate a customer’s adoption constraints around slow approvals, enterprise red tape, but also involves evangelizing why a long term solution is crucial for maximizing value, Parallely, SEs can also identify gaps around why the OOTB product setup and features may not solve the immediate problem, communicate those gaps to various product and engineering teams and suggest enhancements that could pave a path towards faster product adoption. This cross functional alignment between sales and product on the short term and long term customer-product adoption strategy paves a clear path for both teams to achieve their incentives while making customers successful.

Building Scalable Teams: Team expertise involves general leadership / management skills that are needed to manage a team, define direction and execute efficiently without micro managing members. While this expertise goes beyond just product management or sales engineering, SEs can have an edge in building team expertise in the following ways:

• Sales engineers index heavily on product documentation and building out repeatable and consistent solutions frameworks that help create efficient sales motions and distill customer feedback in a structured way. Having a crystal clear understanding of the voice of the customer via product feedback and documented execution frameworks for product implementation gives the broader team and the company, an understanding of how their product is taken to market, what works, doesn’t work and could be improved in ways that is consumable by a wide variety of product teams & members, which creates clarity of thought and organizational alignment.

• Sales engineers index heavily on enablement for account teams such that they can ask the right questions to the customer to obtain information that sets up execution teams for success. This skillset is broadly applicable to running and building product teams where enabling one’s own team or a sister team on how the problems are dissected, products are ideated and UX workflows are designed and PRDs are built, becomes crucial for creating a culture that minimizes asynchronous or synchronous communication and creates organizational efficiencies. This is a natural progression to the focus on documentation.

While being a great player coach goes beyond documentation and enablement, those pillars lay the foundation for creating a more inclusive, accountable and customer focussed culture which is a recipe for a successful product teams.

Bringing it home

• It’s a commonly known that sales is a team sport. Sales engineers are the quarterbacks of the teams that sit at the cross section of crafters and sellers, holding key traits of customer focus, curiosity & the tenacity to solve problems.

• Sales engineering is a critical function in building a great product. A great sales engineer can make a product defintion "better", and is critical launching a killer product and refining GTM as well as driving usage, adoption and expansion.

• Sales engineers have the added advantage of spending hours looking over the customer’s shoulder and obsessing on feedback to help improve the product. They hold the responsibility to communicate this feedback and create organizational alignment on what is working or not working, and how the product can be crafted to either serve existing markets in a better way, or serve an underserved market.

Subscribe now

Share

Leave a comment

A walk down the Visual Basic lane…

Business Process automation as a concept has been prevalent since the advent of modern day business applications. VB was first introduced in 1991 for developers. It was defined as beginners’ all-purpose symbolic instruction code and was a language that was used by users to interact with their computer both programmatically and visually. From inception to present date, we’ve seen several generations of tools and software platforms that have taken a stab at this problem, to achieve a vision of automating several manual processes in the enterprise, creating costs savings and increased business efficiency. The ethos of these products has centered around automating the mundane simple tasks involving data capture & entry workflows that are rules bases and don’t need complex human judgement or decision making. A commonly used term called "RPA” or “Robotic Process Automation” has now created a multi-billion dollar industry that aims to democratize workflow automation by automating such simple tasks. With the advent of AI, especially in the areas of computer vision and natural language processing (NLP), such tools have grown in terms of thir coverage of more complex use cases and data types. With the recent progress in large language models (LLMs), there’s a potential turbocharge awaiting such tools that involves semi-structured and unstructured data extraction, strengthened integration of legacy and new systems & complex human in the loop interfaces that unlock higher efficiencies. Let’s dive deeper into the landscape of tools, understand how a typical corporate workflow works, how these tools are designed from a product perspective, things they do well and don’t do well, and evaluate how those limitations and unexplored use cases can help generate enterprise value for customers and the companies serving them.

Understanding the current landscape

The existing landscape includes companies that have been in business for 35+ years to companies that have been incorporated in the past decade. Here’s a quick overview on the value chain:

• Robotic Process Automation as an industry has surfaced post 2010 and comprises of big players such as UiPath, Automation Anywhere, Blueprism & several smaller players that have recently seen stiff competition from incumbents such as Microsoft via their Power Apps platform.

• Integration Platforms such as TIBCO have existed for 2+ decades, but have been turbocharged with the advent of cloud computing, creating this new category of iPaaS tools which includes new entrants such as Workato, Mulesoft, Boomi and Zapier.

• Process Mining and Discovery as an industry has surfaced in the past decade, as a result of business processes being purely managed on modern day CRM & ERP systems, and with the advent of automation, companies such as Celonis, Kryon, Pega and Signavio provide deeper insights into a business process, monitor an employee’s workflow to highlight bottlenecks and manual steps, which then eventually yields input to what should be automated using RPA & iPaaS.

• Document Understanding has also been around the block for 3+ decades with traditional players such as ABBYY and Kofax but has been turbocharged with advancements in computer vision & NLP, giving birth to various generalized document understanding companies such as Hyperscience, Instabase, Google Document Understanding, Azure Forms Recognizer and even vertical-ized solutions such as Ocrulus, vic.ai and Klarity.

• Conversational platforms such as NICE have been around since 1986, primarily for assisting contact centers in managing high volumes of customer support requests. The advent of AI and RPA tools has given birth to general conversational AI platforms such as Dialogflow (Google), yellow.ai, Kore.ai that build multi-modal conversational solutions across a wide variety of customer engagement use cases. Companies such as Observe.ai, Uniphore, AiSERA continue to provide specific solutions for contact centers via more modern interface and advanced voice inference techniques.

Understanding traditional enterprise operations

Having understood the landscape, let’s leverage a few infographics to understand how a typical enterprise operates and the various systems that govern the day to day operations to better understand the problems that these tools solve.

The typical modern day systems architecture leverages multi-channel systems of engagement to interact with customers to sell products or support existing products using websites, email, documents, chat / text & calls. The front office refers to the section of a company that manages customer interactions. Employees in the front office typically leverage systems of record such as ERP (Enterprise Resource Planning), CRM (Customer Relationship Management) and HCM (Human Capital Management) to interact with customers, store information regarding existing products and employees. Systems of action include common productivity and work management tools that are then used to take actionable next step from customer interactions and execute on those steps. The back office perform tasks that are essential to a company's operations to serve the actionable steps coming from those customer interactions. This part of the business also leverages systems of record and systems of action, however, they may also leverage various third party systems of reference which includes public and private databases to validate customer and other information. During the course of this business operation, companies have increasingly leveraged Systems of Insights to harness actionable insights from all the data that flows through the system

Head work versus hand work: A typical employee’s role can be broadly categorized into head work and hand work, where the head work involves the customer interactions, simple to complex decision making and analyzing insights. The hand work involves the manual work item selection, capturing data items from source systems, inputting them to target systems and submitting the finished work items. Most of the automation tools thus far have been able to largely automate the hand work due to its rules based, mundane & repetitive nature that doesn’t involve complex judgement & decision making.

Corporate Micro-processes: Several enterprises with complex processes typically architect the core process into further sub-processes delegated to specific employees with very clearly stipulated roles & responsibilities. As a representative example below, the input work from customers is typically acknowledged by Employee A that captures data, passes it onto employee B who then enters the actionable data into target systems and employee C who may engage in validating the data with third party systems of reference. There may be multiple lines of supervisors who may review & validate the work, and potentially re-send the work item back for correction. Once reviewed and validated, the work item is typically executed based on the inferred output.

Data types

Employees deal with several types of data that increases in complexity, from structured to semi-structured to completely unstructured data. This could be as simple as structured “key-value” pair data found in modern day applications, and in many cases, tons of legacy systems which are still surprisingly prevalent at the enterprise. However, a large proportion of the data is still captured in semi & unstructured formats such as documents which may be as simple as a tax form, invoice, purchase order, or as complicated as a contract, or an email. The latter is where the majority of the business and transactional information between customers and enterprises can be stored and isn’t something that existing tools have been able to extract well, with current capabilities.

What makes a great use case?

The best use cases for workflow automation fall into one of the following categories

Micro-Processing:

• Large single process fragmented across multiple FTEs (full time employees). An example of this includes a financial trade settlement process within securities operations that takes roughly 3 days, involving multiple people & hand-offs.

• One or several small processes conducted repeatedly by a large volume of FTEs. A very apt example includes driver onboarding for ride share companies that involves reading through onboarding documents such as a W-4, driver’s license and inputting information into the company’s internal systems

Semi-Structured & Unstructured Data:

While structured data can be easy to extract from the source systems using existing API integrations to GUI based automation approaches, the following sources of data create high levels of manual effort, thus making a great use case. Following are the high value data types:

• Semi-structured documents with high variability such as invoices, order forms etc.

• Unstructured documents with free form text entities such as revenue contracts that consist of the total contract value, payment schedule etc.

Legacy and Traditional Systems:

The disparity in systems architecture is a result of technical debt that an organization continues to build, as they adapt more and more business systems. However, the nature of such systems can be characterized by:

• Legacy systems with little or no API capability such as mainframes and software built on a monolithic architecture.

• Systems with limited APIs such as legacy versions of SAP, PeopleSoft or ones that operate on legacy SOAP based API architecture (Workday). Such systems may also have legacy UIs including thick clients built on .NET, or web applications built on older stacks that may not be the easiest to scrape programatically.

Making the business case

The key drivers that typically make a strong business case often involve the inability for several modern & legacy systems to integrate with each other via modern API based approaches, managing high volumes that is otherwise manually moved, reduction in existing human error rate and a requirement to adhere to strict SLAs. The outcomes of such drivers include the hard and easily quantifiable dollars such as FTE related costs savings and the reduction in cost of hiring and coordination between FTEs. However, most companies also realize soft dollars from increased customer engagement and reduced financial losses caused due to human error post automation.

Massive Industry opportunities

While business process automation primarily applies to almost all industries that involve corporate data & customer workflows, there are a few that truly standout. Based on how we define the use case, the two industries that stand out the most are Business Process Outsourcing (BPO) & Financial Services. They key reasons to why is this the case comes down to the following:

• These industries have a ton of micro-processes. The BPO industry may have more of smaller mundane processes with a high volume, whereas the financial services industry has more complicated & long workflows.

• They deal with a lot of semi structured and unstructured data, especially documents, emails and text messages / voice based inputs.

• These 2 industries continue to maintain a mix of both legacy and new applications that are incredibly complex and hard to migrate from.

Product workflows for existing Business Process Automation platforms

Most existing products have similar workflows in terms of how they approach the process of identifying a use case, creating a low-code app that automates the process, deployment, interaction to monitoring & analysis.

1. Process Mining includes tracking of business systems event data such as Salesforce to create a process map that provides key metrics and insights into the steps undertaken, time spent at each step, potential bottlenecks etc. Process Discovery includes actual recording of the process steps on a click level to build a high level prototype of the automation based on the recording.

2. Low-Code / No Code Application Development involves the actual creation of a workflow automation, typical done via a simple drag and drop user interface that enables a user to import application integration packages, logical constructs or record UI based actions.

   

3. Deployment and Management comprises of tools that. help a user manage and monitor various automations that have been deployed to specific workstations.

4. Interaction and Monitoring empowers a user to interact with a co-pilot on their workstation that enables them to run various “bots” which they can interact with, to provide inputs or review outputs via tweak-able UIs that one can build within the platform.

5. Analyze & Visualize provides the capability for users to track / flag various data points that are being passed through over the course of the business automation, to dashboard specific analytics that provides both transactional insights for the process & business insights based on the data being actioned.

   

Evolution of the Business - IT Collaboration Model

The key forecasted value for solving via RPA from an overall ease of implementation perspective was that lots of applications do not have APIs and even if they do, most API integrations take a lot of time and specialized integration resources to build, which further adds cost of coordination between business & IT. To solve for this, most of the recent companies have spent tons of time and effort on the citizen development model, which dictates that a business user can build their own automation using the low-code / no-code workbench, thus reducing the amount of dependency on IT. However, most successful implementations have often needed IT involvement in a greater capacity, to eventually take the basic skeletons that business users build and harden them to run reliably over long periods of time. The existing citizen development model still requires heavy dependency on IT and external partners, who market the thousands of “RPA developers” that they have within the company, to be staffed on such projects.

What do existing tools do really well?

Looking at the first versions of visual basic, existing tools have brought the concept of automation to greater life in the modern digital enterprise in the following ways:

• Intuitive drag and drop interfaces have truly simplified workflow automation for the mundane and repetitive tasks to constructs that don’t necessarily need complex programming paradigms. Building a bot is a more visual process and resonates well with business users, who tend to be more process oriented.

• UI driven automation has come a long way and has been hardened over the years to detect complex UI components across a wide range of modern and legacy desktop and web applications in a relatively reliable DOM based mechanism.

• Granular and diverse integration packages with modern day applications such as Salesforce, SAP, Workday provide users with out of the box capabilities to create simple workflow automations. Tools have been able to seamlessly mix API integration based packages with UI workflow tools to create a consistent developer experience.

• Modern tools have been able to implement efficient orchestration on the cloud that can be scaled up or down to multiple instances based on the input workloads, and can also be deployed on-prem for customers with sensitive data thus managing and reducing the total cost of ownership and providing scalability.

• Interactive application builders have created the ability for users to interact with a guided agent that sits on the user’s desktop and provides a human in the loop experience for complex workflows that involve human review steps.

What are the key gaps and how can AI help?

Process Discovery

Process Discovery: Despite such tools running in the background, process discovery is relatively slow, inefficient and rules based in terms of tracking a user’s actions, which thus generally creates a bad user experience and sometimes, inaccurate process maps. The initial automation prototype generated post process discovery is typically very bare bones and relatively non-functional, thus eventually requiring developer input to build complex logics to build a functional automation. Existing AI stacks offer multi-modal inputs that provide a probabilistic approach to ingest image, video & text based inputs while watching a user’s screen to understand / infer the overall scope of a workflow which thus creates the potential to build more than just a barebones workflow application. Having been trained on the whole internet also gives it broader context on how the same process may be done in similar settings, which further helps contextualize the process in scope

Low-Code / No-Code Workbench & Systems Integration

Despite the low-code nature of the workflow, overall development still takes time and can become increasingly arduous as the workflows start requiring complex logical constructs. Most workflows rely on static logics that have to be pruned / hardened over a period of time based on runtime results & added exceptions as the workflow processes more transactions. Such refactoring eventually ends up requiring incremental development effort to embed more complicated logics / edge cases to maintain the workflow. Analogous solutions to what we have today for Github co-pilot could very well extend to such work-benches.

Furthermore, UI based automation workflows tend to break frequently, due to the constantly changing UI of the source systems, and the existing rules based DOM mechanism for detecting specific UI components has limited capacity to ingest such changes. Newly introduced models specialized in image detection w/ multi-shot prompting & modern reinforcement learning approaches provide a compelling path for users to seamlessly communicate such edge cases and UI changes to automations for refactoring to become a more seamless process.

Semi-Structured & Unstructured Data

Documents, be it digital or scanned represent a major proportion of an enterprise workflow. The variability in “keys” and their respective positioning on a page in semi-structured documents has thus far been a considerable barrier for accurate data extraction, especially for use cases such as accounts payables and receivables which involve financial transactions that warrant accuracy and thus, existing tools end up requiring significant human review. The capability for an LLM to understand numerous textual and spatial variations of a search term via embeddings will help turbocharge the extraction and improve accuracy of such workflows.

However, the maximum impact for LLMs, would apply to free form unstructured text in documents such as legal contracts, wherein, extracting revenue amounts, terms and schedule for payment may imply significant cost savings. Existing tools have tried to embed NLP models available from their marketplaces, but these models have often had to be fine-tuned or trained on specific data which is time consuming.

Adaptive Workflows based on Business Insights

Existing tools deliver relatively deterministic insights, with maybe some forecasting functionalities, via dashboards, which are limited to numeric data. There is however, a ton of scope for textual and image based insights, beyond numbers that could paint a much more insightful picture into the automated workflow. Taking an example of an accounts payable specialist processing transactions on a week by week basis, the capability to flag a payment transaction for a potentially higher than stipulated value for a specific vendor, based on contextual understanding of the underlying contract terms of payment and anomalies based on historical payment data, is an example of truly adaptive automation, driven by text based and historical insights. Existing tools do provide model plug-ins to do so, but LLMs provide the capability of deeply integrating such functionalities into existing model workflows.

Workflow Marketplace

While most companies currently operate marketplaces to share automation between various developers to create a developer ecosystem and streamline adoption of the platform, except the truly API based integrations that are more robust and can truly be used in a plug & play format, any automation that’s build using the native workbench’s logical constructs and UI driven screen recordings cannot be used out-of-the-box and requires significant re-factoring to make it work for the specific user scenario. This is mostly because the OOTB automation lacks context of the user process and their underlying system nuances. For example, a UI automation built on Salesforce lightening may almost certainly not work on Salesforce classic. Existing AI techniques can potentially empower users to easily refactor existing automations hosted on marketplaces to create true interoperability. Moreover, the concept of prompting creates a whole new paradigm for marketplaces, wherein sharing prompts might be a helpful way for developers, than sharing low code.

Human in the loop

While existing tools have provided seamless methods to create human-in-the-loop style automations, most of the existing human in the loop interfaces to validate, fine tune and submit the output of existing deterministic workflow automations do not have any reinforcement learning tied to them. The modern reinforcement learning frameworks can be potentially implemented to improve this back and forth workflow and reduce the total time that an operator would spend on reviewing workflow outputs from a software bot.

Multi-Modal Interactions

While interacting with visual forms and trigger based approaches is the easiest way to initiate workflow automations, chat and / or voice based interaction interfaces are still lacking in existing tools. This is where zero to few-shot prompting become very compelling use cases for collecting user input and triggering automated workflows

What are some key considerations for enterprise adoption?

While consumer adoption has grown tremendously over the past few months, we’re finally starting to see enterprises adopting the new stack for specific workflows. However, concept to mass scale adoption will require the following:

• Private / Hybrid Deployments: For heavily regulated industries such as banking, it is quintessential for customer data to stay on-premise and be processed locally, while the overall orchestration, development and management can occur on the public cloud. This may require such solutions to deploy and run models locally and deploy model updates on a frequent basis. As always, companies need to ensure that they follow enterprise grade encryption protocols and go through regular SOC 2 & GDPR audits.

• Data sharing & business models: There will still be customers who may be open to sharing data (especially types that are not sensitive), such that they can leverage features that deliver higher value from updated and fine tuned models, arising from ingesting the shared data. Some creative business models may involve give price credits for customer to share data to further incentivize them to share data. This is yet to be seen

Bull case versus bear case

Turbocharging existing & new companies

The bull case for generative AI turbocharging existing software platforms may essentially come from one of many of the components mentioned above wherein specific use cases that are now solvable with the modern AI architecture unlock massive enterprise value via product changes that integrate existing functionalities with such new AI approaches, or rebuilding parts of the product natively using an AI first approach to create ease of use. The latter will be more prevalent in new companies than old due to the classic innovator’s dilemma. This however, does require re-investing the stack in some specific areas of the platform, while cleverly integrating in the other areas.

Disrupting existing companies

The bear case is primarily governed by the possibility that over the next few years, we may see a flurry of multi-modal interfaces that embed, chat, voice & GUI based approaches to build more AI native methods of interaction, wherein both existing and new vendors for the underlying systems in scope of a workflow may embed co-pilots in existing application layers that can seamlessly talk to other systems via API and visual scraping capabilities. The API revolution was a great historical step towards creating a more connected enterprise systems ecosystem and natively embedded co-pilots may establish a new paradigm, thus making existing automation tools less valuable. That still doesn’t entirely eliminate the need of such tools since enterprises move & adopt slowly, continue to have tons of unstructured data and continue to gather technical debt with multiple systems that may not talk to each other.

References and Motivation

• The new moats - Greylock

• Give to get model for AI startups

The above is a summary of my thoughts from spending 5 years in the beautiful trenches of working in the business process automation space both on simple & mundane, as well as complicated workflow automations, which gave me a deep understanding of how middle and back offices of such organizations function, their systems & process architecture, how they implement existing workflow solutions and manage them.

Find me on X

Find me on LinkedIn

Email Me

Mixed reality is a blend of physical and digital worlds, unlocking natural and intuitive 3D human, computer, and environmental interactions. It’s often signified by two key concepts: Virtual Reality & Augmented Reality.

In this article, I dive deep into the craft of Mixed Reality, which is a sub-section of spatial computing and its application to the enterprise via key use cases that broadly apply across various industries. I talk about the different layers of the value chain from infrastructure & hardware to software & tools, and delve deep into the fundamentals of the content ecosystem, highlighting what it takes to implement spatial computing in the enterprise.

Differentiating pillars of spatial computing

Spatial computing integrates digital information that has been brought about and virtual objects into the real-world physical environment, creating a mixed or blended reality experience:

• Perception via Spatial Awareness: While enterprises have been successfully able to augment digital information via existing compute methods, spatial computing brings tremendous value in jobs that require the use of spatial awareness & cognition in a 3D environment. This is especially important for blue collar jobs where an employee uses a wider field of view to interacts with real objects & move in a three dimensional space to accomplish a job. This is why immersive & guided learning are proven to create high levels of information retention as compared to 2D learning.

• Hearing via Spatial Audio: This is one of the more underrated or unobserved aspects of mixed reality. Spatial audio allows for accurate sound placement and localization within the virtual environment. It enables users to perceive sounds coming from specific directions, just as they would in the real world. This helps create a more immersive and convincing mixed reality experience.

• Interacting via Hands Free, Multi-modal Interactions: Spatial computing enables users to interact with digital content and systems in a hands-free manner using gestures, voice commands, or gaze-based input. This hands-free capability is particularly valuable in enterprise environments where users need to access information or perform tasks while keeping their hands free for physical work (Eg. Engine assembly). Traditional computing devices require manual input methods like keyboard, mouse, or touch, which can be limiting and less efficient in certain enterprise scenarios for transmitting information from your brain to the machine.

• Contextual Information Augmentation: Allows users to overlay contextual information onto their real-world view, enhancing situational awareness and decision-making. Users can access relevant data, instructions, or annotations directly in their field of view, without the need to switch between different applications or screens. Traditional computing devices typically require users to refer to separate displays or switch between applications, which can disrupt workflow and reduce efficiency.

• Real Time Collaboration & Remote Assistance: Users can share a virtual space and interact with digital content simultaneously, regardless of their physical location. This capability is beneficial for remote teams, field technicians, or experts providing guidance from a distance. Traditional desktops, PCs, and tablets, although capable of video conferencing and remote collaboration, lack the level of immersive presence and shared spatial context offered by mixed reality methods.

Drawing parallels to previous waves of computing

Looking back in history, the first iterations of computers were adopted by the government, research institutions and universities in the 1950-60s. The mainframe era from the 1960s - 70s brought ground breaking support for centralized computing for key functions such as accounting, billing & transaction processing. The minicomputer era of the 70s-80s brought computing resources for local data processing & analysis to engineering, sales & inventory management functions. The PC revolution from the 80s - 90s brought by IBM and Apple brought mass adoption via affordable and user friendly graphical user interfaces (GUI). The key value driver for the PCs was centered around improved productivity via advanced compute capabilities and GUI based applications; enhanced data management via the advent of memory management & eventually relational databases; increased collaboration via email & instant messaging and cost savings via the medium of sharing information digitally than physically.

Mixed reality (MR) is rapidly gaining traction in enterprise environments, following a trajectory similar to previous advancements in computing technologies. Initially embraced by pioneering organizations and research institutions, MR is now poised for broader adoption in the enterprise via the use cases mentioned below.

Enterprise Use Cases for Mixed Reality

Virtual reality is a broader concept, which has prevailed in the enterprise for several years in the form of “flight simulators”, however the wave of head mounted displays (HMDs) has played a pivotal role in accelerating adoption of MR in the enterprise. The release of Oculus Rift DK1 in 2013 created a major inflection point in creating awareness regarding mixed reality and its application to the enterprise. Here are some of the high ROI use cases:

1. Immersive & Guided Learning Learning: MR is used to create realistic training and simulation scenarios where employees practice complex procedures, operate machinery, or perform tasks in a safe and controlled virtual setting without the need for physical equipment or risking any real-world consequences. This create enormous value creation for general operational effectiveness, health & safety, soft skills and customer service use cases across several industries such as manufacturing, construction, healthcare, retail & logistics

2. General Remote Collaboration: MR enables geographically dispersed teams to collaborate in a virtual space. Participants can interact with 3D models, share data, and communicate as if they were in the same location. This enhances remote collaboration, accelerates decision-making processes, and improves productivity, especially for industries with blue collar workers, where a three dimensional space / object needs collaboration between multiple personas.

3. Design and Prototyping: MR provides a powerful tool for designing and prototyping products. Designers can create virtual prototypes and examine them in a 3D space, enabling them to assess scale and functionality. This speeds up the design iteration process and reduces costs associated with physical prototypes. This is especially helpful for design & development within the manufacturing, construction and transportation industries.

4. Sales and Marketing: Mixed reality can be used to showcase products or services in an immersive and interactive manner via virtual showrooms or experiences that allow customers to visualize products, customize features, and make informed purchasing decisions.

   

The Enterprise Mixed Reality Universe

The enterprise mixed reality universe can be largely split into two key buckets:

Infrastructure Layer

The infrastructure universe is the plumbing or picks & shovels layer which constitutes of players all the way from the chipset, OS, hardware & wearables layer to the development engine & enterprise mobility management layer upon which the solutions are built and managed, and finally to the systems integration services & network providers layer which delivers the connected hardware & software experiences to end customers.

Chipset Layer

The compute layer as of today is concentrated by Qualcomm, via its XR2 chip, which most headset manufacturers build upon. New entrants include the AMD LiquidVR and the recently launched Apple R1 chip which is currently exclusive to the VisionPro.

Operating System

It’s interesting to see that most headsets that have their own operating systems that are eventually forked off of Android Open Source Project (AOSP). The version ranges from Android 8 to 12, but mostly wrapped in a device manufacturer specific UX layer such as LuminOS by Magic Leap, Pico OS & even the Meta Quest to an extent. The non-Android based OS includes the Windows Holographic Operating System dedicated to the Hololens and the VisionOS, dedicated to the Apple Vision Pro

Enterprise Headset Manufacturers

While most of the headset sales in and outside of the United States are bent towards the Quest 2 & the Quest Pro bought by gamers, fitness fanatics and early adopters focussing on apps such as Roblox & Beat Sabre, a small but growing proportion of headsets being crafted for the enterprise.

• Device manufacturers such as HTC and Lenovo are investing capital in building enterprise grade MR headsets such as Focus 3, XR Elite & ThinkReality VRX.

• Bespoke manufacturers such as Varjo engage in building expensive high-end headsets for use cases with low latency & high compute requirements.

• Consumer focussed companies such Meta & Pico have a considerable share of their business invested in selling enterprise ready headsets. The Pico Neo 3, Pico 4E, Pico G3 have a footprint at various Fortune 500 enterprises.

• Finally, there are category creators for dedicated augmented reality headsets including Microsoft’s Hololens, Magic Leap 2, and very recently the fantastic Apple Vision Pro.

Multi-Modal Wearables

Specific companies focus on elevating multi-modal interactions with mixed reality applications via wearables such as haptic gloves and dedicated hand tracking equipment. Ultraleap offers an external device for hand tracking, while HaptX provides haptic gloves that can track hand interactions, as well as provide haptic feedback based on the nature of the interaction. These are especially helpful for training scenarios which involve hand interactions and decision making based on the corresponding haptic feedback. For example, a gearbox assembly process involves assembly of minute components and heavy gears, which cannot be realistically simulated using controllers or native hand tracking.

Game Development Engines

While Unity and Unreal are the two most common game development engines used across enterprise and consumer use cases, Unity for Enterprise currently has the largest market share of the application via a high number of developers that build enterprise content & applications on Unity. Almost all the headset manufacturers support Unity and even the VisionPro, despite the existing ARKit capability, is looking to provide an integration to applications built in Unity.

Enterprise Mobility Management

The core infrastructure software providers primarily include companies that provide enterprise mobility management solutions (EMM) or Unified Endpoint Management Solutions (UEM). In context of VR headsets, this industry is typically comprised of:

• Incumbent EMM / UEM providers such as VMWare WorkspaceONE UEM, Microsoft Azure Intune, SOTI Mobicontrol & Ivanti Mobileiron who are bundling functionalities to support Mixed Reality device fleets in a similar framework to how they support laptops, mobile phones, tablets & rugged devices.

• Bespoke MDM providers such as ManageXR & ArborXR that focus on management of only enterprise XR device devices, thus providing feature sets that are unique to XR devices.

Network Infrastructure Providers

While this may sound trivial and broad, network infrastructure providers have a crucial role to play in making the runtime of MR applications, as low latency, high bandwidth and thus, as seamless as possible. MR devices can essentially be considered as IoT style devices with high levels of sensory inputs & outputs thus requiring a strong internet connectivity, especially at remote business locations. Incumbents like Verizon & AT&T have rolled out 5G powered IoT networks & devices (such as MiFis) to deploy mixed reality applications at scale.

Systems Integrators & Service Providers: The key partners bringing all of these different solutions to life and integrating them with a customer’s existing business systems to paint an end to end enterprise mixed reality are the big systems integration and management consulting companies. The most prominent examples include Accenture, Deloitte, and several more big & small service providers that touch several facets of enterprise adoption for mixed reality. The last but not the least are the core systems integrators such as Insight, SHI, CDW etc. that provide the “last mile” integration services with respect to hardware procurement, provisioning and deployment for devices to their existing and new enterprise customers.

User Experience Layer

The user experience layer sits on top of the infrastructure stack and brings most of the direct value to enterprises in the form of applications, platforms and content that solve for specific problems or use cases. Here is a representative infographic of this space all the way from game studios & vertical content creators to software platforms that bring end to end value from a core orchestration, management, content & integrations perspective:

Content Creators

Content creation is the most crucial and the most evolving, yet always insufficient piece of the mixed reality industry. Content is king for creating engagement and adoption. Delivering content to the enterprise is typically done via a services based model and the players can be split into the following categories:

• General Content Shops that create content across a wide variety of enterprise use cases. A few examples include StartBeyond, Sweetrush and 3lbxr.

• Vertical Content Shops that typically focus on learning & development for various enterprise use cases such as ELB Learning, Roundtable Learning.

• Big SIs & Consulting companies such as HCL, Accenture, Deloitte etc. that engage in large scale content creation services for their existing customer base. These practices however, are quite small and nuanced at this point but helpful for such companies to increase their account footprint via such bespoke services.

Specialized Software Platforms & Tools

While the total MAU for VR headsets is still nascent enough for true network effects to kick in, especially for the enterprises, we’re still seeing some early enterprise platforms that are often focussed on specific use cases:

• Enterprise Immersive Learning Platforms engage in enabling the creation, deployment and management of immersive learning at scale. They key players are STRIVR, Transfr, PixoVR, OssoVR and SkillsVR.

• AR based productivity or work management tools, targeted at specific use cases such as Taqtile, ScopeAR, Gamma-AR (for construction) and Immersed (for productivity) that leverage existing augmented reality features within headsets to achieve guided learning use cases for construction & manufacturing industries.

• MR Soft Skills platform specifically focus on employee soft skills development across various use cases such as customer interactions, de-escalation, sales & support training. Key players include Ovation, VirtualSpeech, BodySwaps, Talespin that leverage the power of an immersive corporate settings and high fidelity characters to simulate scenarios that may otherwise be hard to simulate in a real life environment.

• Prototyping & Design platforms such as ShapesXR, Masterpiece Studio, Gravity Sketch are natively designed to support 3D design and prototyping in a collaborative setting which is especially helpful for physical design & development use cases in hardware, automotive, energy & construction industries.

• Healthcare & Digital Therapeutics applications that are catered towards specific surgical procedures, as well as general health & wellness use cases. Companies such as OssoVR & Oxford VR engage in various healthcare training scenarios while companies such as Tripp & Reulay engage in building meditative experiences for both enterprises & consumers.

• Virtual Collaboration tools such as Horizon Workrooms, MeetinVR, Glue VR either create their own suite of virtual collaboration tools, or integrate with existing tools such as Zoom. Such tools bring virtual meetings in 3D via digital personas (avatars), environments & object plug-ins such as whiteboards & presentations. Although, Virtual collaboration in 3D is largely an early stage and untested territory from a value creation perspective and may have a long way to go to create true differentiation.

The above list is representative and is the tip of the iceberg of the 100s of companies trying to build out platforms with true network effects and ecosystems that can span across multiple use cases, personas and eventually multiple enterprise industries.

Understanding the Content Ecosystem

Degrees of Freedom

Degrees of Freedom (DoF) refers to the number of independent ways in which a user can move and interact within the virtual environment. It describes the range and complexity of user motion that can be tracked and translated into corresponding actions or interactions in the MR experience. Most mixed reality experiences are typically categorized into 3 DoF & 6 DoF experiences:

• 3 DoF experiences facilitate rotation around the X-axis (pitch), Y-axis (yaw) & Z-axis (roll), thus enabling the user to tilt their head or change its angle up or down (similar to nodding), turning one’s head left or right (similar to looking from side to side) & tilting the head as if the user is shaking it.

• 6 DoF experiences facilitate translation along the X, Y & Z axis, that enables the user to move horizontally from side to side, vertically, either ascending or descending and move forward or backward within the virtual environment.

• Interaction & Object Manipulation refers to the ability to interact with and manipulate virtual objects within the mixed reality environment. It involves performing actions such as grabbing, moving, rotating, resizing, and otherwise manipulating virtual objects as if they were physical objects in the real world. Users can interact with these objects using various input devices, such as handheld controllers, gesture recognition, or even hand tracking technology.

Types of Content

Here is a quick summary of various types of content that exists as of today:

• Fully Immersive - Live Action / 360 / 180 Video: This type of immersive content involves pre-recording real experiences using 360 cameras or several high quality 2D cameras, where the footage is eventually stitched together and rendered as a 360 video. For enterprise use cases, this content type is especially helpful for facility walkthroughs, operational effectiveness & safety use cases which require immersion in a wider field of view. Here’s a video of a 360 factory tour of Tesla’s factory is Fremont, which can be viewed on a virtual reality headset using YoutubeVR. 360 experiences are typically 3 DoF in nature, however, there have been examples of 6 DoF 360 videos.

• Fully Immersive Computer Generated (CG) Experiences: Computer-generated experiences refer to immersive and interactive environments that are created using computer-generated graphics, audio, and other sensory inputs. Computer-generated experiences in virtual and mixed reality are created through a combination of 3D modeling, rendering, animation, audio design, and interactive programming. They are designed to provide users with immersive, interactive, and often realistic experiences. Below is an example of a recorded in-headset CG immersive training targeting a specific healthcare use case. Such experiences can be both rendered in 3 or 6 degrees of freedom.

• Augmented Experiences: Augmented experiences involves rendering computer generated 3D objects either within an actively rendered real environment, which is typically done in a passthrough mode via MR device cameras, or super-imposing such objects in a pre-recorded 360 experience.

Understanding the MR Infrastructure Plumbing

Adoption of mixed reality infrastructure in the enterprise today seems to follow a similar pattern to how the computer adopted in its first few iterations of the technology. Here are the key characteristics of how MR is adopted within the enterprise and the various considerations that need to be kept in mind, to make it acceptable & conducive to a typical enterprise:

1. Network Infrastructure: As mentioned above, high bandwidth, low latency and reliable network infrastructure is crucial for mixed reality applications. While corporate office settings may have cracked the code for strong internet connectivity, blue collar locations such as warehouses, distribution centers & remote offices often struggle with poor connectivity due to the presence of several obstacles & lack of vicinity to network hubs.

2. Running Compute on the Edge: Due to constraints in connectivity at remote business locations, MR applications can work around this constraint by creating locally cached / offline experiences with asynchronous data downloads & uploads to ensure that the products are user ready. However, mixed reality application that require real time information augmentation with very low latency face lots of challenges. Being able to run as much compute on the edge / device, be it high fidelity experiences / object rendering, or information retrieval is crucial for the success of MR applications, especially for the blue collar enterprises.

3. Shared Check-in / Check-out model: Given that most enterprise MR implementations are limited to one or more of the use cases listed above, most enterprises maintain a few headsets per business locations for employees to check in and check out sequentially for respective use. This currently follows a similar model to how rugged devices are used and thus requires sanitation, operational & storage considerations. For applications such as training, enterprises may employee dedicated areas where such devices can be used.

4. Limited use per employee: Given the check-in & check-out model, the maximum use for a headset per employee is often limited to 30 mins to an hour at best per experience since that’s how long experiences / remote collaboration sessions are typically designed for. Some mixed reality applications that may be mission critical for manufacturing operations may be worn for longer periods of time.

5. Plug & Play via Enterprise Mobility Management: Similar to how organizations use rugged devices including tablets, mobile phones which are kiosked to run a specific application such as product catalogue search tools or customer check-in software, MR devices are typically deployed in a plug-and-play fashion wherein, they’re kiosked to the application(s) in scope, auto-connect to the corporate network and facilitate check in & check out functionalities via employee log-ins. Any user restrictions, security policies, software & firmware updates are securely managed via the Mobile Device Management system.

6. Data Privacy & Security: Most HMDs granularly track spatial data such as head & hand / controller positioning and expose this data to applications via their SDKs. Applications use this spatial data to create interactive experiences and some may choose to gather this data for creating immersive analytics that further add value to their product offerings. Given that most enterprises have to abide by specific employee data protection laws such as CCPA & GDPR, creating a strong governance & privacy framework that provides enterprises, the capability to know and control how and where is data is fetched & sent. Understanding how applications store (if at all), any PII, retain it, encrypt it, delete it and allow opt out to users, is a deal breaker for adoption in heavily regulated industries

7. User Management: The shared check-in, check-out model and the need to tie a user’s data to their usage session requires the need for a user to identify themselves via their enterprise identity, typically stored in an Identity Access Management (IAM) system such as Okta, PingFed, ADFC etc. Thus, virtual reality applications adopted in the enterprise are typically required to integrate via a Single Sign On workflow to authenticate employees. This is often tricky when a user is prompted to enter a long email address & password in a 2D browser via a QWERTY keyboard, using a point & click controller. Some applications however, have found creative ways to smoothen this experience by offering QR code based authentication.

Summary

Having explained this detailed context, I’d like to summarize and bring your attention the following takeaways:

• The ability to achieve spatial awareness via vision & audio, multi-modal interactions via hands or haptic devices and the capability to augment contextualized information are the pillars for value creation within the enterprise

• Enterprise Spatial Computing ≠ the social metaverse that largely focusses on social interactions and presence. There are some core components of the metaverse that apply to enterprise spatial computing but the current enterprise spatial computing universe targets to solve specific problems, that otherwise can be solved in a limited way via existing 2D rendering or manual / physical methods.

• As it stands today, the content creation ecosystem is sparse and highly fragmented to say the least. This is the biggest challenge that's endemic to the MR industry as a whole, both in the entertainment and industrial spaces. The industry is having trouble staying afloat due to a dearth of content. And it's partly because not many players have figured out how to build it in a profitable & scalable manner.

• Network effects and interest graphs are possibly two key drivers for helping the industry cross the chasm for specific use cases verticalized by industry. I will. talk about this in depth in my future articles.

• It takes some amount of baseline plumbing / infrastructure, systems, network & privacy policies to implement mixed reality for the enterprise, however the general concepts are largely similar to implementing any other hardware & software solution.

Leave a comment

Share

Subscribe now

Earlier this year, we saw the release of ChatGPT, a prompt based search / text generation tool, that reached 100 million users in two months. It’s the fastest that any platform has scaled to this number of users. While ChatGPT has showed incredible success so far, it’s an indication of market maturity for AI generating tremendous value at various levels, all the way from the chip / wafer level to the application layer. We’ve reached an inflection point where the innovation on a chip & foundational model layer has turbo-charged the capability for the application layer to iterate quickly to create value to end consumers. One such use case that stands out is digital presentations. In this article, I’ll dissect this specific application layer use case, to build a balanced hypothesis on these products and how they boost productivity across the workflow, how they position against traditional tools, evaluate total addressable market and provide my perspective on what could differentiate them from one another.

A bit of history…

Microsoft Powerpoint was created back in 1987 by Robert Gaskins and Dennis Austin for the American computer software company Forethought, Inc. The program, initially named Presenter, was released for the Apple Macintosh in 1987. Before PowerPoint, there were slides. Real ones. They were tiny, and tactile, and delicate, and kind of delightful. Making slides was a trained profession for highly skilled designers and technicians

Generative AI & Relevance to Visual Storytelling

AI generated presentation tools engage in visual & textual storytelling by automating the core “plumbing“ & the baseline creative judgement needed to put together a narrative. A user can generate a full blown narrative by:

• Inputting an initial prompt on the problem statement or theme.

• Feeding the tool, a detailed document which enlists the narrative is a semi-structure format

• Leveraging a “co-pilot” while building the narrative in an existing tool, using visual and textual content recommendations, auto-complete & peer-review style functionalities

What’s under the hood?

AI generated presentation tools operate on an application layer level as a consumable abstraction of:

• Large Language Models (LLMs) like ChatGPT that have the capability to crawl the internet based on a prompt, extract the right level of information and articulate it in a semi-structured way.

• Generative Image models like DALL-E that intake a natural language prompt to create images that visually contextualize those prompts.

Such tools culminate AI generated visual and text outputs in a more structured and compelling way rather than a blob of text or standalone images in a prompt driven chat style product design. They thematically structure the text based responses & generated images into a coherent storyboard, thus creating what we call a “presentation”.

Cost of building presentations

Making a simple presentation with a few slides can take anywhere between a few hours to days depending on the complexity of the narrative. Here’s the cost of building a presentation:

• Defining the narrative: As a consultant, I spent 40% of my time building narratives on new proposals, services offerings & existing project updates. A typical storyboarding exercise involves construction of the problem statement, understanding the end audience & applying frameworks to explain the solution.

• Research & Data Augmentation: Harnessing facts, metrics, or abstracting raw data to create a compelling story via a data visual such as a chart, trend-line or data tables.

• Text & Visuals: Adding succinct bullet points & compelling visuals that can articulate the message in a minimally verbose fashion. The best decks / slides are the ones that lean more on light-weight visuals & minimal words.

• Beautification: While this may seem like a counterproductive exercise to some, corporate teams focus heavily on beautification & alignment. Tools such as Power point & Google Slides provide great features to distribute assets, align text / images, yet, this is largely a time consuming process

• Brand Compliance: Brand compliance is a mandatory step for established enterprise companies that aim to achieve consistency across all their external and internal narratives. It involves pre-defined color pallets, chart styles, iconography, product abstractions etc

Prominent Players

In the recent past, the number of companies offering AI driven presentation building tools have sky rocketed. Having said that, the market can be split into two major categories:

Existing tools bundling an “AI co-pilot”

This comprises of companies that have offered tradition presentation building tools and are now bundling what most people are calling a “co-pilot”. A couple of representative companies include:

• Microsoft Co-Pilot for PowerPoint, which embeds the Microsoft 365 co-pilot built on the OpenAI services, across all Office 365 tools

• Prezi, which has been an established presentation tool for quite some time, but is now bundling AI capabilities to enhance user experience

• Canva which has been an incredibly beneficial tools for graphic designers. Canva has recently rolled out a “Magic Write” functionality to write text responses & “Text to Image” to generate images.

AI-native presentation tools

Most of these tools have been released in roughly the past 3 - 5 years and are built natively on the GPT & DALL-E / Stable diffusion frameworks, thus providing slightly different user workflows.

• The more prominent companies include Tome.app & Beautiful.ai that have a substantial user-base built over the last few years. Tome.app claims to have grown to 1 million users in 2022.

• The up & coming companies include Designs.ai, Presentations.ai, Kroma.ai, DeckRobot, Slidebean, Pitch, Vengage

This list is representative and the number of new companies is ever-increasing, given advancements such as GPT-4 and newly released image generation models.

What typically makes a good presentation tool?

Regardless of whether a tool is or isn’t AI driven, following are some key features that create a coherent user experience for a typical presentation tool.

• Pre-Built Templates: Pre-built templates have traditionally provided a great head start to building coherent presentations by providing the user with a skeletal storyboard, that they can then build upon.

• User Experience: A sizable potion of the world has moved on to web native tools with very responsive UIs and simplistic designs to facilitate ease of use. Tools such as Prezi, Tome.app & Beautiful.ai are a testament to this. On the other hands, tools such as Powerpoint are the 800 pound gorillas with a ton of feature-sets but are relatively hard to access and use.

• Integration with Modern Design Applications: Specific tools such as Tome.app & Beautiful.ai provide a deep integrations with design & white boarding tools such as Figma & Miro which are used by modern project teams to visually and textually articulate ideas.

• Integrations with Analytics Tools & Databases: Every presentation tool should be able to connect to a database to generate visual representations such as charts or succinctly display tabular data on a slide.

How is AI being injected into different parts of the user workflow?

• Automated Visuals & Styling: AI co-pilots speed up the process for researching visuals that are relevant to the scope of information highlighted in the slide, which would otherwise require manually crawling Google Images. This is being injected as prompts that a user can input while building such narratives.

• Automated Speaker Notes: Based on the context of the slides, such tools can automatically generate speaker notes to add further relevant information about the topic at hand, while keeping the slide succinct.

• Fine-Tuned Storyboarding: Since the underlying LLMs may be pre-trained on thousands of similar storyboards on the same topic, such tools can often provide “SME” level comments on the coherence of the storyboards, and help with enriching the content, which essentially solves for what one calls “Peer review”

• Automated Styling & Alignment: Finally, the last few bits of plumbing being automated is the iconography, graphic styling, alignment, which otherwise is a manual and counter-productive process.

Addressable Market

While the the total addressable market for presentations is roughly estimated at 5+ billion USD in an overall workplace productivity tool market of 50+ billion, it’s hard to segregate a specific productivity tool such as a Google Slides or Powerpoint from a broader set of productivity suites such as Google Workplace & Office 365. It is however important to understand the various functions or personas where such tools add tremendous value:

B2B / Enterprise Companies

Management Consulting: Consulting firms spend 20 - 30% of the total time in a project on building narratives in the form of decks. Given that the typically margins for a services engagement can be 25 - 50%, any productivity boosts in streamlining the 20 - 30% effort spent in building presentations helps increase bottomline and improve outcome quality. The predictive & repetitive nature of such storyboards is what makes AI generated presentation tools, a compelling fit for this industry.

Sales Teams: Sales teams spent roughly 50% - 80% of their time in either outreach (typically done by BDRs) and conducting customer demos. Out of this chunk of time, roughly 50% is being spent on crafting presentations. The capability for AI generated presentation tools to create highly contextualized narratives that speak to a customer’s problem through competitive analysis & industry specific research can turbo-charge an AE’s productivity. This indirectly reduces the cost of acquisition for a customer, thus increasing the overall value achieved from the deal.

Product & Engineering Teams: Product managers typically build the vision for a product, translate it into clear product features, prioritize them based on needs of customers & business value, and finally work with engineering teams to execute and implement those features This is articulated in what they call PRDs (Product Requirement Documents) which are essentially narratives aimed at highlighting the core problem, proposed solution and benefits. Some also involve summarizing A/B testing results, generally available user research studies & cost benefit analyses. Building compelling decks from verbose PRDs speeds up the product planning process, thus increasing product velocity.

Early Stage Companies: Early stage companies index heavily on speed to execution, given limited access to capital, ambiguity of the market which they’re trying to capture and the need to iterate quickly on customer feedback. Most efforts around building presentations are spent by extremely nimble sales & management teams (typically founders and early execs) on acquiring new customers and investors to spearhead the company forward. Any tool that boosts productivity by automating the “plumbing” is beneficial for such companies to achieve a faster speed to market.

Business to Consumer

While I’m not an expert at business to consumer markets, the key personas for tools includes (but not limited to) students, job seekers, independent freelance artists and graphics designers, each of whom are looking to showcase their work or credentials to potential customers or companies.

What could differentiate these tools?

Almost every “AI driven” presentation tool may end up using the same set of Model as a Service APIs for information retrieval and search, including GPT, Bard, Cerebras and other companies building foundational models. However, there are moats that each of these businesses can develop:

• A two sided economy for training datasets: Data is the goldmine for each of these models and the expansive “Reinforcement Learning from Human Feedback (RLHF)” is what eventually fine tunes accuracy and depth of information. Building a 2-sided incentive model for customers and partners to contribute to the core models by providing the ability to add new data, update existing data or reporting inaccuracies is what could turbocharge the accuracy and stickiness of such products. In return, such customers and partners can earn credits, that discount the total price they pay for using such services via credits. David Sachs from Craft Ventures talks about this in detail in his post on the Give-To-Get Model for AI.

• Privately Hosted Models: In extremely regulated industries such as banking, healthcare & pharmaceutical, a lot of narratives or storytelling involves consumption of proprietary data such as drug testing results or medical procedures that are still un-approved or are being tested. In such scenarios, corporations may have a ton of existing content or content frameworks which are often rinsed and repeated to create compelling narratives. The possibility of being able to train a privately hosted model for specific corporations or fine-tuning a privately hosted version of the existing generic model with company specific data can turbo-charge adoption of such tools.

• Plug-Ins to Existing Presentation Applications: While natively built AI tools have taken a stab at re-designing the workflow of building presentations from ground up, it is important to acknowledge that a vast variety of users in the world still leverage the existing players in the market such as Powerpoint, Google Slides & Canva. These 800 pound gorillas with their massive distribution advantage can essentially bundle functionalities such as “Co-Pilot” for free, thus putting the distribution channel for the new AI-native presentation tools at jeopardy. While this is an incredibly hard problem to solve, building a “Co-Pilot” style plug-in for such tools which integrates into existing productivity suites may put such tools on an equal footing from a distribution perspective. It still however, it tough to sell a custom co-pilot for a non-zero dollar amount while the core product offers a free version.

• Citations: A large variety of presentations, especially the ones that delve into deep market research about an industry, a product or a service require a user to cite sources for the data. While the GPT framework as of the date of writing this article doesn’t expose the citations, it will become increasingly important to quote data sources to avoid usage of any copy-righted or prohibited content in such presentations. This is particularly important for traditional or regulated enterprise businesses that have strict guidelines around what data they can or cannot use.

• Brand Compliance: As mentioned before, brand compliance is crucial particularly for big enterprise companies to create consistency, visually & sometimes even from a textual messaging perspective. Brand compliance involves using specific icon styles, deck themes,

• Security: Provided that such products focus largely on harnessing public data-sets, adoption of such tools in enterprise businesses requires intense information security reviews. It’s hard to predict the security framework that will be set forth by regulators (if at all), but the companies that achieve mass adoption will have to come up with a transparent framework on what datasets have been used, and providing the customers secure architecture models to host the tools as well as the underlying models.

Who wins the race?

The human brain is not wired to think in exponential terms. In an era where research is being re-written every year (even months), It’s incredibly hard to predict which player wins in the market. Incumbents such as Microsoft & Google could continue to bundle such co-pilots and maintain their position, or some nimble player could come up and disrupt the market. While there may not be a winner-takes-all in productivity tools that provides an application layer on top of the foundational models, the companies that innovate on a chip-set / wafer level such as NVIDIA, Cerebras as well as the ones who build the foundational models, such as OpenAI might end up building the biggest moats. At the end of the day, there is still enormous value creation for such tools, even though the broader market may be highly commoditized in the near term.

References & Notes:

• Thanks to ChatGPT and DALL-E for helping me with content review and a title image :)

• Generative AI: A Creative New World - Sequoia Capital

• Generative AI Is Exploding. These Are The Most Important Trends To Know

• The Give-to-Get Model for AI Startups

Share

Share Siddhant’s Substack

Leave a comment