Twelve Labs

Understanding video content has become increasingly critical as businesses grapple with the explosion of video data. From security footage to corporate presentations, the ability to extract meaningful information from video is transforming how organizations operate. Twelve Labs has emerged at the forefront of this technological revolution, developing sophisticated AI tools that turn raw video into actionable insights. Through groundbreaking models like Marengo and Pegasus, the company has created a suite of capabilities that process and understand video content in ways previously thought impossible. This article explores how Twelve Labs is redefining video understanding through advanced AI technology, and how its enterprise-grade solutions are helping organizations make sense of their video data.

Company Overview

Twelve Labs specializes in video understanding AI, having developed a suite of technologies that analyze and search through video content. The company's most significant achievement to date is its $30 million funding round, which drew investments from major players including Databricks, SK Telecom, and Snowflake Ventures. This funding has propelled Twelve Labs forward, enabling the development of their latest foundation model, Marengo 2.7, which employs a novel multi-vector approach to video understanding.

The company's technological foundation includes the Marengo video encoder model, which examines visual frames and their temporal relationships while also processing speech and sound data. This video-native encoder serves as the basis for Twelve Labs' perceptual reasoning pipeline. Additionally, the Pegasus model bridges text and video data through cross-modal reasoning, combining the reasoning capabilities of large language models with the perceptual understanding of the video encoder.

Twelve Labs offers a comprehensive API suite for video understanding, providing developers with tools for search, vector transformation, and fine-tuning. Their technology processes terabytes to petabytes of video data, extracting features such as actions, objects, text, conversation, people, and speech. The company's platform achieved state-of-the-art performance in the 2021 Microsoft-hosted ICCV VALUE Challenge, outperforming competitors in video retrieval tasks.

Core Technologies and Innovation

Twelve Labs has developed a sophisticated foundation model called Marengo that represents a significant advancement in video understanding technology. This video-native encoder model analyzes visual frames and their temporal relationships, while also processing speech and sound data. The model's design emphasizes context awareness and serves as the foundation for the company's perceptual reasoning pipeline.

The company's most innovative technology is the Pegasus model, which represents a breakthrough in cross-modal reasoning. By merging the reasoning capabilities of large language models with the perceptual understanding of the video encoder model, Pegasus demonstrates how text and video data can be aligned to infer meaning and intent from multimodal representations. This combination allows for more sophisticated and contextually rich interpretations of video content.

According to internal research, Twelve Labs focuses on creating a robust world representation through video processing that closely mirrors sensory data. Their work bridges the gap between video understanding and high-level reasoning tasks using natural language, demonstrating significant advancements over traditional rule-based or tag-based approaches to video analysis. The company's technology processes terabytes to petabytes of video data, extracting features such as actions, objects, text, conversation, people, and speech while maintaining state-of-the-art performance in industry-standard benchmarks.

Product Portfolio

Twelve Labs offers a suite of APIs that enable developers to integrate sophisticated video understanding capabilities into their applications. The company's platform requires minimal training and allows for rapid deployment, with developers able to achieve rich contextual understanding through just a few API calls.

Key features of Twelve Labs' technology include deep semantic search capabilities that allow users to find exact moments within videos using natural language queries instead of traditional tags or metadata. The platform also supports dynamic video-to-text generation, enabling users to create concise summaries or custom reports by providing prompts that detail the desired content format. This text generation functionality has proven particularly effective in applications such as police reports, demonstrating the technology's ability to handle specialized content requirements.

The company's approach to integration is designed to simplify development processes, allowing users to focus on application development rather than managing multiple data sources or separate image and speech APIs. Twelve Labs' technology processes video data through a cloud-native distributed infrastructure that scales to handle thousands of concurrent requests, with results typically available within seconds of the query.

The platform supports multimodal search capabilities, enabling users to query videos based on visual elements, conversations, logos, and text. This multimodal approach represents a significant advancement over traditional rule-based or tag-based search systems, which have struggled to effectively process complex video content.

Enterprise Applications and Partnerships

These partnerships extend Twelve Labs' reach into enterprise infrastructure, particularly through their collaboration with Databricks and Snowflake. The integration with Snowflake's Cortex AI enables advanced AI features for consumer experience improvement, content analysis, and video creative search capabilities. This partnership allows Twelve Labs' multimodal video embeddings to be stored in Snowflake's vector data support, creating a powerful foundation for AI-driven applications with built-in security and governance.

Through these integrations, Twelve Labs has developed several advanced capabilities. Their collaboration with Databricks has created an efficient development framework that reduces the complexity of building advanced video applications, enabling complex queries across large video libraries and improving workflow efficiency. The combined solution provides state-of-the-art video understanding, featuring unprecedented multimodal capabilities that capture video content essence in a unified representation. This approach simplifies deployment architecture and enables more nuanced, context-aware applications in content recommendation systems, advanced video search engines, and automated content moderation tools. The solution offers real-time video analytics capabilities and supports large-scale content classification systems, while also enabling novel generative AI applications.

Data and Security

The company has implemented a comprehensive security framework to protect client data, earning SOC2 and ISO 27001 compliance certifications. These standards ensure that handling exabytes of data occurs within a secure, cloud-native infrastructure designed to scale efficiently.

Data retention policies allow free-tier users to maintain their index for 90 days, while paid customers receive unlimited access with a 600-minute soft limit. For extended storage needs, users can opt for dedicated deployments, though infrastructure costs apply. The company periodically revises its API limits - currently set at 600-minute soft limits for indexing - to accommodate growing customer demands while maintaining service performance.

Twelve Labs employs a flexible deployment model offering cloud-based services, self-hosted cloud options, and on-premises installations to suit different organizational needs. This adaptability ensures that customers can integrate the technology into existing workflows without extensive modifications to their infrastructure setup.