Clips AI

Video files, especially those containing spoken content, offer rich opportunities for content repurposing. Whether creating engaging social media posts from longer interviews or extracting key quotes for presentations, the process of converting raw video into manageable clips can be both time-consuming and technical. This article presents Clips AI, a sophisticated tool that automates this workflow using advanced AI techniques for audio analysis and video processing. Through AI-powered transcription, smart clip identification, and dynamic video resizing, Clips AI enables developers to efficiently extract and reuse valuable content from their video libraries.

Clips AI Technology Overview

Clips AI enables developers to automatically convert longform video into clips, focusing on audio-centric content analysis. The technology segments videos based on transcript analysis and dynamically resizes them to maintain speaker focus.

Transcription and Clip Identification

The transcription process utilizes WhisperX, an open-source wrapper on Whisper that detects start and stop times for each word. This function produces detailed transcript information, including character, word, and sentence-level timestamps. Clips are identified using the TextTiling algorithm, which segments text based on topic shifts and was enhanced by integrating BERT Embeddings as described in the 2021 paper "Texttiling with BERT Embeddings."

Media Processing

The ClipFinder algorithm processes transcriptions to create Clip objects, which contain properties for start and end times in the transcription. These objects can be manipulated using the Clip class methods for copying and dictionary conversion. To create trimmed media files, developers use the MediaEditor class, selecting desired clips and calling the trim method on an AudioFile or AudioVideoFile object.

Video Resizing

Video resizing employs a three-step process using Pyannote for speaker diarization, PySceneDetect for scene change detection, and MTCNN/MediaPipe for face detection. The tool requires a HuggingFace authentication token and offers several parameters for customization, including target aspect ratio, minimum segment duration, and face detection settings. The resized video maintains dynamic speaker focus through the crops object, which contains segment information for the resized file.

Clips AI Components

ClipFinder Algorithm

The ClipFinder algorithm builds upon Marti A. Hearst's 1990s work in text summarization, employing the TextTiling algorithm for content segmentation. This approach detects topic shifts through word usage and distribution patterns in the transcript. Recent enhancements incorporate BERT Embeddings, as detailed in the 2021 paper "Texttiling with BERT Embeddings," which has refined the original formulation's capabilities.

Clip Class

The Clip class encapsulates video or audio clip information, featuring properties such as start and end times within the transcription. It supports operations through methods like copy() and toDict(), facilitating manipulation and integration into broader workflows.

MediaEditor Class

The MediaEditor class enables video trimming based on selected clips. Developers instantiate an AudioFile or AudioVideoFile object, specify the desired clip(s), and employ the trim() method to generate the edited media file. This process requires a HuggingFace authentication token for speaker diarization services.

Resize Functionality

Video resizing requires several parameters including the target aspect ratio (default 9:16), minimum speaker segment duration (1.5 seconds), and face detection configuration settings. The implementation leverages Pyannote for speaker diarization, PySceneDetect for scene change detection, and MTCNN/MediaPipe for face detection. The output includes crops object information containing segment details for the resized video.

Audio Processing with Clips AI

The audio processing pipeline in Clips AI utilizes WhisperX for transcription purposes. WhisperX stands as an open-source wrapper built upon the Whisper framework, specifically enhanced to identify and mark start and stop times for each spoken word within an audio or video recording. This transcription process generates comprehensive Transcription objects containing detailed information about the audio content, including timestamps for characters, words, and sentences.

Once the transcription is complete, the TextTiling algorithm processes the text to identify sections of the recording that should be segmented into clips. This algorithm builds upon foundational work from the 1990s by Marti A. Hearst, originally designed for text summarization. The latest version of the algorithm has been augmented with BERT Embeddings, as described in the 2021 paper "Texttiling with BERT Embeddings." This enhancement enables the algorithm to more effectively detect shifts in topic within the transcript, resulting in more natural and coherent clip boundaries.

The resulting clips are represented using the Clip class framework, which captures essential metadata such as the start and end times of each clip within the original transcription. This structured approach facilitates seamless integration of the clips into various content repurposing workflows, allowing developers to easily manipulate and utilize the segmented audio content.

Video Resizing Features

The video resizing functionality in Clips AI employs a sophisticated workflow combining Pyannote for speaker diarization, PySceneDetect for scene change detection, and face detection powered by MTCNN and MediaPipe.

Speaker Diarization with Pyannote

Pyannote handles speaker identification and segmentation through the provided authentication token from HuggingFace. The tool analyzes audio regions, determines speaker changes, and produces speaker-tagged annotations that form the foundation for dynamic resizing.

Scene Change Detection with PySceneDetect

PySceneDetect processes the video stream to detect significant changes between scenes. This step helps maintain contextual relevance by ensuring that transitions between different speaking segments are handled appropriately during the resizing process.

Face Detection with MTCNN and MediaPipe

The face detection component uses MTCNN for initial face localization and MediaPipe for subsequent refinement. These tools work together to detect and track faces within the video frame, enabling the system to maintain focus on speakers while resizing the content.

The tool requires several parameters for customization:

• Aspect ratio (default 9:16)

• Minimum segment duration (default 1.5 seconds)

• Number of samples for face detection (default 13)

• Video downscaled width for face detection (default 960 pixels)

• Margin around detected faces (default 20 pixels)

• Application of post-processing (default False)

• Number of batches for GPU processing (default 8)

• Minimum scene duration for detection (default 0.25 seconds)

• Scene change merge threshold (default 0.25 seconds)

The output from the resizing process includes detailed segment information packaged within the crops object, which tracks the resizing adjustments applied to the video content. This information is essential for maintaining accurate metadata about the edited video's structure and content.

Integration and Usage

To utilize Clips AI, developers begin by transcribing their audio or video content using the WhisperX wrapper, which detects start and stop times for each word. The transcribed data is structured into Transcription objects containing detailed word-level timestamps, enabling precise content analysis. This transcription forms the foundation for clip identification and video resizing operations.

Once the content is transcribed, clips are automatically generated using the ClipFinder algorithm. Developers can then use the MediaEditor class to select and combine desired clips into new audio or video files. The API provides comprehensive control over the clipping and resizing process, allowing for precise segmentation and dynamic resizing to maintain speaker focus.

For video processing, Clips AI requires a Pyannote authentication token for speaker diarization services. The resizing functionality operates on a three-step workflow: speaker diarization, scene change detection, and face detection. The process parameters include target aspect ratio, minimum segment duration, and face detection settings, offering flexibility for different content types and production requirements. The tool outputs detailed crops information, including segment details for each resized clip, ensuring accurate metadata retention while maintaining the original content's narrative flow.