Multimodal Video Search: From Notebook to Production

The problem

A client had tens of thousands of hours of video content. Finding a specific scene meant scrubbing through footage manually or relying on sparse manual tags. Their team spent hours each week just looking for clips.

They wanted natural-language search: type "person presenting at whiteboard" and get timestamped results.

Architecture

We built a two-stage pipeline:

1. Indexing — extract frames at 1fps, encode each with CLIP ViT-L/14, store embeddings in a vector database alongside metadata (video ID, timestamp, thumbnail URL).
2. Retrieval — encode the user's text query with the same CLIP model, run cosine similarity search against the index, return top-k results with timestamps.

# Simplified retrieval
query_embedding = clip.encode_text(query)
results = vector_db.search(query_embedding, top_k=20)

We spent a while on frame sampling rate. 1fps hit a good balance: enough coverage to catch most scene changes without blowing up storage. A 1-hour video produces about 3,600 frames, roughly 2MB of embeddings.

What we learned

Batch indexing matters more than we expected. Encoding frames one-at-a-time on a GPU wastes most of the available parallelism. Batching 64 frames per forward pass cut indexing time by 10x.

Metadata turned out to be half the product. A timestamp alone is useful, but a timestamp with a thumbnail, video title, and a 10-second clip preview is what people actually clicked on.

Vector DB choice was less important than we thought — at least at this scale. Under 10M vectors, most options perform fine. Past that, you start caring about indexing strategy (IVF vs. HNSW) and memory.

Results

Queries dropped from minutes of manual scrubbing to under a second. New video gets indexed within minutes of upload. The content team stopped asking us about search after the first week, which we took as a good sign.