Building a High-Quality Video Annotation Pipeline at Scale

The Challenge

Interhuman AI is building a social intelligence system designed to understand how people communicate beyond words alone. The platform detects subtle behavioral cues such as hesitation or confidence from video, audio, and text, and translates them into interpretive information for downstream AI systems.

To train this platform, Interhuman needed human-labeled data across a 12-signal taxonomy. These signals represent distinct but related social behaviors that unfold during real conversations. Unlike simple categorical labels, each signal must be precisely aligned to moments in the video, requiring sustained attention and careful judgment from annotators.

This level of complexity introduced a core challenge: consistency. The team needed annotations that were reliable not just within a single batch, but across multiple waves of collection, while maintaining agreement between annotators and minimizing noise introduced by participants unfamiliar with such demanding tasks.

"The specific task we designed was not very easy. It's a sophisticated and uncommon type of annotation, and we spent a lot of time figuring out the best possible design."

The Solution

Interhuman used CloudResearch Connect as the backbone for recruiting, qualifying, and managing its crowd-based annotation workforce, all while running the labeling itself inside a custom-built annotation interface tailored to video-based signal detection.

Work was structured across multiple phases: pilot, calibration, and production. Early waves were used to validate task design and signal definitions, while later waves focused on scaling participant volume. Intentional overlap between annotators on selected videos allowed the team to monitor agreement and recalibrate definitions as needed.

CloudResearch Connect enabled Interhuman to screen participants based on real engagement with the task. Signals such as time-on-task and interaction with the annotation timeline made it possible to identify low-quality submissions without relying on guesswork.

"We could clearly see when someone wasn't taking the task seriously—for example, when time on task was shorter than the video itself, or when they didn't interact with the timeline at all."

Iteration Without Disruption

Because the annotation task was demanding, rejection rates were higher than those seen in typical labeling projects. Rather than treating this as a concern, Interhuman used rejection as an intentional quality control mechanism.

CloudResearch Connect made it possible to reject low-quality work, relaunch tasks, and quickly replenish the annotator pool without manual coordination. This flexibility allowed the team to continue collecting high-quality data even as they refined task instructions and simplified interaction patterns for future runs.

This approach ensured that rigor was maintained without slowing the overall pace of data collection — a critical requirement for a fast-moving product team.

"Even with a high rejection rate, we always had a way to compensate. That flexibility mattered a lot, especially while we were learning how to simplify the task for future runs."

Results

By centralizing recruiting, screening, and payments within CloudResearch Connect, Interhuman significantly reduced the operational burden typically associated with managing large annotation efforts.

The resulting dataset provided consistent, time-aligned labels across the full 12-signal taxonomy. These annotations are now used both to train Interhuman's social intelligence models and to benchmark how accurately the system interprets complex human behavior.

The structure of the program also enabled faster iteration, allowing Interhuman to incorporate findings from one wave directly into the next without retooling the entire workflow.

"Overall, my experience has been very, very positive. I've worked in academia and with many online experiment platforms, and I've actually been recommending Connect to colleagues."

What's Next

CloudResearch Connect will continue to serve as a core component of Interhuman's long-term data strategy as the company expands its annotation program. Future plans include additional large-scale annotation waves to support new behavioral signals, as well as smaller, rapid-turnaround projects driven by specific client needs. These smaller waves allow the team to respond quickly when new signals are prioritized, without disrupting the broader training pipeline. The fundamental approach will remain consistent: crowd-sourced annotations enhanced with expert input.

"For future waves, we'll be using the same high-level setup — crowd annotations combined with expert input. Connect will continue to be our solution for crowd-based annotation."