Enabling High-Quality Image Annotation and Data Annotation for Vision & Language Research

The Challenge

The Barnard Visual Cognition Lab studies high-level visual perception and the relationship between vision and language. Many projects in this field require open-ended written descriptions of images at scale, demanding both nuance and consistency in data annotation.

For years, the lab relied on Amazon Mechanical Turk for data labeling. Increasing bot activity and declining response quality made the platform unreliable as a data labeling tool, while technical overhead created friction for undergraduate researchers.

As the lab searched for a Mechanical Turk alternative, the team evaluated several large-scale data labeling platforms, including Scale AI. While these enterprise platforms were powerful, they were designed primarily for large-production machine learning pipelines. For the lab's academic, iterative research workflow, the rigidity of enterprise tooling didn't meet the need.

Without a flexible, research-scale data annotation solution, projects were delayed or paused entirely.

The Solution

After evaluating both enterprise labeling platforms and academic-focused tools, the Barnard Visual Cognition Lab selected CloudResearch Connect's Data Labeling Template for its balance of flexibility, data quality, and ease of use at the research scale.

Simple Setup Without Custom Code

Using the Data Labeling Template, the team could upload CSVs and launch image annotation and data labeling tasks immediately—without writing custom HTML or JavaScript. This allowed both faculty and undergraduate researchers to iterate quickly and launch studies independently.

Rapid Iteration Through Built-In Piloting

Because open-ended annotation tasks are highly sensitive to wording, the lab relies on piloting to refine instructions. Built-in piloting tools made it easy to preview tasks, observe participant interpretations, and adjust prompts before full launch—improving response quality and reducing downstream cleanup.

Reliable Human Participants at Scale

Compared to prior platforms, Connect consistently delivered real, engaged participants who took the task seriously. The lab not only observed fewer low-effort responses and minimal bot interference, but also clear evidence of genuine interest in the work itself.

Participants frequently completed multiple annotation tasks thoughtfully, and some even reached out proactively to express that they enjoyed the study or ask whether additional work was available. This level of engagement gave the research team added confidence that participants were doing the data annotation work they intended—carefully, attentively, and with a clear understanding of the task goals.

"The biggest thing for us was that we weren't getting bots in every other response."

Results

With CloudResearch Connect, the Barnard Visual Cognition Lab was able to execute large-scale image annotation and data annotation projects that had previously been impractical or impossible.

In one flagship study, the lab collected 10 open-ended written descriptions for each of approximately 6,000 images, generating a rich dataset designed to support nuanced analysis at the intersection of vision and language and to serve as machine learning training data. Collecting data at this scale using traditional in-lab methods would have required extensive recruitment, scheduling, and oversight—stretching across an entire semester or longer.

Using Connect, the team completed data collection in roughly one day, dramatically accelerating the research timeline while maintaining confidence in response quality. This speed allowed the lab to move quickly from data collection to analysis and manuscript preparation.

Beyond scale and speed, participant engagement stood out as a meaningful outcome. Rather than treating the task as rote work, participants demonstrated sustained attention and interest, often completing multiple annotations and providing detailed open-ended responses. In some cases, participants even contacted the lab directly to share positive feedback about the task—an uncommon signal that the work was both understandable and motivating.

The quality and consistency of the data directly supported peer-reviewed academic output, with the lab's first paper using Connect-generated machine learning training data now in press. More importantly, the platform restored the lab's ability to pursue research questions that depend on large-scale, open-ended human input—work that had previously been paused due to the lack of a viable data collection solution.

Equally important, the workflow proved repeatable. The lab now routinely reuses the same Data Labeling Template configuration across studies, allowing new datasets to be launched with minimal setup while preserving consistency across experimental conditions. This transformed data labeling from a one-off logistical challenge into a sustainable part of the lab's ongoing research pipeline.

"It's not really about time saved. It's more about whether we can do this research at all – we wouldn't be able to do it without CloudResearch."

What's Next

The lab plans to continue using the Data Labeling Template for future vision-and-language studies and expand into additional annotation formats. With a reliable, repeatable workflow in place, the team can move much faster from idea generation to execution.