Not just another pilot

AI training projects often focus on hypothetical scenarios but the Response Preferences project flipped the script by putting real-world situations front and center. 

We gathered 30 to 40 experts, within four domains, to challenge the models with cases that only someone immersed in the field could dream up. These weren’t abstract tests; they were lifted from the messy reality of classrooms, hospital rooms, and financial offices.

Another layer to this challenge was the use of multi-turn dialogues. Each task required building a unique multi-step conversation rather than relying on a single question-answer pair. This forced the AI to reveal its reasoning rather than jump straight to an answer, giving the experts a chance to highlight strengths and weaknesses in real time.

And because the project ran on two different platforms, Mindrift and our client’s external tool, experts had to adapt quickly. It added a layer of complexity but also gave them a better sense of how AI fits into different professional workflows.

Why it matters beyond the project

AI isn’t coming, it’s here. Teachers are testing it for lesson planning. Doctors are exploring whether it can support clinical decision-making. Financial professionals are weighing its role in risk management. By stress-testing models with real cases, experts were able to see where AI helps and where it risks harm.

In medicine or finance, even small reasoning slips could lead to harmful outcomes, so surfacing those risks early is critical. In education and linguistics, AI needs to handle tone, nuance, and ambiguity, or even switch gracefully between languages. 

“[This project was] intellectually stimulating, challenging, meaningful. It felt very similar to conducting an advanced experiment: you define constraints, test outputs, refine methods, and rely on peer feedback,” said Yigit, a Linguistics expert on the project.

And what became clear from this experiment is that AI can’t be left to evolve in isolation. It needs professionals, the people who know what’s at stake, to push, test, and shape it.

The hard work behind the scenes

Of course, building this dataset wasn’t simple. Each expert had to stretch both their creativity and their critical eye. Some common challenges our experts faced were:

• Balancing creativity and reality: Every dialogue had to be different, diverse, and realistic. This required a balanced combination of imagination and deep professional knowledge.

• Guiding the conversation: Experts had to guide the model into showing its reasoning process, not just giving final answers, which is oftentimes easier said than done. 

• Finding (and describing) the weakness: Beyond picking the “better” answer, they had to clearly articulate flaws in both outputs — from logic gaps to unsafe advice to lack of nuance.

• Managing two platforms: Juggling the Mindrift platform and the client’s platform added friction, requiring extra care and adaptability.

But with every challenge comes a learning opportunity. As Tatiana, Medicine expert, said:

“It has been fascinating to learn how AI thinks when processing medical data and generating answers, and to see how critical it is for experts to guide it throughout development, adaptation, implementation, and quality control.”

Lessons learned, inspiration sparked

While the dataset created from this project is valuable, the learning opportunities that came from it are what truly set the project apart. Experts walked away with a sharper sense of where AI can truly assist and where it still falls short. 

“I see AI having a big impact on my field in the next few years. It will help streamline information management, improve data analysis, and assist in decision-making. It won’t replace human expertise … I hope! But it will definitely enhance productivity and create new opportunities to work more efficiently and creatively,” said Ana, Education expert. 

Some even found themselves inspired, taking new ideas back to their classrooms, clinics, or research labs after watching how the model tried (and sometimes failed) to solve problems.

“AI will increasingly support clinicians by improving diagnostic accuracy, speeding up decision-making, and personalizing patient care. I see it as a powerful tool that can enhance, but not replace, the human aspect of medicine,” pointed out Milos, Medicine expert.  

Most importantly, the process built confidence. Breaking down AI outputs, instead of just picking the “best” option, helped experts strengthen their own digital literacy. 

“This project has made me more aware of both AI’s power and limitations. I now see AI tools less as replacements and more as collaborators—systems that can be highly effective when guided by human reasoning and expertise,” explained Evangeline, Linguistics expert. 

And collaborating with peers, project leads, and QAs reminded them that their expertise isn’t just useful, it’s central to shaping the future of AI.

“It was great that we could discuss some of the feedback on Discord, because text can always be interpreted in different ways, even when written according to specific criteria … the beauty of the human mind,” said Anastasia, Education expert. 

Looking ahead

Ultimately, the Response Preferences project showed that real-world expertise, not just more data, is what training AI responsibly is all about. Success comes from combining professional experience with thoughtful design. 

As AI continues to evolve, projects like this serve as a reminder that technology will only ever be as strong as the people guiding it. And when those people come from classrooms, clinics, offices, and labs, the future of AI looks not just smarter, but more human.