STAI CDT PhD student Shahin Honarvar presented the paper, ‘Turbulence: Systematically and Automatically Testing Instruction-Tuned Large Language Models for Code‘ at the 18th IEEE International Conference on Software Testing, Verification and Validation (ICST 2025).
We spoke to Shahin about the paper and his experience at ICST 2025:
Congratulations on presenting the paper! What is your paper about?
Our paper focuses on a simple but important question: Can we really trust AI tools that write code? These tools, like ChatGPT, are increasingly used to help people write programs. But while they often get things right, they can still make surprising mistakes.
To investigate this, we built Turbulence—a system that tests how reliable these AI tools really are. Instead of asking just one question, Turbulence presents hundreds of small variations of the same question to see whether the AI stays consistent. The idea is to find out whether the AI gets all the variations right—or only some. The surprising discovery: even the most advanced models can stumble when they encounter small variations.
We tested several of today’s most popular AI models. The results showed that while some models performed better than others, none were fully dependable, and many struggled to be consistent. Our findings mark a significant step toward understanding where AI still needs to improve before it can be fully trusted in real-world software development.
This work was done in close collaboration with my supervisor, Alastair Donaldson (Imperial College London), with additional contributions from Mark van der Wilk (University of Oxford) — many thanks to both for their valuable contributions.
That sounds like a hugely exciting piece of research. Aside from presenting the paper, how did you find being at the conference?
ICST 2025 was held in Naples, Italy which was a great location. It brought together leading minds from around the world to focus on making software safer and more reliable—especially in the age of AI. Our research fit in perfectly and it was exciting to see that it attracted strong interest from fellow researchers.
Beyond the sessions, the conference offered a truly inspiring environment—lively exchanges, beautiful surroundings, and the chance to connect with people from across the globe.
Conferences are valuable experiences for early career researchers. How have you been finding your PhD journey so far?
So far, I’ve really enjoyed my PhD journey—it’s been both challenging and rewarding. Being part of STAI CDT has been a huge support. The programme has not only given me access to brilliant researchers and training, but also a strong sense of community. It’s a great environment for exploring ambitious ideas while staying grounded in the real-world impact of AI.