26.11.2026 IfI Colloquium: Can We Trust the Scoreboard? Toward Reliable AI4SE Evaluations
Speaker:
Prof. Dr. Maliheh Izadi
Delft University of Technology, The Netherlands
Date: Thursday, 26 November 2026, 17:15
Location: room BIM 4-033 at the Department of Informatics (IfI), Binzmühlestrasse 11, 8050 Zürich
Details about the format of the talk shall be checked always just ahead of a certain presentation date: (information here)
Abstract
Large language models and autonomous agents are increasingly used to write, test, and repair software, and benchmarks are the primary instrument for measuring their progress. Yet as AI for Software Engineering (AI4SE) has grown explosively, rigorous evaluation has not kept pace: benchmark knowledge is fragmented across tasks, selecting contextually relevant benchmarks is difficult, benchmark creation lacks standardization, and many widely used benchmarks suffer from flaws, such as data contamination and unrealistic tasks, that undermine the conclusions drawn from them. This talk maps out these challenges and presents our efforts to address them: systematically organizing and assessing the benchmark landscape, developing guidelines and tooling for rigorous benchmark construction, and building new benchmarks that evaluate LLMs and agents on realistic software engineering tasks.
Bio
Dr. Maliheh (Mali) Izadi is currently a senior research scientist at Google LLC, Switzerland. She is also an assistant professor in the Faculty of Electrical Engineering, Mathematics, and Computer Science at Delft University of Technology, the Netherlands. Her research focuses on solving challenges of building and tailoring LLMs and autonomous agents to source code and software engineering processes, such as rigorous evals, model memorization, IDE integration, and in-IDE Human-AI interaction. At TU Delft, she leads theAISE (AI-enabled Software Engineering) research lab. Her work has been supported by various funding and awards, including a Google Research Scholar Award (2025) and an Amazon Research Award (2024), as well as industry collaborations such as JetBrains Research (AI4SE) and Meta (FUSE lab).