Towards Reliable Machine Learning Models for Code - SICSA Guest Seminar by Foutse Khomh

We are pleased to announce Foutse Khomh, Professor of Software Engineering at Polytechnique Montréal, will deliver the inaugural seminar of the System & Software research theme during his visit to Edinburgh Napier University.  The seminar will be hybrid and is open to all academics and students based at Scottish universities.

Title

Towards Reliable Machine Learning Models for Code

Abstract

Large Language Models (LLMs) trained on code are increasingly being integrated into software engineering workflows, supporting tasks such as code synthesis, bug fixing, and refactoring. While their empirical performance is often remarkable, several foundational questions remain insufficiently addressed: What do these models learn? How do they reason about source code? Under what circumstances do they fail, and why.

This talk presents findings from a series of empirical investigations that seek to address these questions. First, I will examine how LLMs complement the capabilities of human developers and highlight recurring patterns of bugs and inefficiencies in their generated outputs. Second, I will discuss a systematic evaluation of behavioral regressions across model updates, focusing on logical correctness, code quality, and runtime performance.

Finally, I will introduce PrismBench, a dynamic, multi-agent benchmarking framework designed to rigorously assess and characterize the failure modes of LLMs on code generation tasks. Collectively, these studies aim to advance our understanding of LLM behavior in software engineering contexts and inform the development of more robust and trustworthy AI-assisted programming tools.

Bio

Foutse Khomh is a Full Professor of Software Engineering at Polytechnique Montréal, a Canada Research Chair Tier 1 on Trustworthy Intelligent Software Systems, a Canada CIFAR AI Chair on Trustworthy Machine Learning Software Systems, an NSERC Arthur B. McDonald Fellow, an Honoris Genius Prize Laureate, and an FRQ-IVADO Research Chair on Software Quality Assurance for Machine Learning Applications. In 2011, he received a Ph.D. in Software Engineering from the University of Montreal, with the Award of Excellence. He has received a CS-Can/Info-Can Outstanding Young Computer Science Researcher Prize for 2019, the Excellence in Research and Innovation Award of Polytechnique Montréal, and the prestigious IEEE CS TCSE New Directions Award in 2025. His work has received four ten-year Most Influential Paper (MIP) Awards, eight Best/Distinguished Paper Awards at major conferences, and two Best Journal Paper of the Year Awards. 

Professor Khomh has initiated and co-organized the Software Engineering for Machine Learning Applications (SEMLA) symposium and the RELENG (Release Engineering) workshop series. He also co-organized the FM+SE Summit series (https://fmse.io/), a platform where leading industrial and academic experts discuss and reflect on the challenges associated with the adoption of foundation and large models in software engineering. In addition, he is co-founder of the NSERC CREATE SE4AI: A Training Program on the Development, Deployment, and Servicing of Artificial Intelligence-based Software Systems and one of the Principal Investigators of the DEpendable Explainable Learning (DEEL) project. He is also a co-founder of Quebec's initiative on Trustworthy AI (Confiance IA Quebec) and Scientific co-director of the Institut de Valorisation des Données (IVADO). He is on the editorial board of multiple international software engineering journals (e.g., TOSEM, IEEE Software, EMSE, SQJ, JSEP) and is a Senior Member of IEEE.

Systems and Software Mailing list 

To receive emails about similar events, subscribe to the SICSA System and Software Engineering mailing list: https://sicsa.ac.uk/contact/