In the landscape of black-box optimisation, benchmarking plays a crucial role in understanding algorithm behaviour and guiding algorithm design. Yet, much of current practice remains focused on surface-level comparisons, offering limited insight into why certain algorithms perform well and how specific components contribute to their success or failure.

This talk explores a new direction: explainable benchmarking. Drawing on techniques from explainable AI, it becomes possible to move beyond performance tables, towards uncovering the underlying mechanisms that drive algorithmic behaviour. The IOHxplainer framework demonstrates how performance data—collected across a broad range of configurations and problem instances—can be analysed to attribute outcomes to individual components and hyperparameters, especially for highly modular and parameterised algorithm designs. Examples from recent studies on modular CMA-ES illustrate how these methods reveal evolving module contributions over time, quantify sensitivity to random seeds and problem instances and identify design choices that genuinely matter. Being robust and scalable, this approach opens up new possibilities for principled algorithm development, configuration and selection.

The talk concludes with reflections on the future of benchmarking: how to make it more representative, more insightful and ultimately more useful for building optimisation methods that are not only powerful but also better understood.