Project MEDAL
Mission: Channel the resources of the brightest minds in AI away from irrelevant or trivial problems towards the biggest clinical challenges of our time. The MEDAL benchmark will be able to assess the capabilities of medical imaging AIs by providing highly relevant and challenging medical imaging tasks crowdsourced from the entire world - with a strong incentive in the shape of 1 Mil EUR to be distributed among data contributors.
About Project MEDAL
Data is a key driving force behind progress in artificial intelligence (AI), particularly in medical imaging. The provision of high-quality, relevant datasets can be used to strategically channel international scientific and technical resources toward solving the most pressing and impactful clinical problems. Benchmarks provide a framework for evaluating AI systems on such data and can thereby help shape the direction of research. However, existing benchmarks often fall short by focusing on narrow, repetitive, and clinically irrelevant tasks. To address this critical gap, Project MEDAL introduces a paradigm-shifting, top-down approach to benchmark design. MEDAL aims to create the world's most challenging and clinically impactful benchmark, providing a rigorous framework for evaluating whether AI systems in medical imaging can deliver transformative clinical utility.
Through a global crowdsourcing campaign, Project MEDAL will collect high-stakes, real-world clinical questions and multimodal data that reflect the complexity and variability of everyday medical practice. A multidisciplinary, international consortium, which comprises leaders from diverse fields including AI, clinical medicine, global health, biomedical ethics, and regulatory science, will guide its design and implementation to ensure alignment with real clinical needs. The project tackles four core limitations of existing benchmarks: low diversity (addressed via global crowdsourcing), limited clinical relevance (countered by a top-down approach driven by diverse stakeholders), data sparsity (mitigated through incentivized contributions), and benchmark saturation (countered by using unpublished datasets). The result will be a trustworthy benchmarking framework for evaluating and advancing AI in healthcare, helping to direct research efforts away from trivial exercises and toward addressing the most pressing challenges in medical imaging.
Project MEDAL is funded by the Carl-Zeiss-Stiftung (CZS) and is a joint initiative of the MICCAI Special Interest Group for Challenges and Helmholtz Imaging. The project is led by a core team with extensive experience in coordinating large international consortia and community-wide efforts to improve benchmarking and validation in biomedical imaging. This includes research on the reliability and interpretation of biomedical AI challenges [1; 2], guidelines for transparent competition reporting [3], and Metrics Reloaded [4], which established standardized practices for performance evaluation.
Building on this foundation, Project MEDAL will engage international communities through a global crowdsourcing campaign. Funding from CZS includes €1 million dedicated to incentivizing data contributions to the MEDAL benchmark.
MEDAL universe
The MEDAL universe is centered on a shared mission: directing medical imaging AI research toward clinically important problems where advances in AI could make a meaningful difference. The MEDAL satellite projects contribute insights by tackling different aspects of this mission, such as mapping the landscape of clinical needs for AI in oncology, investigating current model capabilities and developing approaches for robust evaluation.
COMPASS takes a clinical perspective and maps where medical imaging AI could have the greatest potential to improve clinical practice.
The project will engage the clinical community to identify pressing questions, unmet needs, emerging topics, and common priorities across oncology through a public submission form, open to contributors worldwide.
A global expert consensus process will help determine which clinical questions could benefit most from advances in AI.
These insights will help MEDAL direct research attention toward problems with high potential impact rather than topics that primarily reflect the availability of existing datasets.
Where relevant, they will also guide the solicitation of data contributions for the broader MEDAL Benchmark.
Explore COMPASSDYNAMO addresses the limitations of static benchmarks such as data leakage, data contamination, and repeated optimization, by continuously incorporating new tasks and unpublished test data from MICCAI challenges
This dynamic approach reduces the aforementioned risks while allowing model capabilities to be tracked as both AI systems and clinical tasks evolve.
Due to the diversity of MICCAI challenges, DYNAMO can systematically investigate the capabilities and limitations of current multimodal AI models across different diseases, imaging modalities, institutions, and task types.
Explore DYNAMOAdversarial Anatomy Bench complements DYNAMO on the technical side by investigating how well current vision-language models generalize beyond common and expected anatomical presentations.
The project uses rare anatomical variants to test whether models rely on the visual information in an image or on learned expectations about typical human anatomy.
These cases help uncover important limitations in current model capabilities that conventional evaluation may overlook.
Explore Adversarial Anatomy BenchThe MEDAL Benchmark will develop rigorous evaluation tasks informed by clinical and technical perspectives, including relevant insights from the satellite projects.
A global crowdsourcing campaign will collect high-stakes clinical questions and multimodal data that reflect the complexity and variability of real-world medical practice.
A funding of €1 million for data contributors will create a strong incentive for the global community to contribute diverse, high-quality clinical data.
Clinical priorities identified through satellite projects such as COMPASS will help determine where medical imaging AI has the biggest potential, while insights from DYNAMO, Adversarial Anatomy Bench, and related work will inform how MEDAL evaluates the capabilities required to address them.
A multidisciplinary international expert consortium will guide benchmark development to ensure that the tasks, data, and evaluation criteria are aligned with real clinical needs.
Explore MEDAL Benchmark