Machine Learning Seminar Series Fall 2025 | Asta: Building and Evaluating Scientific AI Assistants

Abstract: AI has tremendous potential for accelerating science---helping researchers access the literature, analyze data, execute experiments, and more. Asta, a project at the Allen Institute for AI, aims to deliver comprehensive scientific assistants that support all aspects of a scientific workflow.  Rigorously evaluating such systems is a fundamental challenge, which led us to develop AstaBench, a holistic benchmark suite and leaderboard spanning the scientific discovery process.  The work includes evaluations for scientific “deep research” systems, introducing methods for measuring coverage and precision of both answers and their attributions. Fully realizing AI's potential for scientists will require breakthroughs in scalable supervision, robust evaluation, and personalized and proactive assistance.  To facilitate progress toward these goals, the Asta project provides open-source corpora, tools, and APIs.

Bio: Doug Downey is Sr. Director of Semantic Scholar Research at the Allen Institute for AI (Ai2) and a Professor of Computer Science at Northwestern University. His research focuses on Natural Language Processing, with an emphasis on AI systems that support scientific discovery. At Ai2, he helps lead the Asta project, and his publications have received paper awards at ACL, EMNLP, IJCAI, and AKBC.