Astronomy16 papers
Artificial Intelligence and Machine Learning in Planetary Science
A companion collection for the ACM 2026 talk by Valerio Carruba
VC
Valerio Carruba
All Reviews
AstronomyMust Read
intermediate
Evaluating multimodal commercial and open-source large language models for dynamical astronomy: a benchmark study of resonant behavior classification
Smirnov, Evgeny & Carruba, Valerio (2026)
- Published
- Mar 28, 2026
- Journal
- Scientific Reports · Vol. 16 · No. 1
- DOI
- 10.1038/s41598-026-45926-y
At a GlanceAI
This benchmark shows multimodal LLMs can classify orbital resonances from images with high accuracy, even without task-specific training.
SummaryAI
The study tests whether modern multimodal language models can recognize resonant behavior in dynamical astronomy directly from images, a task usually handled by specialized methods. It introduces four benchmark datasets spanning easy, ambiguous, and transient cases, and compares commercial, open-source, and small local models using standardized prompts. The results show that top commercial models and some open-source models achieve very strong performance, with most mistakes concentrated in the hardest transient regimes. By releasing these benchmarks, the work provides a reproducible way to assess LLMs for astronomy tasks and suggests they may serve as practical tools even without fine-tuning.
Method SnapshotAI
The authors benchmark multimodal large language models on image-based resonance classification using standardized prompts across four curated datasets.
BackgroundAI
Familiarity with basic orbital dynamics, resonance concepts, and general ideas behind multimodal large language models is helpful.
This work establishes the first comprehensive benchmark for evaluating multimodal large language models on resonance classification in celestial mechanics. Its standardized datasets, evaluation methodology, and comparison of commercial and open-source models provide a foundation for future research on generative AI in dynamical astronomy. Essential reading for anyone interested in the intersection of large language models and Solar System dynamics.
— VC