Evaluating multimodal commercial and open-source large language models for dynamical astronomy: a benchmark study of resonant behavior classification(pdf)
Smirnov, Evgeny, Carruba, Valerio · 2026 · Scientific Reports
At a GlanceAI
This benchmark shows multimodal LLMs can classify orbital resonances from images with high accuracy, even without task-specific training.
SummaryAI
The study tests whether modern multimodal language models can recognize resonant behavior in dynamical astronomy directly from images, a task usually handled by specialized methods. It introduces four benchmark datasets spanning easy, ambiguous, and transient cases, and compares commercial, open-source, and small local models using standardized prompts. The results show that top commercial models and some open-source models achieve very strong performance, with most mistakes concentrated in the hardest transient regimes. By releasing these benchmarks, the work provides a reproducible way to assess LLMs for astronomy tasks and suggests they may serve as practical tools even without fine-tuning.
This work establishes the first comprehensive benchmark for evaluating multimodal large language models on resonance classification in celestial mechanics. Its standardized datasets, evaluation methodology, and comparison of commercial and open-source models provide a foundation for future research on generative AI in dynamical astronomy. Essential reading for anyone interested in the intersection of large language models and Solar System dynamics.
— VC
- Method:AI
- The authors benchmark multimodal large language models on image-based resonance classification using standardized prompts across four curated datasets.
- Background:AI
- Familiarity with basic orbital dynamics, resonance concepts, and general ideas behind multimodal large language models is helpful.