Published benchmark reference

Benchmark structure

64 tests × 3 prompt variants = 192 images per model. Categories match the upstream CSV exactly.

• Text Rendering5 tests — Spelling, multi-line layout, typography style
• Spatial Reasoning19 tests — Counting, relative position, scale, compositionality
• Human realism14 tests — Faces, expressions, hands, full-body coherence
• Professional Studio9 tests — Camera and lighting control, color precision
• Graphical design8 tests — Layout, style diversity, data visualization
• Truthfulness9 tests — Physics, reflections, world knowledge constraints

Design goals

• Broad capability coverage across six categories and multiple difficulty tiers.
• Three prompt variants per test to reduce lucky or unlucky single-sample variance.
• VLM-graded PASS/FAIL decisions using concrete, test-specific evaluation criteria.
• Reproducibility through fixed prompt definitions and deterministic routing policy.

Evaluation pipeline

1. Generate 192 images for the target model from the fixed V1 prompt suite.
2. Ask vision judges a concrete binary question per image and parse a PASS/FAIL verdict.
3. Apply category-level preferred/fallback routing to produce one blended verdict per image.
4. Aggregate overall, category, subcategory, and difficulty pass rates for publication.

Multi-VLM routing

ImageBench assigns a preferred vision judge per category. Ghost Palette maps these routes to Gemini 2.5 Flash via fal OpenRouter for grading.

Scoring

Capability pass rate: score = PASS count / total evaluated images, reported per category, subcategory, and difficulty tier. The headline Overall score blends it 50/50 with an aesthetic Estimated Preference axis: Overall = 0.5 × Capability + 0.5 × Estimated Preference. See the methodology write-up for the full definition.

Published V1 leaderboard

Copied from imagebench.ai/imagebench-v1. Re-sync with node scripts/sync-imagebench-leaderboard.mjs.

Published pass rates from ImageBench.ai as of 2026-06-22. Every output is inspectable on their site — click a model to browse all 192 images.