Benchmark V1 Methodology

Benchmark V1 evaluates text-to-image models with fixed prompts, fixed scoring questions, and a fixed multi-VLM routing strategy. It is designed to be reproducible and fully automated — the goal is a meaningful ordering of models at capability level, even when individual tests are imperfect.

Three scores: Capability, Preference, Overall

The headline number is Overall, an equal-weight blend of two axes: Overall = 0.5 × Capability + 0.5 × Estimated Preference. Capability is the routed multi-VLM PASS/FAIL pass rate over all 192 images — did the model do what the prompt asked. Estimated Preference is a separate aesthetic axis: how much a human would prefer the image, independent of prompt adherence. You can still sort by either axis alone.

Benchmark structure

V1 contains 64 tests. Each test has 3 prompt variants, so each model run produces 192 images across six categories:

• Text Rendering (5 tests) — spelling accuracy, multi-line layout, typography style
• Spatial Reasoning (19 tests) — counting, relative position, scale, compositionality
• Human Realism (14 tests) — faces, expressions, hands, full-body coherence
• Professional Studio (9 tests) — camera and lighting control, color precision
• Graphical Design (8 tests) — layout, style diversity, data visualization
• Truthfulness (9 tests) — physics, reflections, world-knowledge constraints

Evaluation pipeline

• Generate 192 images for the target model from the fixed V1 prompt suite.
• Ask vision judges a concrete binary question per image and parse a PASS/FAIL verdict.
• Apply category-level preferred/fallback routing to produce one blended verdict per image.
• Aggregate overall, category, subcategory, and difficulty pass rates for publication.

Each question is written for its test. A spatial test asking for “three red balls on the left and two blue cubes on the right” gets a question that asks exactly that — anchoring the score to prompt intent, not generic aesthetics.

Scoring

The headline capability score is the blended routing pass rate: score = PASS count / total evaluated images. The same formula applies per category, subcategory, and difficulty tier, so you see not just the rank but exactly where a model excels or struggles.

Limitations

• VLM judges can hallucinate or be inconsistent on edge cases; routing reduces but does not eliminate this.
• 64 tests is a small set — three prompt variants per test mitigate variance but do not remove it.
• The benchmark measures capability adherence, not aesthetic quality; a model can score low and still make beautiful images.
• Scores reflect the model at evaluation time. API models change without notice; local models are pinned to checkpoints.