New benchmark confirms AI models still perform poorly at visual perception

Moonshot AI’s PerceptionBench tests how well multimodal AI models can actually “see,” separate from logical reasoning. No frontier model reaches 60 percent accuracy, and GPT-5.6 Sol leads by a narrow margin.

Many supposed reasoning errors actually happen as early as the image-reading stage. The team behind the Chinese AI assistant Kimi has introduced PerceptionBench, a benchmark that isolates and tests the visual perception of multimodal language models. Unlike standard testing methods, PerceptionBench breaks vision down into ten atomic sub-skills instead of lumping perception, knowledge, and reasoning into a single task. Every question can be answered just by looking at the image, with no reasoning or outside knowledge required. The authors explain their approach by pointing out that existing benchmarks each capture only a narrow slice of perception errors. The 42 open-source benchmarks they analyzed show little overlap in their error profiles, so each one covers a different subset of visual weaknesses. No single test or small group of tests was enough to capture visual perception as a whole.Ad Rather than defining categories up front, the authors built their taxonomy from actual model errors and traced each one back to the earliest failed step in existing benchmarks. The result is ten “skill domains”: Visual Relation, Counting, Attributes, Depth & 3D, Localization, Comparison, Fine-grained Recognition, Context Integration, OCR, and Hallucination.Ad From an internal pool of over 17,000 verified questions, Moonshot AI is publishing 3,000 tasks. Sixty percent were derived from attributed model errors, while 40 percent were reformulated using augmented images. The tasks seem trivial on the surface: figuring out where a symbol sits on a clock face, counting flowers inside a red box, or deciding which of two pencil cups shows a gray-pink combo versus a solid pink with a cartoon design. Among the 16 frontier models tested, the highest overall accuracy is 59.7 percent, scored by GPT-5.6 Sol. Kimi K3 follows at 58.5 percent, Claude Fable 5 at 57.2 percent, and Gemini 3.1 Pro at 56.2 percent. GPT-5.5 comes in at 55.8 percent. Open-source models like Qwen3.5-397B-A17B (47.5 percent) and GLM-4.6V (32.5 percent) trail far behind.Ad The category-level results are more telling than the overall ranking. Models with nearly identical aggregate scores diverge sharply in individual categories.