Optima tackles AI benchmarking’s biggest flaw by letting users test models against their own data

Optima lets users build custom benchmarks and compare AI models on quality, cost, and speed for their specific use cases. Artificial Analysis, known for its independent LLM evaluations and benchmark implementations like GDPval-AA and AA-Briefcase, has launched a new platform called Optima.

The premise is simple: public benchmarks compare models on predefined tasks and criteria, but they don’t necessarily reveal which model works best for a particular use case. Optima aims to close that gap with comparisons tailored to individual workflows. Users can build their own benchmarks using their own data, workflows, or descriptions of a use case, then run them across leading current models and compare results on quality, cost per task, and time per task, according to Artificial Analysis. Optima is available now.Ad Optima accepts several types of source material. Users can upload existing evaluation datasets from their own files or from Hugging Face, as well as AI agent traces from platforms like Arize, Braintrust, or Langfuse. Developers can also install a skill that gathers information from their coding environment and past sessions, Artificial Analysis writes.Ad Users who don’t have that kind of data can instead describe their intended use case and provide sample inputs and outputs. Optima then generates suggested test inputs, evaluation criteria, and example tasks. Users can review and refine these through feedback before running the actual benchmark. Two scoring approaches are available: rubric-based evaluation against objective criteria, or a pairwise comparison method that Artificial Analysis also uses for benchmarks like GDPval-AA and AA-Briefcase. In the pairwise approach, users first evaluate a sample of response pairs and indicate which answer they prefer. Optima then derives the full ranking across the test dataset from those preferences.Ad Cost and speed become first-class comparison metrics Beyond raw model quality, Optima tracks cost per task and time per task as standalone comparison dimensions. This makes it possible to check whether a performance gain actually justifies the higher cost or longer processing time of a given model. For agentic applications, raw token price alone tells you very little. A cheaper model can end up costing more overall if it needs more attempts, fails more often, or requires extra cleanup work. Cost per completed task is often the more meaningful number.Ad Early testers built benchmarks for finance and accounting agents to find which model could cut costs by a factor of ten without major quality loss, according to Artificial Analysis.