{title}
+{body}
+{title}
+{body}
++ Methodology at a glance +
++ We hold constant +
++ Same exercises, same tools, same time limit, same scoring. +
++ We measure +
++ Pass rate, latency, and cost signals across multiple languages. +
++ We recommend +
++ A default model and agent lineup for an objective. It’s a + baseline, not a guarantee. +
++ Evals help you pick a better default. They don’t predict how a model behaves + in your repo, with your tests, tooling, and constraints. +
+ ++ Quick definitions +
++ Methodology +
++ The recommendations page is organized around what you’re trying to ship. You + pick the objective and tradeoff. We show the best default setup based on the signal + we have. +
++ Scoring and signals +
++ Tradeoffs +
++ Choosing an optimization mode is how you tell the system what matters most for + your objective. If you care about merge confidence, optimize for Quality. If you + care about throughput, Speed and Cost matter. +
++ When two models are close on pass rate, the most practical tie-breakers are + latency and $/task. +
++ Limitations +
++ A model can score well on this suite and still struggle in your repo because + your stack, tests, dependencies, and CI constraints are different. +
++ The right move is to treat our results as a starting point, then run your + objective end-to-end in Roo Code Cloud and inspect the PR output. +
++ Links +
++ If something feels off in the recommendations, that’s a signal too. The + fastest path is to run your objective in Roo Code Cloud and compare the PR + output. +
+Data preview
++ Composite score vs estimated daily cost for the primary agent, with dot size mapped to success rate. +
+ +{p.name}
++ {shortProvider(p.provider ?? "")} · {p.score} score ·{" "} + {p.successRate}% success +
++ Est daily cost {formatDollars(p.dailyCost ?? NaN)} +
++ {objective.description} Explore why this setup is recommended, compare options, and + tune it to your constraints before you start in Roo Code Cloud. +
+ ++ Configuration +
+ ++ Optimized for +
+Primary model
++ {primaryCandidate ? ( + <> + + {primaryCandidate.displayName} + {" "} + + ({shortProvider(primaryCandidate.provider)}) + + + ) : ( + + Pick an objective to see recommendations. + + )} +
+ {primaryCandidate ? ( ++ Score +
++ {primaryCandidate.compositeScore} +
++ Success +
++ {primaryCandidate.successRate}% +
++ Daily +
++ {formatDollars(primaryCandidate.estimatedDailyCost)} +
++ This is a starting point. Adjust the lineup and model picks to match your + repo and delivery constraints. +
++ For {objective.name}, this lineup balances momentum and safety. Swap models per agent if + you have different constraints. +
+ ++ For +
++ {rec.role.name} +
++ {rec.role.salaryRange} +
++ Success +
++ {selected ? `${selected.successRate}%` : "—"} +
++ Daily +
++ {selected + ? formatDollars(selected.estimatedDailyCost) + : "—"} +
++ Time +
++ {selected + ? formatSeconds(selected.avgTimePerTask) + : "—"} +
++ Why this works +
++ How it runs +
++ Example prompt +
+
+ {examplePrompt}
+
+ + {candidate.provider} +
+ + {/* Model name */} +Success
+{candidate.successRate}%
++ Daily Cost +
++ ${Math.round(candidate.estimatedDailyCost)}/day +
++ (${candidate.avgCostPerTask.toFixed(3)}/task) +
+Avg Time
++ {candidate.avgTimePerTask.toFixed(1)}s +
+
+
+ {candidate.provider} +
+{candidate.displayName}
+ + {/* Score + tier */} ++ {highlight === "cost" ? "Daily Cost" : "Avg Time"} +
++ {highlight === "cost" + ? `$${Math.round(candidate.estimatedDailyCost)}/day` + : `${candidate.avgTimePerTask.toFixed(1)}s`} +
+ {highlight === "cost" && ( ++ (${candidate.avgCostPerTask.toFixed(3)}/task) +
+ )} +Success
+{candidate.successRate}%
++ Daily Cost +
++ ${Math.round(candidate.estimatedDailyCost)} +
++ (${candidate.avgCostPerTask.toFixed(3)}/task) +
+Time
+{candidate.avgTimePerTask.toFixed(1)}s
++ {role.description} +
+| + # + | ++ Model + | ++ Provider + | ++ Score + | ++ Tier + | ++ Success + | ++ Daily Cost + | ++ Time + | ++ {i + 1} + | +{candidate.displayName} | ++ {candidate.provider} + | ++ + {candidate.compositeScore} + + | ++ + {tier.label} + + | ++ {candidate.successRate}% + | ++ ${Math.round(candidate.estimatedDailyCost)} + + (${candidate.avgCostPerTask.toFixed(3)}) + + | ++ {candidate.avgTimePerTask.toFixed(1)}s + | +
|---|
{label}
+{data.name}
++ {role.name} +
++ Interactive comparison across composite score, success rate, cost efficiency, and + speed. Filter by provider, language, and minimum success rate. +
++ Upper-left = higher score at lower spend. Each dot is a model. Size reflects success rate. +
+ + {/* Tier legend */} +No models match the current filters.
++ Try adjusting the provider or success rate filters. +
++ Cost Efficiency and Speed are inverted — higher bars mean cheaper / faster. Daily costs + assume ~{TASKS_PER_DAY} tasks per agent per day (~6 productive hours). +
+ + {chartData.length === 0 ? ( +No models match the current filters.
++ Try adjusting the provider or success rate filters. +
+{data.modelName}
++ Profile +
++ {selectedOutcome.name} +
++ {profileDescription} +
++ Example prompt +
+
+ {examplePrompt}
+
+ + Optimized for +
++ {selectedModeLabel} +
++ Default model +
++ {objectiveDefaultModelLabel} +
++ Agents +
++ {agentCapabilities.length} +
++ Skills +
++ {skillCapabilities.length} +
++ Optimize for +
++ Agent lineup +
++ {capability.name} +
++ + {candidate ? ( + + {candidate.displayName} + + ) : ( + + View models + + )} +
++ Skills included +
++ {capability.name} +
++ {capability.description} +
++ No skills listed for this profile yet. +
+ )} ++ Rationale +
++ {selectedOutcome.builderProfile + ? "How it works" + : "Why it works"} +
++ Why it works +
++ No objective selected +
++ Pick an objective first, then open the profile view. +
++ {outcome.name} +
++ {outcome.description} +
++ Profile snapshot +
++ {selectedOutcome + ? selectedOutcome.name + : "Pick an objective"} +
++ {selectedOutcome + ? profileDescription + : "Select an objective to see the suggested lineup and default model."} +
++ Signal +
++ {totalEvalRuns.toLocaleString()} runs +
++ Agents +
++ {agentCapabilities.length} +
++ Default model +
++ {objectiveDefaultModelLabel} +
++ Agent lineup +
++ Open candidates & settings +
++ {capability.name} +
++ + {candidate ? ( + + {candidate.displayName} + + ) : ( + + View models + + )} +
++ No objective selected +
++ Pick an objective to see the recommended agent lineup. +
++ Example prompt +
+
+ {examplePrompt}
+
+ + Pass rates on our eval suite by model release date. Several current models hit 100% on + this suite. +
+