AI Enterprise SolutionsFinalistGPT & Claude

DiabloAI

The user: the ML or evaluation engineer, or the AI product lead, who owns an LLM feature in production — for example the team behind a telecom's customer-support assistant. The problem: their AI changes every week. Prompts are rewritten, models are swapped, temperature and tools are tuned, often several at once (in Amplify Partners' 2025 survey of ~500 AI engineers, 70% update their underlying model at least monthly). When answer quality drops, today's eval dashboards and tracing tools show THAT a score moved — 86% to 71% — but not WHICH change caused it, or whether the drop is real or noise. So the team argues, rolls back everything, or ships the regression. Each wrong guess costs engineering days and customer trust, and AI is now in regular use at 88% of organisations (McKinsey, State of AI 2025), so this happens in almost every company that ships AI. What Diablo does: you ask "why did it get worse?" in plain language. Diablo designs competing hypotheses, runs one controlled, paired experiment per suspected change on the real model, and answers which change caused it — with the size of the effect, a 95% confidence interval, an exact statistical test and a grade for how much to trust the result. The AI does the reasoning; code computes every number, so the answer cannot be hallucinated.

Open the demo Video GitHub

Preliminary score

84.0/ 100Strongaverage of 2 judges

Value for the user21.0 / 25
A specific problem and a clear outcome from solving it.
Prototype and use of AI23.5 / 30
A working core scenario, and what AI actually contributes to the result.
Quality testing18.0 / 20
Tests, examples of failures, and comparison with the current approach.
Feasibility13.5 / 15
Data requirements, running costs and a clear next step.
Originality8.0 / 10
A meaningful difference from common solutions.

A finalist: the live jury scored it 68.0 / 100 on stage.

What the judges wrote

Claude Opus 5.5Preliminary85/100
Strengths
Your problem framing is sharp. A release changed several things, quality dropped, and the team needs to know which change to revert. The worked McNemar/Holm example makes the outcome concrete. Your split between the AI and the code is well engineered: the model drafts hypotheses and studies, code runs the paired experiments and computes every statistic, and a grounding checker rejects any number the model writes. Your testing is unusually honest and systematic. It includes 293 unit tests checked against SciPy/statsmodels, a 45,000-scenario planted-cause benchmark compared with the 'blame the biggest drop' habit, real runs that report a too-easy first test, and openly admitted limits such as cautious recall and rule-based checks rejecting correct replies.
To improve
Your benchmark measures the statistical protocol, not the LLM investigator. Add an evaluation of the investigator itself, for example how often its drafted hypotheses and test cases include the true planted cause across many real runs. The model under test is currently one hosted model varied only by prompt and temperature. A connector to even one real third-party system, or a replayed real regression, would show the core enterprise scenario end to end. Your deck and submission disagree on several points: 251 vs 293 tests, Haiku 5.5 vs 4.5, and whether the Claude connection is live or 'being added'. Align them so judges can trust the claims.
GPT-6 AstraPreliminary83/100
Strengths
Your project targets a concrete engineering decision: which change to roll back after an AI regression. Your separation of AI-generated hypotheses from code-computed statistics is meaningful, and your reported real-model run includes outcomes, runtime, cost and a grounding-check failure. Your testing stands out for its baseline comparisons, explicit precision-versus-detection trade-off, and honest disclosure of overly easy cases and scorer limitations.
To improve
Your next validation should connect a team's actual AI endpoint and measure diagnosis time and rollback correctness against its existing workflow. Validate generated cases and rule-based scorers against independent human labels: numerical grounding alone does not establish semantic correctness or causal validity. Extend testing to smaller regressions and interacting changes, with sample-size planning rather than relying on the dramatic 40/40-to-0/40 example.

The team

Ilham Orujov
developer
Jahad Abdulla
developer
Usubzada TunarLead
data

Disclosure

Required by the rules. Judges verify it against the repository.

Models: Claude Opus 5.5 — the investigator: designs the study for the user's question and writes the answer. Claude Haiku 4.5 — the model under test. Optional admin switch: Gemini 3.8 Flash (investigator) and Gemini 3.5 Flash-Lite (under test). Python engine (experimental): Gemini via google-genai. Development: built with AI coding assistance (Claude Code). No model is trained or fine-tuned, and no model output is ever used as a number or a score: every check and statistic is computed by code.
Data: No training data, no customer data, no personal data, nothing scraped. Live investigations: test cases are written for each question by the investigator model and checked by code; nothing is collected or stored. Default scenario (a run with no question): seeded synthetic arithmetic with exact answers computed by code. Benchmark: 45,000 seeded simulated scenarios. Tests use hand-written fixtures. Google sign-in keeps only name and email in a signed cookie; there is no database.
Components: Libraries: Next.js 16, React 19, Tailwind CSS 4, zod, motion, jose, Radix UI, cmdk, lucide-react; Vitest, Playwright, axe-core for testing. Services: Vercel (hosting, Blob storage), Google OAuth. Fonts: Geist, Geist Mono, Newsreader (SIL Open Font License). No public UI templates or starter kits were used. Full list with versions: docs/DISCLOSURE.md in the repo.
Built during the event: Confirmed

Quality testing

Automated: 293 unit tests pass, including statistics verified against SciPy/statsmodels to within 5×10⁻⁵, plan validation, the paired runner, the code checks, and the grounding checker that rejects any number the model types itself. End-to-end browser tests run with axe accessibility checks; typecheck and lint are clean. Benchmark against today's habits (seeded simulation of 45,000 releases with a known planted cause; no LLM involved, so it tests the measuring protocol, not the AI). When Diablo names a cause, it is right 97.3% of the time; "blame the biggest drop" — what a team does with a dashboard — is right 71.7% of the time. When nothing actually changed, Diablo still names a cause in only 1.1% of releases, against 81.0% for that shortcut. The price of that reliability: Diablo is cautious. It names the cause in 41% of scenarios and misses small 5-point drops, saying "no attributable cause" rather than guessing. Real runs on production models: 1) First run (Claude Opus 5.5 investigating Claude Haiku 4.5): every setup scored 100% — our test was too easy. Diablo reported "no measurable effect" and invented nothing. We made the items harder. 2) Rerun: Diablo found the planted cause. The shortened prompt took accuracy from 40/40 to 0/40 (exact McNemar p < 0.001); the temperature change, 40/40 to 39/40, showed no clear effect. 123 model calls, 63 seconds, about $0.20. The model's first written conclusion used a number word ("zero"); the checker rejected it and the published answer cites only code-computed values. 3) A user's own question — "Why do some biodegradable plastics take years to decompose in landfills?": Diablo built a 12-case study with three setups and found that a plain-language prompt cut technical precision from 100% to 66.7%, while an expert persona made no difference. An earlier run of the same question hit 100% everywhere, so the investigator is now instructed to write cases hard enough to tell the hypotheses apart. What still breaks: the investigator sometimes writes cases that are too easy (reported honestly as "no clear effect"); rule-based checks can fail a correct but unusually phrased reply (every reply is shown with its check); there is no connector to a customer's own AI system yet. Compared with how it is done today: a dashboard shows 86% → 71% with no interval and no cause. Diablo runs one paired experiment per suspected change and returns which change caused it, by how much, with a 95% confidence interval, an exact test, a multiple-comparison correction and an evidence grade — all traceable to the raw replies.

Worth 20 of the 100 points.

Running it

1. Open https://app.diablo.pnoia.dev and click "Continue with Google" (any Google account; nothing to install). 2. Type a question about how an AI behaves, for example: "Does asking for a one-word answer make the assistant worse at multi-step arithmetic?" — or open /live and click an example. 3. Watch it work (about 1–2 minutes): the investigator designs a study for your question, real model calls run, code computes the statistics, and a number-checked answer appears. Tabs show the results table, the study design, every reply with its check, and the run record (calls, tokens, cost). 4. The investigation is saved in the sidebar; "Open in workspace" shows its research graph, evidence and printable report. Models: Claude Opus 5.5 investigates; Claude Haiku 4.5 is the model under test. One run per user at a time, with a 30-second pause between runs. Run locally: npm install; set AUTH_SECRET, GOOGLE_CLIENT_ID, GOOGLE_CLIENT_SECRET and ANTHROPIC_API_KEY; npm run dev. npm test runs 293 automated tests. Known limits: the model under test is one hosted model, varied by system prompt and temperature (connecting a customer's own AI system is the next step); checks are rules, so every reply is shown for review.

Project ownership

The team keeps the rights to this project. Judges were given access for verification only.