AI Enterprise SolutionsGPT & Claude

Slytherin

Primary user: the internal control / compliance officer (or finance manager) at a mid-sized company who owns the expense, procurement or HR policies and has to sign off that they are sound before they go into force. In companies without a dedicated audit team, this is often the CFO or an HR/finance manager doing it alongside their main job. What goes wrong today: policies are reviewed by people re-reading them. Gaps like "no cap on how many nights", "no rule that the approver must be a different person" or "deadline with no start date" are invisible to the author because the author reads the policy the way it was meant. They are found by employees, not reviewers, and usually only after money has been lost or an external audit flags it. Hiring a specialist to adversarially test every policy revision is slow and costly, so most revisions are never tested at all. What Policy Fuzzer does: the user pastes a policy. Three adversarial AI personas (fraudster, lazy employee, legal technicalist) try to defeat each rule using only its literal wording. A separate judge agent rejects scenarios that need lying or forged documents, so only genuine policy flaws remain. Each unique loophole comes with a severity, a concrete replacement wording, and an illustrative exposure estimate per incident in AZN. How we measured it: on a synthetic expense policy with 10 planted loopholes, the system found 7-10 of 10 across runs, and every result is reported with its limits (synthetic policy, precision is a lower bound, estimates are illustrative, not real loss statistics).

Video GitHub

Preliminary score

75.5/ 100Strongaverage of 2 judges

Value for the user20.5 / 25
A specific problem and a clear outcome from solving it.
Prototype and use of AI21.5 / 30
A working core scenario, and what AI actually contributes to the result.
Quality testing15.5 / 20
Tests, examples of failures, and comparison with the current approach.
Feasibility10.0 / 15
Data requirements, running costs and a clear next step.
Originality8.0 / 10
A meaningful difference from common solutions.

What the judges wrote

Claude Opus 5.5Preliminary74/100
Strengths
Your problem framing is sharp: a named user (compliance officer or CFO at a mid-sized company), a concrete pain (authors cannot see their own loopholes), and an output a reviewer can act on (severity, replacement wording, exposure estimate). The adversarial design is a real use of AI rather than decoration. Three personas attack the literal wording, a separate judge filters out forgery-based scenarios, and a verbatim-quote check suppresses hallucinated facts. Your evaluation is unusually honest and systematic. You planted 10 loopholes with an answer key, tracked recall and precision across runs, named the specific misses, and explained how the dedupe change moved recall between 70% and 100%.
To improve
You left the re-judging result as a placeholder ('[fill in after rejudge]'), so the effect of your stricter judge, arguably the key quality fix, is not shown. Run the evaluation on the unseen policy2 you already prepared, since results on the tuning policy alone cannot show generalization. Add even a small human baseline, such as one finance person reviewing the same policy for 30 minutes, so you can claim time saved or extra gaps found. Spell out per-run cost in money, not just tokens, and name a pilot partner or real policy as your next step to strengthen feasibility.
GPT-6 AstraPreliminary77/100
Strengths
Your product addresses a concrete sign-off problem for finance and compliance teams, pairing candidate loopholes with replacement wording rather than just warnings. Your adversarial personas, separate judge, and verbatim-quote checks give AI a substantive role in testing policy wording. Your reporting of missed loopholes, rate-limit failures, and deduplication trade-offs is specific and appropriately qualified.
To improve
Your next evaluation should complete the stricter-judge re-scoring, use unseen policies with independently reviewed labels, and compare reviewer time and accepted findings against manual review. Your submission would be stronger with an inspectable end-to-end report showing policy quotations, rejected fraud scenarios, and final proposed fixes. Your deployment plan should quantify monetary cost and latency for 80–100k-token runs and explain how confidential company policies are protected.

The team

Miribrahim HüseynliLead
product
Mirpasha Pashazade
developer
Əli Əsədov
designer

Disclosure

Required by the rules. Judges verify it against the repository.

Models: openai/gpt-oss-120b via Groq API (all agents)
Data: Synthetic: 1 fictional expense policy + answer key of 10 planted loopholes, written by the team
Components: Python, FastAPI, Uvicorn, Pydantic, Groq SDK, plain HTML/JS
Built during the event: Confirmed

Quality testing

We built a synthetic expense policy (fictional Caspian Logistics) with 10 planted loopholes and an answer key, and scored the system by recall (planted loopholes found) and precision (reported loopholes matching the key). Precision is a lower bound: extra findings may be real loopholes nobody planted. Results by run: - Run 1 (before tuning): recall 70% (7/10), precision 71% (10/14). Missed: self-approval, no-receipt claims with no count cap, per diem paid when meals are already covered. - Run 2: recall 80% (8/10), precision 71% (12/17). - Run 3 (with the same 93 verified findings, two dedupe versions): aggressive cross-rule dedupe gave recall 70% (7/10); per-rule dedupe gave recall 100% (10/10), precision 62% (32/52). Both used the earlier, more lenient judge. - Re-judging with the stricter judge: [fill in after rejudge: recall X/10, precision Y/Z]. What broke and what we changed: 1. The model invented dates, vendors and document types. We added a verbatim-quote check: every scenario must quote the policy text, otherwise it is dropped. 2. The judge was too lenient and accepted scenarios that only work by lying (recording a business ticket as economy, listing fictitious clients). We added a rule classing these as forgery, i.e. fraud, not a policy flaw. 3. Deduplication failed on one large batch (rate limit), so the first precision figure was computed on undeduplicated findings. We chunked it. 4. Our first aggressive dedupe merged distinct loopholes and dropped recall from 100% to 70%. We changed it to merge only within the same rule and to keep findings separate when in doubt. Honest limits: the prompts were tuned on this same policy (3 targeted additions), so these numbers are optimistic and not evidence of generalization. We have not yet run an unseen policy. LLM output varies between runs, so treat the figures as ranges. Run 3's recall of 100% and 70% come from the same findings, not from independent runs. Exposure figures in AZN are illustrative estimates derived from scenarios, not real loss statistics. Comparison with today: these policies are checked by people re-reading them. We did not measure a human baseline, so we make no claim of being better than manual review, only that the tool surfaces candidate gaps with concrete fixes for a reviewer to confirm.

Worth 20 of the 100 points.

Running it

Requirements: Python 3.10+, a free Groq API key (https://console.groq.com). 1. Install: python -m pip install -r requirements.txt 2. Create a .env file in the project root AND in backend/ (same content): GROQ_API_KEY=<your Groq key> MODEL=openai/gpt-oss-120b REASONING_EFFORT=low USE_MOCK=false 3. Start the server from the project root: python -m uvicorn backend.api:app --reload --port 8000 4. Open frontend/index.html in a browser, paste a policy (e.g. data/policy1.txt) and click Analyze. No API key? Judges can still see a real saved run: with the server running, open http://localhost:8000/demo (served from cache/last_run.json, no LLM call). With USE_MOCK=true (the default if .env is missing) the API returns a tiny built-in mock instead of calling the model. Reproduce the evaluation (from the project root): python data\evaluate.py Optional: python data\evaluate.py --policy data\policy2.txt --truth data\ground_truth2.json Known limitations: - Groq free tier allows 200k tokens/day per organization; one full analysis costs about 80-100k tokens, so a second run on the same key may hit the daily limit (the app then stops with a clear message). Use /demo in that case. - A full run takes several minutes because of per-minute rate limits and automatic retries. - LLM output varies between runs; evaluation numbers are ranges, not exact. - Exposure (AZN) figures are illustrative estimates from scenarios, not real loss statistics. - Evaluation policies are synthetic.

Project ownership

The team keeps the rights to this project. Judges were given access for verification only.