AI Enterprise SolutionsFinalistGPT & Claude

RAM Team

Managers struggle to evaluate developers fairly. Lines of code and commit counts don’t show the real quality or complexity of their work. Our tool analyzes Pull Requests and provides clear insights to make performance reviews more fair and transparent.

Open the demo Video GitHub

Preliminary score

79.5/ 100Strongaverage of 2 judges

Value for the user19.0 / 25
A specific problem and a clear outcome from solving it.
Prototype and use of AI25.0 / 30
A working core scenario, and what AI actually contributes to the result.
Quality testing17.5 / 20
Tests, examples of failures, and comparison with the current approach.
Feasibility11.5 / 15
Data requirements, running costs and a clear next step.
Originality6.5 / 10
A meaningful difference from common solutions.

A finalist: the live jury scored it 70.0 / 100 on stage.

What the judges wrote

Claude Opus 5.5Preliminary80/100
Strengths
Your prototype runs on real data: 150 merged pydantic PRs, a live static dashboard, and an end-to-end pipeline from the GitHub API to the scores. The model is essential here, giving four rubric scores with reasons and verified code-line references, and the author is hidden from it. Your quality testing is exemplary for a hackathon. It includes a blind 20-PR human-vs-model comparison broken down by criterion, risk AUC against PR size with confidence intervals, 186 offline tests, and candid failure reports such as the top maintainer getting the lowest multiplier because of the config-based senior norm.
To improve
Your adoption metric is still mostly speed (150 PRs in 15 minutes). Show evidence that managers' ratings become more consistent or fairer with the tool, for example by measuring inter-manager disagreement with and without it. Run-to-run stability is untested even though your README describes median-of-three scoring, so run it and report the variance. The 2-of-103 revert sample is too small, so validate risk on a repository with more reverts. Also derive level norms from data rather than config, and spell out the monthly cost per team plus a concrete pilot step.
GPT-6 AstraPreliminary79/100
Strengths
Your submission documents a concrete GitHub-to-dashboard pipeline over 150 real PRs, with author masking and code-linked explanations rather than unexplained scores. Your blind human comparison, PR-size baseline, and candid reporting of clarity bias and inconclusive risk prediction make the evaluation particularly informative. Your cached results and offline dashboard provide a practical way to inspect the prototype without API access.
To improve
Your next evaluation should re-fetch leakage-free data and repeat scoring across multiple repositories, using multiple blinded human raters. Your level norms and multiplier need calibration: the maintainer result highlights why explainable PR scores do not yet establish fair employee evaluations. Your pilot should account for non-PR contributions and developer appeals before informing compensation decisions. Measure manager review time and dispute rates against today's process, and report actual per-PR costs and subscription throughput limits.

The team

Mubariz Huseynzade
developer
Ali Alekberov
product
Said Kerimli
data
Michael James Jones
developer
Rasul HuseynzadeLead
developer

Disclosure

Required by the rules. Judges verify it against the repository.

Models: Claude Haiku 5.5 via Claude Code CLI; coding assisted by Claude
Data: Public GitHub data (pydantic/pydantic, 150 PRs) + 20 human-scored PRs
Components: Python, requests, matplotlib, GitHub API, vanilla JS, GitHub Pages
Built during the event: Confirmed

Quality testing

Automated: 186 tests (offline stubs for GitHub and the LLM), data consistency check on all files (0 errors). On 150 real PRs: 0 failed runs, 0 rejected answers, 0 hallucinated file references, author never visible to the model. Model vs. human (20 PRs, scored blind): 79% within 1 point, 35% exact. Complexity and risk 95%, quality 85%, clarity only 40% — the model is 1.6 points stricter: it penalizes empty PR description templates, which we graded leniently. Risk vs. reality: inconclusive. Only 2 of 103 PRs were reverted or urgently fixed, too few to prove anything. Risk score separates them better than PR size (AUC 0.68 vs 0.42), but the interval is wide (0.47–0.86). What broke: - Free Gemini tier was slow and unreliable; Claude API had no credits. We switched to Claude Code CLI on a subscription. - First real outputs had long English summaries; fixed via schema limits and prompt. - The top maintainer (98 PRs) got the lowest multiplier: the senior norm comes from config, not data (only 3 seniors). - Not tested: run-to-run stability (1 run per PR). Some reviews written after merge may remain in the data (fixed in the fetcher, not re-fetched). Today: managers count PRs and lines or judge subjectively. Our tool scored 150 PRs in 15 minutes, every score with a reason and code lines.

Worth 20 of the 100 points.

Running it

No installation needed. 1. Open the live dashboard: https://sataan1c.github.io/pr-evaluator/ Works in any modern browser, on any laptop or phone. All data is built into the page. 2. Offline option: download docs/index.html from the repository and open it with a double click. No internet, Python or accounts required. 3. Suggested 2-minute walkthrough: - Team: developers with their multiplier relative to the norm for their level. - Click a developer: how the multiplier was calculated, score profile, their PRs. - Click a PR: four scores (complexity, quality, risk, clarity), the reason for each, and the highlighted code lines the model relied on. "Open on GitHub" links to the real PR. - Method check: model vs. human scores (79% within 1 point on 20 PRs), risk score vs. reverted/urgently fixed PRs. Language switch (AZ / RU / EN) is in the top right corner. 4. Data: 150 most recent merged PRs of pydantic/pydantic (real data, not mock), scored by Claude (claude-haiku-5-5). The author is hidden from the model. 5. To run the full pipeline locally (optional, Python 3.9+): git clone https://github.com/sataan1c/pr-evaluator.git cd pr-evaluator pip install -r requirements.txt python app.py This opens the same dashboard at http://127.0.0.1:8765 using the committed data; no API keys are needed. Re-scoring PRs requires a Claude Pro/Max login (see README).

Project ownership

The team keeps the rights to this project. Judges were given access for verification only.