Developer Session Productivity Estimator

Capture real coding sessions, prompt devs to estimate counterfactual time-without-AI, and compute a calibrated rlog productivity metric.

Difficulty: 1-month | Stack: TypeScript, VS Code Extension API, Node.js, Postgres, Next.js, Anthropic SDK

Who this is for

Engineering managers and AI tool vendors who need financial-guarantee-grade evidence of AI productivity impact, not self-reported vibes

Build steps

1. Build a VS Code extension that logs session events: files touched, time-in-file, AI completions accepted/rejected, git diff at task close
2. On task completion (detected via git commit or manual tag), prompt developer: ‘How long would this have taken without AI? [<30min / 1-2h / half-day / 1+ day]’
3. Store sessions in Postgres with task metadata, language, complexity proxy (lines changed, cyclomatic delta)
4. Compute rlog (ratio of estimated manual time to actual AI-assisted time) per session; aggregate by user, language, task type
5. Build a Next.js dashboard showing rlog distribution, held-out validation set accuracy, and confidence intervals — expose as shareable report link
6. Add holdout validation: for a random 10% of tasks, ask a second developer to independently estimate time; use disagreement rate to calibrate confidence bounds on the guarantee

Risks

• Developer time estimates are notoriously biased — anchoring on actual time spent skews counterfactual high, inflating rlog; need careful survey framing
• Session data volume is small per user; need 50+ sessions per person for stable rlog estimates, requiring weeks of adoption before results are meaningful
• Privacy concerns around keystroke/file logging may block enterprise adoption entirely without a self-hosted deployment path