Engineering productivity, without the vanity metrics
Lines of AI-generated code and suggestion acceptance are not outcomes. Compare delivery and quality in a real repository, week by week, before and after AI coding tools arrived.
Repository and rollout
Assumptions & sources
Source: GitHub search API, merged pull requests only, fetched one week at a time. GitHub gives an exact count for each week and returns up to 100 of its PRs. When a week has more, its PRs are weighted by that week's exact count, so busy weeks count fully in the medians.
Definitions: time to merge = merged time − opened time. Reverts = merged PRs whose title starts with "Revert". Bots = accounts GitHub marks as bots, or names ending in [bot], or starting with dependabot, renovate or github-actions.
Limits: a before/after comparison shows correlation, not cause. Team size, release cycles, holidays and repository policy changes all move these numbers. Check the weekly trend for changes that started before the rollout, use enough weeks to smooth out noise, and add a control repository that did not adopt AI tools.
Signal: a permutation test on the weekly values. The weeks are shuffled between before and after thousands of times; if a gap as large as the real one appears less than 5% of the time, the change is marked clear, otherwise it is within normal week-to-week variation. A clear change is still not proof the AI tools caused it.
Control comparison: net of control is a difference-in-differences estimate: your repository's relative change minus the control's over the same weeks. Pick a control of similar size, language and release rhythm. It assumes both would have moved alike without AI tools; check the indexed chart to see whether they tracked each other before the rollout.
What to measure
The repository comparison covers the delivery and quality dimensions that public data supports. For the other two, the AI Adoption page accepts a usage export, and the developer experience survey turns an 8-question team survey into scores.
Questions
How do I measure the impact of AI coding tools like GitHub Copilot or Claude Code?
Compare delivery and quality outcomes before and after the rollout, not lines of AI-generated code or suggestion acceptance. Useful measures from GitHub are merged pull requests per week, median and 75th-percentile time to merge, the share merged within 24 hours and the revert rate. Use several weeks on each side, exclude bot PRs, and compare against a similar repository that did not adopt AI tools.
Why exclude bot pull requests?
Bots such as Dependabot and Renovate open many small PRs that merge quickly. They inflate throughput and shorten merge times, so a change in bot activity can look like a change in developer productivity.
What is a control repository and why use one?
A control is a similar repository that did not adopt AI tools, measured over the same weeks. Subtracting its change from yours (a difference-in-differences estimate) removes effects both saw, such as holidays, release cycles or ecosystem events, and leaves a fairer estimate of the AI effect.
Does a faster time to merge prove AI tools worked?
No. A before-and-after comparison shows correlation, not cause. Team changes, release schedules and review policy also move these numbers. Check the weekly trend for changes that began before the rollout, and pair delivery data with quality measures and a developer survey.
Is my repository data stored by this site?
No. Your browser queries GitHub's public search API directly, and nothing is sent to or stored by Learn AI Engineering. Private repositories are not supported.