What 1 Million Scans Taught Me About AI-Generated Code

Eight months ago, I launched aislop because AI helped me write code faster than I could confidently review it. I did not expect the tool to travel this far, or to teach me this much about what code quality means when machines can produce code all day.

On 11 March, I wrote about why I built aislop. At the time, it was a small open-source CLI built around a problem I kept seeing in my own projects: the code worked, but the structure got worse with every fast AI-assisted change.

I thought a few developers might try it. I hoped some would tell me where it was wrong. I did not expect people to run it at this scale, send detailed issues, or contribute support for languages I was not yet using.

Eight months later, aislop feels less like a tool I made for myself and more like an ongoing investigation into AI code quality.

The numbers surprised me

As of 15 November 2026, telemetry had recorded 1,104,722 completed scan and CI commands across all available history. The npm registry recorded 273,148 all-time downloads through 14 November, covering the package’s history since its first publication on 11 March, including 98,084 in the latest 30 complete days. The Python package had 43,017 all-time PyPI downloads according to pepy’s total-download badge checked on 15 November. When I checked that day, the GitHub repository had 745 stars.

There were also 11,981 distinct anonymous installation IDs on cli_installed events explicitly marked as non-CI between 11 March and 14 November. That is not a count of people: reinstalls and different machines can produce separate identities. npm and PyPI downloads can include CI and repeat installs, and one installation can run many scans. These are separate signals of use, not numbers to add together.

That still feels strange to write. A scan is not a vanity metric when someone has put it inside a coding hook. It means the tool gets a chance to interrupt a bad pattern before that pattern becomes another reviewer’s problem.

Building in public changed the project

The best part has been seeing other people recognise the problem and improve the answer. As of 15 November, eight contributor accounts beyond mine have had 31 pull requests merged, excluding bot accounts.

Yash Rajoria made single-issue scoring more proportional. Michael Awoniran added configuration-based exclusions, then followed it with include patterns. Matt Schoen has contributed across accuracy, performance, Windows support, and new language coverage, including C# and .NET and C and C++. More recently, @notlakshitha added scan-scope explanations and opt-in project checks with rollback. Other contributors have fixed CLI behaviour and extended integrations.

These were not cosmetic commits. They came from people running the tool in environments I had not covered, finding assumptions I had made, and leaving the project better than they found it.

People have also reached out privately. Some messages are about false positives. Some are about adding aislop to a team. Some are simply: “I have been seeing this too.” That last one matters. It tells me the problem was never only inside my own repositories.

I understand the problem differently now

When I started, I described aislop as a tool for catching AI slop. I still like the phrase because developers know the feeling immediately. But the deeper problem is not proving that AI wrote a line of code.

aislop cannot reliably tell you who or what authored your code, and that is not the useful question anyway. The useful question is whether the code is safe to inherit.

Can somebody else review it without reconstructing a 200-line function in their head? Does a fallback hide a real failure? Did a type cast silence information the compiler needed? Has the same block appeared in four files? Is the code full of comments that explain the syntax while missing the decision?

These problems existed before coding agents. AI changes their frequency. A model can produce a plausible implementation in seconds, then produce another one before a human has understood the first. Small structural mistakes can now arrive at machine speed.

That has made me much more careful about precision. A noisy quality tool becomes another thing developers learn to ignore. Every false positive spends trust. Every vague warning sends the reviewer back into work the tool should have already done. A good rule needs a clear reason, a useful location, and a realistic fix.

The market is moving from generation to verification

AI coding is no longer a niche workflow. GitHub’s 2025 Octoverse found more than 1.1 million public repositories using an LLM SDK. DORA’s March 2026 summary of its 2025 research reported that 90% of technology professionals use AI at work, while also describing AI as an amplifier of both strong engineering systems and weak ones.

The tension is visible in developer sentiment. In the 2025 Stack Overflow Developer Survey, 66% of respondents said their biggest frustration was AI output that was almost right, but not quite. Forty-five per cent said debugging AI-generated code took more time. More developers distrusted AI accuracy than trusted it.

Even the productivity evidence resists a simple headline. An early 2025 METR study found experienced open-source developers were 19% slower with AI on familiar repositories. In a 2026 update, METR said its later experiment could not produce a reliable estimate because of selection bias, while noting that AI tools had probably become more useful.

That is closer to what I see in practice. AI can make the first draft dramatically faster. It can also move the bottleneck into review, testing, integration, and maintenance. The market has spent years improving generation. The next important layer is confidence.

aislop has had to grow up

The latest release I checked, v0.22.1, supports 12 language targets, including Scala in beta, C#, and C/C++. Its release notes report 2,626 regression tests and nine real-tool checks. The project has Linux and Windows CI, coding-agent hooks, structured output, and a much larger set of accuracy checks than the version I wrote about in March.

The biggest work is often invisible. It is adjusting severity so one issue does not destroy a score. It is recognising workspace layouts before calling an import fake. It is making a detector useful on Windows. It is proving that a new rule catches a real pattern without punishing valid code.

I have also learned to dogfood more seriously. aislop scans aislop. New rules meet the repository that created them. If a rule is too noisy to live with there, I should not ask somebody else to live with it.

What I am building towards

I want aislop to become a dependable quality layer for people and coding agents. Not a replacement for tests. Not a replacement for code review. A fast, deterministic check that catches the maintainability and safety problems most likely to be multiplied by AI-assisted development.

That means going deeper on accuracy, using real scan patterns to understand where rules fail, and making the feedback useful at the moment code is written. The goal is not to stop developers using AI. I use it every day. The goal is to keep the speed without handing tomorrow’s developer a codebase nobody wants to touch.

Eight months in, the thing I believe most strongly is this: the market does not need more code. It needs more confidence in the code already being produced.

aislop is MIT licensed. You can run npx aislop@latest scan, read the source on GitHub, and tell me where it gets your codebase wrong. That feedback is how the project got this far.

What these figures measure

The all-time scan total was queried on 15 November 2026 UTC and includes that day’s activity available at the time of the query. Installation and npm figures end on 14 November, the latest complete day. The PyPI total is pepy’s all-time snapshot retrieved on 15 November; its public badge can be cached for up to 12 hours. GitHub stars, merged contributor PRs and the latest release were checked on 15 November. The scan total counts cli_command_completed events where command is scan or ci, with the date range set to all time. The first matching event in the available history is 16 May 2026. CI and internal activity are included. Separate hook-scan events and legacy scan events are not added to this figure, avoiding an unverified combined total. This is a count of completed command events, not unique projects, users, or a claim that every command passed its quality gate. Telemetry-disabled runs are not observed. The installation figure counts distinct anonymous_install_id values on non-CI installation events. These are direct event aggregates, not an audited user count.

The previous draft’s repository-connection figure is omitted because I could not verify a comparable current event definition. Research figures above remain labelled with their original study years; they are context, not measurements of aislop users.