Published October 15, 2026 8 min read By Jason Shotwell

We Ran Four Open-Source EU AI Act Scanners on the Same Code and Published Every Verdict

Open-source EU AI Act scanners publish lists of what they check. None of the three we compared here publishes how often those checks are right. We already measure our own scanner, so we ran AIR Blackbox, ArkForge, Systima Comply and Asqav on the same 144 hand-labelled programs and put every single verdict in the repository, including the results that do not flatter us.

Key Takeaways

In Python, TypeScript and JavaScript the tools are close. On the three checks it shares with the corpus, Systima matched AIR on every sample in the languages it reads.
The real gap is reach. Of 145 Java and Kotlin programs that call an AI SDK, AIR recognised all 145 as AI code. The best of the other three recognised 7.
The second gap is breadth. AIR checks 38 code practices in this corpus. The others have three, three and five code-level checks that map onto it.
The others check things AIR does not, such as prohibited-practice indicators and whether compliance documents exist. Those are listed, and nobody is scored on them.
We wrote the test, and AIR was developed against it, so AIR's numbers are an upper bound. The method, the mapping and every verdict are public so anyone can check them.

Why accuracy, and why publish it

A compliance scanner makes a claim about your code: this practice is there, that one is missing. If the claim is wrong, you either fix something that was fine or ship something that was not. A list of check names tells you nothing about which of those will happen.

We already measure our own scanner against a labelled corpus. The obvious next question is how the other open-source tools do on the same programs. The answer turned out to be more interesting than a leaderboard, because the tools are not really trying to do the same job.

How the test works

The programs. The parity corpus holds 144 small programs covering 38 practices, such as error handling around model calls, logging, human approval before risky actions, and GDPR consent. Each program either has the practice or deliberately lacks it, and each is written in Python, TypeScript, JavaScript, Java and Kotlin. Each Python program is also run as a Jupyter notebook.

The tools. AIR Blackbox 1.17.0, ArkForge eu-ai-act-scanner 2.0.39, Systima Comply 0.3.1 and the Asqav compliance scanner 1.0.1. Each was installed from its own registry or repository and run through its own CLI or exported functions.

Fair scoring. Each competitor is scored only on the practices it says it checks, and AIR is scored on exactly the same samples. We read each tool's source and documentation to map its checks to the corpus; the mapping is published with its sources. Only programs that import an AI SDK are scored, because some Python cases are short fragments and a tool scoped to AI code is right to skip them. A scored program where a tool gives no verdict counts as a miss, because a user running that tool on that program learns nothing.

Results

Practice detection

On every scored program, across all six languages and forms:

Tool	Practices	Programs	Tool correct	AIR correct
ArkForge	3	24	17 (71%)	24 (100%)
Systima Comply	3	26	19 (73%)	26 (100%)
Asqav	5	71	7 (10%)	71 (100%)

That table mostly measures which languages each tool reads. Here is the same comparison restricted to the languages each tool says it supports:

Tool	Languages it reads	Programs	Tool correct	AIR correct
ArkForge	Python, TypeScript, JavaScript, Java	16	13 (81%)	16 (100%)
Systima Comply	Python, TypeScript, JavaScript	12	12 (100%)	12 (100%)
Asqav	Python	12	7 (58%)	12 (100%)

Inside its own languages, Systima ties AIR. ArkForge's misses there all come from its content-marking check, which reads only .py files. Two of Asqav's misses come from one pattern: it needs a space or a dot after langchain, so from langchain_openai import ChatOpenAI, the usual LangChain import, is skipped.

Does the tool notice the AI code at all?

Before any check can run, a scanner has to recognise that a program calls an AI model. Ground truth here is nothing but the program's import lines.

Tool	Python	Notebook	TypeScript	JavaScript	Java	Kotlin
AIR Blackbox	52/53	52/53	53/54	52/53	78/78	67/67
ArkForge	49/53	34/53	53/54	52/53	7/78	0/67
Systima Comply	52/53	52/53	52/54	51/53	0/78	0/67
Asqav	48/53	0/53	0/54	0/53	0/78	0/67

AIR's three misses are real gaps in its framework list: a Python program using only LangGraph, and the TypeScript and JavaScript versions of a program using @openai/guardrails. Both are small additions to that list. The published numbers stay pinned to 1.17.0, misses included.

What the other tools do that AIR does not

This benchmark only sees checks that read code. Each of the other tools also does something AIR does not, and none of it is scored here, for or against anyone:

ArkForge checks whether compliance documents such as HUMAN_OVERSIGHT.md and RISK_MANAGEMENT.md exist and scores how complete they are. That is its main EU AI Act check.
Systima Comply looks for Article 5 prohibited-practice indicators, validates a declared risk classification, and checks declared documentation files. It also posts findings on pull requests.
Asqav checks for revocation and kill-switch patterns.

If those are what you need, the code-level numbers above are not the whole story.

What the numbers do not show

We wrote the corpus. The translations were written without access to the scanner's rules, but AIR was improved against these cases before release. Its scores here are an upper bound. Across the whole corpus it gets 96% right, and its six misses are listed in the eval report. The labels were written before this comparison and were not changed for it.
The overlap is small. Between 24 and 71 scored programs per competitor. Treat the percentages as indicative, not precise.
The mapping is a judgement. If one of the other tools' checks means something different from what we mapped it to, the fix is one line, and we will rerun.
Dependency files are synthesised. Python and JavaScript samples get a requirements.txt or package.json built from their imports, as a real project would have. Some of the other tools rely on those files. Java and Kotlin samples get none.

Where the signing keys live

Scanning is half of AIR. The other half is a signed, tamper-evident record of what an agent did, and among these tools the closest alternative is Asqav's SDK, which also offers post-quantum ML-DSA-65 signatures. The difference is where the signing happens.

Asqav's README says "Cryptography runs server-side", and that its receipts use "ML-DSA-65 for cloud-issued receipts, Ed25519/ES256 for locally signed ones". Its client SDK is source-available under the Elastic License 2.0, which is not an OSI open-source license, and its signing service is commercial. To its credit, its verifier is Apache-2.0 and anyone can verify a receipt without an account.

AIR signs on your machine. The keys are generated in your own process and never leave it: ML-DSA-65 when the pqc extra is installed, Ed25519 otherwise. There is no account, no API key and no hosted signing service, and the whole project is Apache-2.0. Scanning never sends your code anywhere. The CLI does send an anonymous usage count for some commands, never code or file names, and AIR_BLACKBOX_TELEMETRY=off turns it off.

Check it yourself

The harness, the mapping, the caveats and every verdict from every tool are in eval/competitors. The run takes about three minutes:

git clone https://github.com/airblackbox/airblackbox
cd airblackbox
cat eval/competitors/README.md   # installs all four tools, then runs:
python3 eval/competitors/run_h2h.py --json results.json

If we got one of the other tools wrong, open an issue or a pull request. That goes double for the tools' own maintainers. A benchmark you cannot argue with is not worth much.