Sql InjectionInjectionTester
InjectionSQL_INJECTION_TEMPLATE_LITERAL Detected · control passed
A security benchmark you can reproduce.
This checked-in corpus tests labeled security rules against paired vulnerable and safe controls. The result is deterministic, public, and enforced in CI.
Ship Safe CLI v9.5.2 · Corpus v1.0.0 · First-party evaluation
12/12labeled scenarios detected
100%scenario recall on this corpus
100%target-rule clean-control pass rate
From command to evidence
The same local scanner, tested in CI.
The visual demo shows Ship Safe running against a repository. The benchmark itself runs without animation, accounts, or external AI calls so every assertion remains deterministic.
Scenario results
Conventional and AI-native risks.
Every row maps a labeled fixture to one expected rule and a paired safe control.
Jwt NoneAuthBypassAgent
AuthenticationJWT_ALG_NONE Detected · control passed
User Controlled SsrfSSRFProber
SSRFSSRF_USER_URL_FETCH Detected · control passed
Api Mass AssignmentAPIFuzzer
APIAPI_SPREAD_BODY Detected · control passed
Llm Output ExecutionLLMRedTeam
LLMLLM_OUTPUT_TO_EVAL Detected · control passed
Mcp Shell ToolMCPSecurityAgent
MCPMCP_TOOL_SHELL_EXEC Detected · control passed
Agent Auto ApprovalAgenticSecurityAgent
Agentic AIAGENT_TOOL_NO_CONFIRMATION Detected · control passed
Rag User UploadRAGSecurityAgent
RAGRAG_USER_UPLOAD_TO_VECTORDB Detected · control passed
Hermes Tool DispatchHermesSecurityAgent
HermesHERMES_FUNCTION_CALL_NO_ALLOWLIST Detected · control passed
Unsafe Model LoadModelScanAgent
Model supply chainMODEL_TORCH_LOAD_UNSAFE Detected · control passed
Install Script ExfiltrationInstallGuardAgent
Package supply chainWORM_LIFECYCLE_EXFIL Detected · control passed
Clickfix LureClickFixAgent
Social engineeringCLICKFIX_PASTE_RUN Detected · control passed
Methodology
Small, explicit, and regression-focused.
First-party paired synthetic scenarios; one vulnerable and one safe control per target rule.
- Run the named local agent against the vulnerable fixture.
- Require the labeled target rule to appear.
- Run the same agent against its safe control.
- Require the labeled target rule not to appear.
- Fail CI if either assertion regresses.
$ npm run benchmark:corpusNo inflated claims.
This corpus measures deterministic scenario detection, not prevalence or performance on arbitrary production repositories.
Target-rule clean-control pass rate is not a real-world precision estimate.
The evaluation is maintained by the Ship Safe project and is not independent validation.
Independent review, pinned third-party vulnerable repositories, and comparative scanner testing remain separate evidence tracks.
Benchmark FAQ
Read the result correctly.
What does the Ship Safe benchmark measure?
It measures whether a labeled target rule detects its vulnerable synthetic fixture and stays quiet for that same rule on a paired safe control.
Is this an independent security benchmark?
No. This is a transparent first-party regression corpus maintained by Ship Safe. It does not claim independent validation or production-repository precision.
How can I reproduce the benchmark?
Clone the Ship Safe repository and run npm run benchmark:corpus. The command exits with an error if a labeled detection or paired target-rule control regresses.