Benchmarks

    How we benchmark the FYEO Scanner

    We seed known vulnerabilities into real Web3 codebases, run the FYEO Scanner and other systems blind against the same commit, and score detections by CWE weakness class. Below is every number from the current runs — including the bugs we missed.

    Seeded-bug recall

    The only metric required for every comparison on this page.

    Methodology

    Seeded ground truth

    We plant a known set of vulnerabilities into a real codebase. Every bug has a documented file, function, and CWE class, so a detection can be scored objectively instead of judged by eye.

    Blind, identical runs

    Each system gets the same repository, the same commit, and no hint that bugs were seeded. No prompt tuning per fixture, no retries, no cherry-picking the best run.

    Dedup by weakness class

    The (file, function, CWE) triple is the primary key. Dozens of report lines pointing at the same root cause count once — which is why raw alert counts are far higher than the detected counts below.

    Recall, not noise

    Recall = true positives / seeded bugs. A system that reports everything does not win: unmatched alerts earn no credit, so volume alone cannot inflate a score. Noise = raw alerts minus alerts that matched a seeded bug.

    FYEO vs Codex

    Private Fixture #1

    Rust · 14 planted bugs total · FYEO found 10, Codex found 6

    5FYEO ONLY5BOTH1CODEX ONLY

    59 FYEO raw alerts vs 20 Codex raw alerts before deduplication. Noise = alerts that did not match a seeded bug.

    Only FYEO

    HIGH
    CWE-294, CWE-345
    gift_recipient_unbound
    MEDI
    CWE-841
    admin_midsale_mutation
    MEDI
    CWE-20
    drop_create_missing_validation
    MEDI
    CWE-684
    special_bonus_not_credited
    MEDI
    CWE-770
    vec_max_len_overflow

    Only Codex

    LOW
    CWE-684, CWE-20
    admin_set_winners_ignores_pending

    Both

    CRIT
    CWE-294, CWE-345
    cross_type_sig_replay
    CRIT
    CWE-347
    ed25519_header_bypass
    HIGH
    CWE-770
    sig_receipt_griefing
    MEDI
    CWE-367, CWE-840
    gift_live_read
    LOW
    CWE-330
    select_winner_randomness_gaps
    MetricFYEOCodex
    Seeded bugs found10 / 146 / 14
    Raw alerts generated5920
    Model, token count & runtime disclosedYesNo

    Detector stage only. Grouped by weakness class — (file, function, CWE) is the triple the scorer matches on and the primary key dedup uses.

    Scope and limits

    • Current runs: 2 comparisons, 39 seeded bugs across Solidity and Rust codebases. More languages and fixtures are in progress.
    • Detector stage only. The Scanner's triage stage and any human auditor review are excluded, so these numbers understate what a customer receives.
    • Seeded bugs are a proxy for real ones. Recall here does not guarantee recall on novel, protocol-specific business-logic flaws — that is what our Web3 security audits exist for.
    • Fixture source is withheld where the underlying code is not public.

    Run it against your own code

    The fastest way to evaluate the Scanner is on a repository you already know the weak spots in.

    Start free trialTalk to our team