Evaluation protocol
Status: evaluation protocol. Worthify has not published a verified ReAgent lift result. Any future result must keep model, tool-system, and human-workflow comparisons separate and identify the tested task, budget, environment, and claim gate.
Buyer questions
Cybench contributes reverse-engineering CTF tasks. CyberSecEval contributes malware-analysis tasks. CrackMeBench contributes agentic binary tasks with executable validation. SRE-Bench defines a contamination-resistant software reverse-engineering benchmark; task availability must be confirmed before use. Worthify will report each suite and selected task subset separately because the tasks and scoring differ.
No. The defensible comparison asks whether the same model completes the same held-out tasks more successfully when ReAgent is available. ReAgent changes the tool system around the model; it does not change the model weights.
A future human study would compare each analyst with and without ReAgent using counterbalanced, difficulty-matched tasks with no task repeated across conditions. Blinded reviewers would measure time to a validated finding, false leads, corrections, and review work.
Worthify has not published a broad, statistically verified ReAgent accuracy-lift claim. Any future result must name the benchmark, models, task count, baseline, budget, effect size, uncertainty, failures, and cost.
Positioning
External suites provide a reference point. Held-out reverse-engineering tasks reduce the chance that a model recalls a public answer. A result names the exact suite version and selected task subset.
Before publication, the comparison must freeze the model and tool versions, prompt objective, binary, scoring oracle, time limit, spend ceiling, repeat policy, and exclusion rules. The treatment changes ReAgent access and records actual treatment use.
Before publication, the result must retain the full resource envelope and every attempt. More completed tasks do not establish a better system when the treatment consumes far more time, context, or compute.
The protocol uses exact-oracle pass rate as the primary LLM endpoint. The pilot sets the powered task count. Paired analysis reports effect size and confidence intervals, clusters related questions by binary, corrects multiple comparisons, and retains every exclusion.
A future human study would use a counterbalanced within-subject design, matched tasks, and no repeated task across conditions. The publication record would include cohort size, experience, training, reviewer blinding, validation rubric, and inter-rater agreement.
Research references
Related Worthify pages
