How We Trained Decision-1

Illustration of records entering a choice-scoring system, with a separate route to review — Decision-1 scores the choices you provide; your application handles review and action.

The idea in brief

Many workflows already know which answers are allowed. Given text and a question, Decision-1 scores 2–16 supplied choices; the application decides when a person must review the result.

Teams with defined choices and test cases can use the public weights and code to run local comparisons. The released results do not establish an operational benefit.

Two tasks to evaluate

Government cybersecurity

In a fictional agency packet, inventory lists asset A-17; a September 9 scan flags a vulnerability, and a September 10 patch/change ticket is approved. No install record or confirming rescan exists. Ask: “What do these records establish about remediation?” Offer remediation verified (install record and clean rescan), current exposure observed (positive scan after the change), remediation unverified (approval or plan only), and records conflict / analyst review. Compare with analyst-adjudicated cases; a human analyst controls status and action.

Government data labeling

For a fictional acquisition excerpt, offer defined labels: deliverable for a required work product, security requirement for a required control or practice, period of performance for contract dates or duration, and needs review for ambiguous text. Compare choices with human-adjudicated held-out labels.

These are proposed tests, not evidence of performance in a government workflow.

Training and evaluation, in brief

Worthify continued training all 11.9 billion text parameters of pinned Gemma 4 12B on 35,210 task rows. Cross-entropy trains the model to score the offered answer slot; shuffled option order reduces position shortcuts. The mixture spans routing, evidence, procedures, uncertainty, cyber scenarios, and option menus so one model can be tested on more than one bounded decision pattern.

The completed seed used two H200 GPUs with FSDP2 sharding for two epochs and 4,402 optimizer updates. Worthify selected step 3,200 on 3,200 validation rows, then scored a separate sealed 3,200-row test set. Validation chooses the checkpoint without test feedback; the sealed rows provide an independent check within these task families. A second training seed remains unfinished, so this is a single-seed preview. The pinned model card records the optimizer, precision, data provenance, and limits.

Training / checkpoint pathValidation chose the checkpoint. Sealed test rows checked it afterward.

01 / STARTGemma 4 12BPinned text checkpoint plus 35,210 rows of routing, evidence, procedure, uncertainty, cyber, and choice tasks.
02 / TRAIN4,402 updatesAll 11.9 billion text parameters continued training across two epochs.
03 / SELECT3,200 validation rowsValidation selected the step-3,200 checkpoint without test feedback.
04 / CHECK3,200 sealed rowsAfter selection, the checkpoint scored seven held-out task families.

One seed scored higher in seven test families

The sealed evaluation spans three families based on transformed routing or evidence data and four built from authored or supplied-option fixtures. Decision-1 scored above frozen Gemma 4 12B on each family's primary metric.

Decision-1 reached 67.29% routing macro-F1 versus 62.21% for frozen Gemma. On WANLI-derived evidence, macro-F1 was 74.41% versus 67.24%. An earlier task LoRA remained ahead on WANLI and MNLI evidence point estimates. These bounded tests do not establish operational performance.

The pinned model card has the full evaluation and limitations.

Seven held-out family primary scores comparing frozen Gemma, earlier task LoRA, and the Decision-1 full-weight seed-42 preview. Decision-1 exceeds frozen Gemma on all seven; the task LoRA leads on two evidence families. — The full-weight preview scored higher than frozen Gemma on all seven primary family metrics; an earlier task LoRA led on two evidence families. All models used the same BF16 scoring path, in batches of four, on one A100. Metrics differ across families; compare models within each family. Open full-size chart

Held-out scores · compare within each family

Intent routing · macro-F1

Frozen Gemma: 62.21%
Earlier LoRA: 66.38%
Decision-1: 67.29%

WANLI evidence · macro-F1

Frozen Gemma: 67.24%
Earlier LoRA: 77.01%
Decision-1: 74.41%

MNLI evidence · macro-F1

Frozen Gemma: 76.61%
Earlier LoRA: 90.66%
Decision-1: 89.61%

Authored cyber · accuracy

Frozen Gemma: 83.98%
Earlier LoRA: 98.05%
Decision-1: 100.00%

Supplied menus · accuracy

Frozen Gemma: 99.32%
Earlier LoRA: 99.32%
Decision-1: 100.00%

Authored procedures · accuracy

Frozen Gemma: 91.02%
Earlier LoRA: 99.22%
Decision-1: 100.00%

Authored uncertainty · accuracy

Frozen Gemma: 85.55%
Earlier LoRA: 100.00%
Decision-1: 100.00%

Source: pinned model card. Metrics differ by task family.

Try one decision without tuning

You can score a new question with the published checkpoint as it is. “Zero-shot” here means no task adapter and no labeled examples in the request. Decision-1 has already been trained on a mixture of decision tasks, so test it on held-out cases from your own workflow.

Inference / request schemaYour option descriptions carry the meaning; the scorer returns your IDs.

Request
state
Evidence as text or JSON
question
The criterion to decide
options[]
2–16 IDs and descriptions
↓→
Decision-1
One forward pass
Descriptions map to internal A–P token slots. The scorer reads those logits without generating an answer sentence.
↓→
Result
option_ids
Your original IDs, in order
option_logits
Scores for the offered slots
probabilities
Normalized across this menu

Your application maps the scores to a route or a human-review rule. A missing option cannot win, and the normalized values are not calibrated confidence.

This fictional government acquisition excerpt uses the four labels above, including a human-review choice. The input does not assert which label the model will choose. Download the exact JSONL row used below.

01 / decisions.jsonl · shown with line breaks for reading

{
  "id": "fictional-solicitation-1",
  "state": "Fictional agency solicitation: The contractor shall submit an asset inventory report to the contracting officer.",
  "question": "Which label best describes this excerpt?",
  "options": [
    {
      "id": "deliverable",
      "description": "A work product the contractor must provide."
    },
    {
      "id": "security_requirement",
      "description": "A required security control or practice."
    },
    {
      "id": "period_of_performance",
      "description": "The contract performance dates or duration."
    },
    {
      "id": "review",
      "description": "Needs human review because the excerpt is ambiguous or outside these labels."
    }
  ]
}

Clone the pinned public scorer, install its dependencies, and run direct scoring on one visible CUDA GPU. The package calls its dependency group train; this command loads no adapter and updates no weights.

02 / score the request · shell

git clone https://github.com/Worthify-AI/worthify-decision-1.git
cd worthify-decision-1
git checkout c66740244446c9cb0adce88a37efb8e98913ed71
python -m venv .venv
.venv/bin/python -m pip install -e '.[train]'
curl -fsSL https://worthify.ai/examples/decision-1-zero-shot.jsonl -o decisions.jsonl

CUDA_VISIBLE_DEVICES=0 .venv/bin/worthify-lora score \
  --model Worthify/Decision-1 \
  --revision cb54dcd238da782a21035bb39a1b057d743af832 \
  --quantization nf4 --max-tokens 2048 \
  --input decisions.jsonl --output scores.jsonl

The scorer writes one JSON result per input row. Read the top option ID and the scores attached to every original ID:

03 / inspect the result · shell

.venv/bin/python - <<'PY'
import json

with open("scores.jsonl") as file:
    result = json.loads(file.readline())
best = max(range(len(result["option_ids"])),
           key=lambda i: result["option_logits"][i])
print("Selected ID:", result["option_ids"][best])
print("Conditional scores:", dict(zip(
    result["option_ids"], result["probabilities"])))
PY

Use a new output filename for each run; the scorer will not overwrite an existing file. NF4 is the documented local path; the published held-out comparison used BF16, so this example is a starting point for your own evaluation, not an exact benchmark replay. Inspect the pinned source and model card for the runtime contract.

A better starting point for adapters remains unproven

The matched BANKING77 and CTU-13 studies did not establish Decision-1 as a better LoRA starting point than original Gemma. With a 388-update budget, the selected BANKING77 adapters scored 93.54% for original Gemma and 93.21% for Decision-1 on the same 3,080 test utterances, presented as 16-option menus. The point estimate favored original Gemma by 0.32 percentage points.

BANKING77 / 16 offered choices

Task training improved both models

These deterministic 16-option menus are not standard 77-way BANKING77 classification. “Zero-shot” means no BANKING77 adapter or task examples in the prompt; “Task LoRA” uses the validation-selected adapter.

Held-out BANKING77 accuracy on 3,080 test utterances before and after task LoRA training
Starting model	Zero-shot	Task LoRA	Gain
Decision-1	87.60%	93.21%	+5.62 points
Original Gemma 4 12B	87.89%	93.54%	+5.65 points

Decision-1

Zero-shot87.60%

Task LoRA93.21%

+5.62 points after LoRA

Original Gemma 4 12B

Zero-shot87.89%

Task LoRA93.54%

+5.65 points after LoRA

Both models improved after task LoRA; this does not establish a Decision-1 advantage. The 16-option menus are not standard 77-way classification. Decision-1 had prior related routing training, and 28 test rows were flagged for lexical overlap. This comparison does not establish transfer to a new domain. Inspect the score report.

The matched rank-16 adapter pairs used the same task rows, prompts, update budgets, and initial adapter tensors; checkpoints were selected on validation. An earlier 194-update BANKING77 snapshot favored Decision-1, but the exploratory 388-update study reversed that result. Its selected-pair descriptive 95% interval for the difference spanned zero. On sealed CTU-13 network-flow labels, original Gemma scored 0.432 macro-F1 versus 0.115 for Decision-1. Neither study supports a general task-adaptation or faster-convergence claim. See the full method and results and aggregate report.

Evaluate one task

Choose a routing or evidence decision your team is authorized to evaluate. Define the question, permitted answers, and cases that need human review. Reserve held-out cases before tuning. Use the LoRA training recipe to train matched adapters, select checkpoints on validation, then compare them and the original models on the reserved cases. Record wrong routes and review cases alongside the scores. Check the model card for the pinned revision, results, and limits.

Use it with these limits: The highest-scoring choice can still be wrong, and an omitted choice cannot be selected. Scores compare the supplied options; they are not calibrated probabilities that a claim is true. The application enforces human review and any action boundary.