Security judgment that concludes only when the evidence is enough: a generation-free 27B evidence-sufficiency model
In autonomous security response, the moment you conclude is the moment action begins. The instant a detection is concluded to be an active intrusion, isolation or account suspension follows. So the important question is not whether the model writes a good answer, but whether it may answer with the evidence it has right now.
This post introduces a model that judges whether security evidence is sufficient. It takes a claim, the required facts that must be established for that claim to hold, and the evidence collected so far. It decides whether the claim is supported, refuted, short of evidence, or contradicted; which input evidence supports and which refutes; which required facts remain unsatisfied; and whether a conclusion is allowed at all. The model does not write freely; it only fills in the verdict form. The model is named Thaki-SecDecision-27B-V2, and the four verdict states are SUPPORTED, REFUTED, INSUFFICIENT, and CONFLICTED. It is written for people building security agents, or deciding what to place where in an agent stack.
The model, inference code, and model card are public on Hugging Face. A smaller research preview with the same input format and decision rule, Thaki-SecDecision-4B-Preview, is public as well. Its verification status is much weaker, and it has its own section below.
The moment of checking each piece of evidence against the claim, before the verdict is rendered.
In plain terms
Think of a courtroom. The prosecution’s allegation is the claim, and the facts that must be proven for it to stand are the required facts. The submitted evidence is the evidence, and the judge must conclude from that evidence alone.
The judge has four possible rulings: accepting the allegation because the evidence is sufficient, the evidence overturning the allegation, the evidence still being short, and the evidence contradicting itself. These are SUPPORTED, REFUTED, INSUFFICIENT, and CONFLICTED. The judge does not write a new case report to invent facts; the judge looks at the submitted evidence and picks one of the four rulings. Likewise, this model generates no text.
The question a judge ponders longest is not the ruling itself but when a verdict can be handed down at all. Sometimes evidence never arrives, and sometimes one more check flips the verdict. The model has a separate output for this question. When evidence is short, it answers “not sufficient” instead of lowering the bar, and the side collecting evidence learns what to collect next. So in plain language, this model is a judge who treats “not enough yet” as one of its rulings.
Infographic generated by NotebookLM from the sources.
What this model does, and what it does not
In a security agent stack, this model is a gate. The agent receives a detection and collects evidence (telemetry, collection attestations, attack-reference documents) and asks this model whether it may conclude on this claim now.
The model judges four things. First, the claim status. Second, which evidence IDs from the input support the claim and which refute it. Third, which required facts remain unsatisfied. Fourth, whether a conclusion is allowed now (can_conclude). These four come out as a single line of fixed-schema JSON.
For example, for the claim “this host is exfiltrating data to an external IP”, the required facts decompose into (1) did an outbound connection actually occur, (2) is the destination an external IP, (3) does the data volume exceed the threshold. If the agent first collects telemetry and asks, it may get back that (1) and (3) are satisfied and only (2) is not. The next collection target then narrows to “verify the destination-IP attribute of that connection”.
Naming what the model does not do is equally important. It does not approve responses, execute detections, or make incident judgments. can_conclude being True is a sufficiency statement (“with this evidence, this claim can be concluded to be in this state”), not an action authorization. Keeping “No Evidence, No Action” only as a policy document is different from the model’s output schema itself enforcing that principle. This model is the latter.
Architecture: three heads over a frozen backbone
The model is a combination of a frozen Qwen3.8-27B backbone, a LoRA adapter, and three small fp32 heads. The backbone is pinned to a fixed revision and never changes, not even during training. LoRA is applied to the top 16 layers (48 to 63 of 64) only. The three heads each do a different job.
flowchart TB
IN["Input: claim + required facts + evidence set"] --> BB["Frozen Qwen3.8-27B backbone<br/>bf16 · SDPA · LoRA on top 16 layers"]
BB --> EV["Evidence span mean-pool"]
BB --> CL["Claim span mean + last-token hidden"]
EV --> HR["H_relation (fp32)<br/>SUPPORT / REFUTE / RELATED_ONLY / IRRELEVANT"]
EV --> HF["H_fact (fp32)<br/>required-fact × evidence satisfaction"]
HF --> HS["H_sufficiency (fp32)<br/>4 statuses + can_conclude"]
CL --> HS
HR --> RB["Decision Rule B (pre-registered)<br/>τ = 0.9995245433226797"]
HF --> RB
HS --> RB
RB --> OUT["Fixed-schema JSON<br/>status · evidence IDs · unsatisfied facts · can_conclude"]
The relation head decides the direction each piece of evidence takes relative to the claim: one of SUPPORT, REFUTE, RELATED_ONLY, IRRELEVANT. The fact head decides, for each pair of a required fact and a piece of evidence, whether that evidence satisfies that fact. The sufficiency head takes the claim status and the degree of fact satisfaction together and emits the four statuses plus can_conclude.
The absence of generation matters a lot for the output. A general LLM first writes a sentence that looks like a justification, then answers on the basis of that sentence. This model does not write the verdict narrative; it only fills in the verdict form. The failure mode that generative models show, fabricating a plausible-sounding reason that is not true, is structurally absent.
The input format is fixed too. Plain text with no chat template: the claim, then the required-facts block, then evidence blocks tagged with source_type, and finally a DECISION marker. Examples exceeding the 4096-token context are excluded, not truncated. A judgment made on a truncated input is not a judgment on the original input, so such examples are not put to the model.
Training ran one epoch (3,334 steps), and the checkpoint was selected on the train_dev and calib splits only. The selection criterion was “maximum ENOUGH recall among checkpoints whose False-ENOUGH rate stays at or below 2%”. The chosen checkpoint is step 2200. After selection, τ was fixed on the calibration split, and only then was the TEST opened.
The decision rule: Rule B demands a basis for refutation too
The heads’ probability distributions are not the output. On top of them sits a pre-registered decision rule, Rule B.
Support requires the status to be SUPPORTED, no unsatisfied required fact, and at least one piece of evidence in the SUPPORT direction. Refutation requires not only the status REFUTED but an observation-based basis. Concretely, at least one REFUTE-direction piece of evidence whose source_type is collection_attestation or telemetry counts as a refutation basis. In courtroom terms: even when declaring the defendant not guilty, the judge demands directly observed evidence, not a witness’s testimony. A non-observed document like “I read some reference” (attack_reference) cannot serve as a refutation basis.
The same principle applies to the absence of a record. “A command is not in the logs” does not mean “the command did not run.” It may simply not have been collected yet. Until the collection scope and completeness are verified, absence cannot be a refutation basis.
A conflict is judged when the status is CONFLICTED, or when there is at least one piece of evidence in each of the SUPPORT and REFUTE directions. can_conclude becomes True only after passing all these gates and the sufficiency score crossing the pre-set threshold τ. τ is the value chosen on the calibration split as “the lowest threshold whose False-ENOUGH rate is at or below 2%”, with maximum recall among those. Its full precision is in Table 1 below. It was chosen without seeing the results, so it is not changed after the results are seen. The moment you retune the threshold against a different split, the number is no longer trustworthy.
Three things were fixed before the TEST was opened: the decision rule, the threshold, and the rule that chose the threshold. Table 1 lists them.
Table 1. Fixed before the TEST was opened
| Item | Value |
|---|---|
| Decision rule | Rule B, the pre-registered amendment refutation_coverage_gate_correction (ledger seq 274) |
| τ selection rule | calib split only: the smallest τ with False-ENOUGH ≤ 0.02, maximizing recall |
| Sufficiency threshold τ | 0.9995245433226797 |
Rule B came out of earlier runs, and those runs are separate from the result above. Table 2 keeps them apart and names the run on each row. The V1 regression run used an earlier model (V1). Its macro F1 beat the stock model (0.627 → 0.731), but its false-conclusion rate rose too (1.79% → 3.19%), and it failed its pre-registered gates. The V2 calibration run under the pre-amendment rule was conservative in the refutation direction. Overall conclusion recall on the calibration split stayed around 0.48, and on the cases where the evidence overturns the claim (REFUTE_COVERAGE) it was 0.048. Requiring an observed refutation basis is the correction that became Rule B.
Table 2. Earlier runs that motivated Rule B (not the sealed TEST)
| Run | Metric | Value |
|---|---|---|
| V1 regression run (stock → V1) | Macro F1 | 0.627 → 0.731 |
| V1 regression run (stock → V1) | False-conclusion rate | 1.79% → 3.19% |
| V2 calib, pre-amendment rule | Conclusion recall, all calib cases | about 0.48 |
| V2 calib, pre-amendment rule | REFUTE_COVERAGE recall | 0.048 |
For comparison, the sealed TEST under Rule B gives a REFUTE_COVERAGE recall of 0.9985 (Table 3).
The support/refute evidence IDs in the output are picked only from the input’s evidence_ids. An ID that is not in the input therefore cannot structurally appear in the output.
The results
The sealed TEST has 4,002 examples: 2,202 from the standard generator and 1,800 from a generator with different generation rules (generator_disjoint). Ground truth was set by the generation rules only (0 teacher-model, human, or LLM judgments). Splits share no world, group, host, IP, or user, and there are no exact duplicates across splits.
That is entity-level separation, not content-level independence. A masked-text audit run after release (2026-10-31) masked digits, IPs, hex IDs and timestamps, then compared each claim, its sorted evidence and its variant against the training side. It found that 50 of the 4,002 TEST examples (1.25%) had a near-copy in train, train_dev or calib. All 50 are in the standard stratum and none are in generator_disjoint. This does not mean the TEST set is close to the training data: the other 3,952 examples had no near-copy under this audit. It does mean the results below include those 50 examples, because the TEST numbers were not recomputed without them.
Table 3. Sealed TEST results and pre-registered requirements
| Metric | SecDecision-27B-V2 | Stock (zero-shot) |
|---|---|---|
| False-ENOUGH | 1/2668 (0.04%) | 202/2668 (7.57%) |
| ENOUGH recall | 1297/1334 (97.2%) | 390/1334 (29.2%) |
| FULL recall | 0.946 | 0.0555 |
| REFUTE_COVERAGE recall | 0.9985 | 0.5292 |
| Relation Macro-F1 | 0.9634 | not measured |
| Pairwise accuracy | 0.9975 | not measured |
| ADJACENT stratum False-ENOUGH | 0 | not measured |
| generator_disjoint stratum False-ENOUGH / recall | 0.0 / 0.9567 | not measured |
| Safety-gap paired bootstrap 95% CI | [−0.08771, −0.06408] | n/a |
| Pre-registered CI upper bound | +0.005 | n/a |
The False-ENOUGH in the ADJACENT (adjacent-claim) stratum is 0, and mis-concluded conflict cases are 0 as well. The 95% confidence interval of the paired bootstrap (group-cluster, 2,000 resamples) of the safety gap (V2 minus Stock) is [−8.77%, −6.41%] (full precision in Table 3). The pre-registered requirement, an upper bound of +0.5% or less, is satisfied with wide margin.
All eight pre-registered gates (G0 to G7) were passed; the verdict is GO.
We state the limit of the Stock comparison as well. The Stock baseline ran Qwen3.8-27B zero-shot under a fixed evaluation contract (same input render, greedy, thinking off, max_new_tokens=64). Under that contract, Stock failed to emit a parseable status at all on 3,370 of 4,002 examples (83.96%, PARSE_ERROR). In some cases the thinking output consumed the 64-token budget before the decision output could start. That reads oddly next to “thinking off”, so we report it the way the card does: an observed behaviour of the stock model under that contract. We did not investigate why thinking text appeared with thinking turned off. This comparison is therefore not a complete proof of superiority over an optimized Stock model (tuned prompts, larger output budget, fine-tuning). The pre-registered GO verdict stands; we disclose this limit so the Stock comparison is not over-read.
Left: the rate of not missing cases that allow a conclusion (ENOUGH recall). Right: the rate of mis-concluding on cases that do not allow one (False-ENOUGH).
Verifying the public package as-is in a fresh process
The numbers were not read from the training run’s log; they were reproduced with the package that is being published. The package containing the exact bytes to be published was loaded in a fresh GPU process that had not been used for training or evaluation, and we checked that the base resolves to exactly the pinned revision, that all weights and heads load without error, and that Rule B and τ are applied with numeric precision intact.
The most important check is forward-identity. On held-out calib examples, the recomputed probabilities matched the sealed dump within a maximum difference of 1e-3. That means this package reproduces the arithmetic of the sealed TEST. All four statuses (SUPPORTED, REFUTED, INSUFFICIENT, CONFLICTED) fired normally, and out-of-input evidence IDs in the output were 0.
How to run it
The commands below are copied from the model card’s Quick start. Run them inside a download of the model repository.
pip install -r requirements.txt
# base from a local path (or HF id — the revision is pinned in config.json):
python3 inference.py --base /path/to/Qwen3.8-27B --example smoke_example.json --check-sha
# base from Hugging Face:
python3 inference.py --base Qwen/Qwen3.8-27B --example smoke_example.json --check-sha
--check-sha checks the adapter, heads and rule-module hashes against config.json. The input is one JSON object, as in smoke_example.json:
{"claim": {"text": "...", "required_facts": [{"fact_id": "f1", "desc": "..."}]},
"evidence": [{"evidence_id": "e1", "source_type": "telemetry", "text": "..."}]}
The output is one JSON line, for example:
{"example_id": "...", "claim_status": "SUPPORTED", "can_conclude": true,
"support_evidence_ids": ["e1"], "refute_evidence_ids": [], "missing_required_facts": [],
"sufficiency_score": 0.9995, "rule": "b", "gated": true}
Two warnings. First, do not load the adapter alone. The model is the adapter, the fp32 heads (heads.pt) and Rule B as one bundle, and adapter_model.safetensors by itself does not reproduce the verified behavior. Second, this is not a standard transformers classification pipeline, so use inference.py rather than pipeline("text-classification", ...).
The verified environment is narrow. The card lists one NVIDIA B200, bf16, SDPA attention, inputs of at most 4,096 tokens, and the vllm/vllm-openai:v0.28.0 image. No other GPU or software environment is listed as verified.
The Paxis and Metis view (target architecture): a gate between evidence and response
What exists today is the model release: weights, heads, and inference code. Neither SecDecision model ships inside Paxis or Metis today. The release includes no Paxis, Autonomous-SOC or Web-Chat integration and no resident serving. Everything in this section is target architecture, a design we intend to build, not a running feature.
In the intended Paxis security workflow, this model would sit between evidence collection and response planning. As the agent collects evidence for a detection, it would ask the model at every step whether a conclusion is allowed now. If can_conclude is False, the agent would not guess a response. It would take the unsatisfied required facts reported by the model as the next collection targets. When it is True, it would move to response planning with the supporting and refuting evidence IDs in hand.
The model card places SecDecision next to two public sibling models. In that pipeline, SecProof surfaces candidate issues and SecDecision judges whether the evidence collected for them is sufficient to conclude.
| Role (from the model cards) | Model |
|---|---|
| Cybersecurity retrieval embedding | Thaki-SecEmbed-8B-Preview |
| Audits small Python AI-agent runtimes for 12 named security boundaries | Thaki-SecProof-27B-V1 (the smaller 2B, 4B and 8B Previews share its recipe) |
| Evidence sufficiency and conclusion-allowance judgment | Thaki-SecDecision-27B-V2 (and the 4B-Preview below) |
In the design, a Paxis policy gate and audit log would take this model’s output as the basis of judgment. A record of an executed response would then say “based on these evidence IDs, at this sufficiency score”, not “the agent judged it intuitively”.
The design rests on an assumption we have not measured: that a wrong response costs more than one more round of evidence collection. If that holds, the gate trades the first cost for the second. The measured part is only the error rate of the gate itself, a False-ENOUGH of 1 in 2,668 on the synthetic TEST.
On the Metis side, this is a different serving workload from a chat model. Each decision is one forward pass of the frozen backbone and LoRA, the fp32 heads, and a deterministic rule, with no generation and no sampling. Inputs are capped at 4,096 tokens, and longer ones are excluded rather than truncated. The verified environment is the one in the usage section. We have not published latency, throughput or cost figures, so we make no claim about them. Serving it as a Metis endpoint is a possible later step, not something available now.
flowchart TB
A["Detection"] --> B["Security agent<br/>evidence collection (telemetry · attestation · documents)"]
B --> C["Thaki-SecDecision-27B-V2<br/>evidence-sufficiency judgment"]
C --> D{"can_conclude?"}
D -->|"False: unsatisfied facts + evidence to collect"| B
D -->|"True: status + support/refute evidence IDs"| E["Response planning<br/>policy gate + audit log"]
Target architecture only. No Paxis integration ships with this release.
The smaller sibling: Thaki-SecDecision-4B-Preview
Thaki-SecDecision-4B-Preview uses the same input format, head structure and Rule B on a Qwen/Qwen3.5-4B backbone. LoRA covers the top 8 of 32 layers, and it has its own τ (0.3484134802001108, fixed on calib for False-ENOUGH ≤ 0.01). Its verification status comes first: it has no pre-registered independent TEST pass. Its one pre-registered holdout (dev, n = 3,000) ended in ABORT_OVERLAP. An audit before the model was loaded found 58 of 3,000 examples (1.93%) near-copied from the training side, above the 1% limit, so the model was never scored on that set.
A separately labelled post-hoc holdout with those 58 removed (n = 2,942) was run once and passed its gates (FILTERED_PASS). There, False-ENOUGH was 4/1,968 = 0.0020, ENOUGH recall 834/974 = 0.8563, and FULL recall 351/484 = 0.7252. That set comes from the same synthetic generator and does not replace the pre-registered result. The development targets were FULL ≥ 0.80 and ENOUGH ≥ 0.88, and the 4B model does not meet them. Roughly one in four fully supported claims is left unconcluded. It is not validated as a 27B replacement, on real SOC logs, or for production.
The 4B card also reports a useful negative result. 27B → 4B knowledge distillation, run with the same examples, order, step count and seed as gold-label training, did not beat gold-only continuation under the pre-registered criterion. The released 4B is therefore the gold-only model. Its install steps differ from the 27B package (PyTorch 2.13.0 from the CUDA 13.0 index, then causal-conv1d 1.7.0 built from source), so copy them from its card. Don’t reuse one model’s τ for the other.
What to not trust
A judge honed only in a mock courtroom is hard to say ready for a real one. This model’s TEST is a synthetic claimworld: 4,002 examples whose ground truth was set by generation rules. It measures controlled generalization within the synthetic range, not the real SOC operational performance that involves real logs, real detections, and analyst judgment. Accuracy on real logs is disclosed as unverified.
The structural parser records 1.0 inside the generator, so in-generator performance can be saturated; generalization claims rest only on the generator_disjoint stratum (FE 0.0, recall 95.7%). The full-precision value is in Table 3. Even that stratum is synthetic, so it says nothing about real logs or about scenario families outside claimworld.
The post-release audit adds one more limit. 50 of the 4,002 TEST examples (1.25%) had a near-copy in the training data, all in the standard stratum, and the TEST numbers were not recomputed without them. Before release, the splits were checked for entity separation, not for content independence.
To re-emphasize: can_conclude being True is not an action authorization. It is a statement that the evidence is sufficient. Approving a response is a decision of a separate policy layer, and this release provides neither Paxis or Autonomous-SOC integration nor resident serving. Only the model weights and inference code are public.
Infographic generated by NotebookLM from the sources.
In one line
This model does not write a story. It looks at the submitted evidence and decides whether the claim is supported, overturned, short of evidence, or contradicted, and whether a conclusion is allowed now. On the sealed TEST, only 1 of the 2,668 cases that do not allow a conclusion was mis-concluded, and 1,297 of the 1,334 cases that do were found.
In a security agent, the moment you conclude is the moment action begins. The model that decides “may I conclude now” is the gate that guards that moment. The model, inference code, and model card are public on Hugging Face. Anyone can load the package as shipped and run it. The TEST numbers come from the sealed set, published as aggregates only (SEC_DECISION_V2_TEST_EVAL.json). The dataset is not in the repository, so outside readers cannot recompute them. What we verified is narrower: the public package, loaded in a fresh process on one B200 in bf16 inside vllm/vllm-openai:v0.28.0, reproduces the sealed computation within max |
Δ | < 1e-3. |
Related posts
- From vulnerability discovery to fix verification: building a security agent with SecProof-2B and Paxis
- One 27B model running answer generation and evidence judgment together: a shared backbone that only the last 4 layers diverge
- Thaki-CiteRank-4B: a 4B model that picks what a RAG agent should cite
Update log
2026-11-01
- Corrected the gate count: G0 to G7 is eight gates, not seven.
- Added the post-release near-copy audit of 2026-10-31: 50 of 4,002 TEST examples (1.25%) had a near-copy in the training data, all in the standard stratum. The TEST numbers were not recomputed.
- Split the old Table 1. Values fixed before TEST (Rule B, τ, τ selection rule) are now separate from the earlier V1 and pre-amendment runs that motivated Rule B.
- Matched the model-family table to the model card (Thaki-SecEmbed-8B-Preview, Thaki-SecProof-27B-V1) and added the 4B-Preview.
- Replaced “anyone can reproduce the numbers” with what was actually verified and in which environment.
- Added a usage section copied from the model card.
- Reworded the Paxis section as target architecture and stated that neither model ships inside Paxis or Metis today.
- Fixed the Mermaid edge-label syntax in the workflow diagram.