Model evaluation · shell-command safety classification

Jev: a better shell-command safety classifier at half the price of DeepSeek 4.1 Flash

An auto-classifier stands between an AI coding agent and its shell: it lets safe commands run and stops dangerous ones, with a model judging whatever its rules can’t. I built one for Antigravity and OpenCode, and Jev was the safest model I tested, at half the price of DeepSeek 4.1 Flash.

By Zack Frank · September 25, 2026

Head to head

Safety against cost, every model tested

Both test sets, 5 passes each. Cost is per 1,000 decisions, at each host’s list price.

Results

0 no false allows n dangerous commands let through, summed over five passes false deny = a safe command the gate stopped (costs a retry, not a disaster) errors = model calls that failed in transport; the gate denies them, and they count as misses $ / 1k decisions = USD cost per 1,000 model decisions (all models priced in USD per million tokens)

Safety and cost

The gate's fixed rules settle the obvious commands for free. Only the rest reach a model, so only those are counted here.

False allows

main + blind tests, 5 passes each · lower is better · zero is the bar

Cost per 1,000 model decisions

USD at list price · measured tokens × published price

Where each model slips

Accuracy per category of the main test, over five passes. A red outline means that category contains at least one false allow.

Every miss

Each case a model got wrong in any pass, with the model's own stated reason from one of those passes. Nothing is summarised away.

How Jev is asked

Thresholds, re-scored from recorded answers

main test · 5 passes · Jev's model-decided cases only · each cell is false allows / false denies · outlined cell ships