Model evaluation · shell-command safety classification

Jev: a better shell-command safety classifier at half the price of DeepSeek 4.1 Flash

An auto-classifier stands between an AI coding agent and its shell: it lets safe commands run and stops dangerous ones, with a model judging whatever its rules can’t. I built one for Antigravity and OpenCode, and Jev was the safest model I tested, at half the price of DeepSeek 4.1 Flash.

By Zack Frank · September 19, 2026

Head to head

Safety against cost, every model tested

Both test sets, 5 passes each. Cost is per 1,000 decisions, at each host’s list price.

Results

0 no false allows n dangerous commands let through, summed over five passes false deny = a safe command the gate stopped (costs a retry, not a disaster) errors = model calls that failed in transport; the gate denies them, and they count as misses

Safety and cost

The gate's fixed rules settle the obvious commands for free. Only the rest reach a model, so only those are counted here.

False allows

main + blind tests, 5 passes each · lower is better · zero is the bar

Cost per 1,000 model decisions

USD at list price · measured tokens × published price

Where each model slips

Accuracy per category of the main test, over five passes. A red outline means that category contains at least one false allow.

Every miss

Each case a model got wrong in any pass, with the model's own stated reason from one of those passes. Nothing is summarised away.

How Jev is asked

Thresholds, re-scored from recorded answers

main test · 5 passes · Jev's model-decided cases only · each cell is false allows / false denies · outlined cell ships