Jev content moderation: severity tiers you can threshold

The problem

Moderation is never binary. You need "obviously fine", "borderline, show to a reviewer" and "block immediately" — and you need to move the line between them without retraining anything.

How Jev handles it

A Score question against an ordered rubric gives you the tier plus the full distribution. Separate Noul questions cover specific policies. Because the threshold lives in your code, tuning policy is a config change, not a model change.

response = client.system_one(
    model="jev-latest",
    state=user_post,
    questions={
        "severity": Score(
            instructions="How harmful is this content?",
            criteria=["Benign", "Questionable", "Harmful", "Severe"],
        ),
        "targets_person": Noul(
            instructions="Does this target a specific real person?",
            criteria=NoulCriteria(
                true="Names or clearly identifies an individual",
                false="No specific individual targeted",
            ),
        ),
        "is_spam": Noul(
            instructions="Is this commercial spam?",
            criteria=NoulCriteria(
                true="Unsolicited promotion or link farming",
                false="Genuine participation",
            ),
        ),
    },
)

a = response.answers
# severity is a 0..3 weighted mean of the rubric, not a label: round it for the band
if round(a["severity"].score) == 3 or a["is_spam"].noul > 0.9:
    block()

Turning your policy into Jev content moderation questions

Good Jev content moderation starts from the policy you already publish. Each rubric level should correspond to an action you actually take — leave up, send to review, remove, remove and escalate — so the band that comes back maps to an action without translation. Name the levels in your policy's own words; "Questionable" in the sample stands in for whatever your reviewers call the grey zone.

Give each specific rule its own Noul instead of folding it into severity: targeting a real person, commercial spam, and whichever rules your community actually breaks. Separate questions get separate thresholds, and adding another costs almost nothing because every question reads the same post in parallel. Write each criteria pair so that true is the violation; TypeSafe reports that a Noul whose true side means "no" performs worse.

What to put in the state

A Jev content moderation call judges only what is in the state, so send the post plus the context a human moderator would glance at: the comment it replies to, the surface it appears on, and whether it quotes someone else. Quoting an insult to condemn it and using one are different acts, and without the parent comment the state cannot show which is which.

Leave out the author's post history unless your policy actually uses it — a long trail of old posts is the kind of unrelated detail TypeSafe says lowers accuracy. Screen images, video and audio elsewhere: the model accepts text only.

What one decision costs

Every submission is a Jev content moderation call, so cost is that call multiplied by volume. First line: the measured post with its three questions. Second: an assumed 800-token post with its parent comment attached. Third: a million posts a month at the measured size.

# the measured post, three questions
420 tokens × $0.042 / 1M = $0.0000176

# an 800-token post with its parent comment
800 tokens × $0.042 / 1M = $0.0000336

# 1,000,000 posts a month at the measured size
1,000,000 × 420 = 420,000,000 tokens × $0.042 / 1M = $17.64

Edits count as well: re-checking an edited post is another call of the same size, so budget on submissions plus edits rather than on posts.

When not to use Jev for this

• Most harmful material on your platform is images or video. Jev can moderate captions, comments and transcripts, not the media itself.
• Users actively probe the filter. TypeSafe documents that content in the state is treated as data, not as hostile, and text that argues for its own classification can move the answer. Test deliberate misspellings and "it's just satire" framing before trusting the scores.
• Every removal needs a written reason for the user or a regulator. Jev returns probabilities; the explanation has to come from your policy mapping or from another model.
• The community mostly posts in languages other than English, where TypeSafe says accuracy is lower. Measure on your own samples before letting it act without review.

Where it fits in your stack

Jev content moderation works best as the first pass in a three-lane flow: clear passes publish, clear violations are removed and logged, and everything in between goes to a human review queue with the distribution attached. The width of that middle lane is a dial — wider where a wrong removal is expensive, narrower where speed matters more.

Keep deterministic rules in front of it: exact-match blocklists and known spam domains are cheaper and more predictable in code. Behind them, a Jev content moderation call makes the judgements, and a separate image or video classifier handles media alongside it.

Notes from the field

• Tune thresholds per surface — a comment section and a landing page do not need the same line.
• Because it cannot produce text, Jev cannot be prompt-injected into writing something harmful; the worst case is a misclassification.
• Keep human review on the borderline band. The distribution tells you exactly how big that band is.

Common questions