jev·agent

← All use cases

Use case · Mixed

Phishing and email security classification with Jev

Jev's single raw verdict lost on accuracy to a cheap LLM in an independent test — but combining five cheap signal questions from the same call beat it decisively.

The problem

Phishing classification is high volume and latency-sensitive, which makes the speed and cost profile attractive. But security is exactly where you should not take a vendor's accuracy claim on faith.

How Jev handles it

Rather than asking one "is this phishing?" question, ask several orthogonal signal questions in the same call — sender plausibility, urgency pressure, credential solicitation, link/domain mismatch — and combine them in your own classifier.

python
def signal(instructions, yes, no):
    return Noul(instructions=instructions,
                criteria=NoulCriteria(true=yes, false=no))

response = client.system_one(
    model="jev-latest",
    state=email_text,
    questions={
        "urgency": signal("Does this create artificial time pressure?",
                          "Deadlines or threats of loss", "No time pressure"),
        "credentials": signal("Does this ask for credentials?",
                              "Requests login, password or MFA code", "No credential request"),
        "sender_odd": signal("Is the sender identity inconsistent?",
                             "Display name conflicts with address", "Sender looks coherent"),
        "link_mismatch": signal("Does link text disagree with its destination?",
                                "Anchor text and href point elsewhere", "Links are consistent"),
        "unusual_ask": signal("Is an unusual financial action requested?",
                              "Wire transfer, gift cards, payment change", "No financial request"),
    },
)

# Five probabilities in, your own classifier decides
features = [response.answers[k].noul for k in
            ("urgency", "credentials", "sender_odd", "link_mismatch", "unusual_ask")]
is_phishing = my_logistic_model.predict_proba([features])[0][1]

Does it actually work?

Independent evidence

Across 2,000 phishing emails, Jev's own verdict was statistically worse than Claude Haiku 4.5 (McNemar p < 0.0001) while winning on speed and cost. A cross-validated logistic regression over five signal questions asked in the same call reached 95.1% accuracy, AUROC 0.988, ECE 0.027. Latency was measured wall-clock from France against US-hosted services.

anisselbd/jev-phishing-bench

Notes from the field