Agentic Fraud Detection and Automated Response

For most of the last decade, the difficult part of fraud detection was the model. Teams competed on features, on gradient-boosted trees and on the final decimal point of AUC. That era is quietly ending. Scoring a transaction is becoming close to free and a free classifier changes the shape of the entire problem.

When the model is no longer the scarce resource, the advantage moves to the system that decides what to do with each score: which cases resolve automatically, which deserve a closer look and which belong in front of a person. This article walks through that system layer by layer and explains why the workflow, not the classifier, is now where fraud programs are won or lost.

The problem: Why fraud detection breaks in production

Fraud is a rare-event problem and rare events are unforgiving. In a healthy portfolio the overwhelming majority of transactions are legitimate, so a model that predicts "not fraud" on everything can still report high accuracy while catching nothing that matters. Accuracy is the wrong lens and many teams learn this only after a model that looked excellent in a notebook fails to move a single number in the business.

The costs are also asymmetric and they are operational rather than statistical. A missed fraud is a direct financial loss. A false positive is a blocked genuine customer, a support call, a refund and a small permanent dent in trust. Analysts drown in alerts that are mostly noise. Fraud patterns drift from one week to the next, so a model trained on last quarter degrades on this quarter without anyone noticing until the losses appear.

So the question that actually matters has very little to do with model accuracy. It has to do with how the business behaves around each prediction and that is a systems problem. It also happens to explain why a strong classifier so often fails to move a single number that leadership cares about. The rest of this article is about building that behavior rather than the classifier.

The blueprint: The architecture at a glance

The design rests on a single principle: spend compute and human attention only where uncertainty is real. Everything else should resolve on its own, instantly and cheaply. Five stages carry a transaction from a raw event to a resolved decision and then feed the outcome back into the model as fresh training signal.

The five stages: score cheaply, calibrate and route by confidence, investigate only the uncertain few, decide with a human and learn from every verdict.

Layer 1: The commodity scorer

Every transaction enters the system and receives a probability of fraud. Fraud teams have produced that number for years, so the probability itself is not the interesting part. What changed is the price. Producing a good score now costs almost nothing and takes almost no bespoke training.

A model like TabPFN is a good illustration. It is a transformer that was pre-trained to perform tabular classification in a single forward pass. You place labelled examples in its context, hand it a new row and it returns a calibrated probability in milliseconds, with no gradient training on your own data. What used to be a modeling project becomes an inference call.

TabPFN, the Prior-Data Fitted Network for tables, is a model trained once, by its authors, on millions of synthetic tabular problems. At use time it does not learn from your data in the usual sense. It reads your examples as context and predicts the new case directly, the way a language model reads a prompt. The practical effect is a strong, reasonably calibrated classifier available without a training pipeline.

In code, the scoring step is almost anticlimactic. There is no training loop to babysit and no hyperparameter search to run.

# TabPFN — a tabular foundation model. No training loop.
from tabpfn import TabPFNClassifier
clf = TabPFNClassifier(device="cuda")
clf.fit(X_context, y_context)     # "fit" = load context, not gradient training
scores = clf.predict_proba(X_test)[:, 1]   # one forward pass

None of this makes TabPFN the only option or the right choice for every portfolio. The real takeaway is simpler than that. A capable, calibrated score has become close to a commodity input and once the score is both cheap and good, your edge has to come from everything that happens after it. That is what the rest of this system is about.

Layer 2: Calibrated deference

A probability is not a decision. A score of 0.70 means very different things depending on how well the model is calibrated and how costly a mistake would be in that context. This is where many systems quietly fail. They threshold a raw score, pick a cutoff that looked reasonable on a validation set and hope it holds. It rarely holds for long.

The stronger approach is to attach a statistical guarantee to the score and then let that guarantee drive the routing. Conformal prediction does exactly this. It wraps any scorer and, instead of a bare number, produces a decision with a controlled error rate that you choose in advance. Cases the model is confident are safe get approved automatically. Cases it is confident are fraudulent get blocked automatically. The genuinely uncertain minority and only that minority, is deferred for investigation.

Conformal prediction is a method that turns a model score into a decision with a formal guarantee on how often it will be wrong. You set a tolerance, for example one percent and the method calibrates thresholds so that the automated decisions respect that error rate on new data. The cases that cannot be resolved within your tolerance are flagged as uncertain rather than forced into a guess. It is a principled way to decide who decides.

The routing logic is only a few lines. You set the thresholds on a held-out calibration set, then send each transaction down one of three paths.

# Split-conformal thresholds on a calibration set
t_lo = np.quantile(cal_scores[y_cal==0], 0.985)   # below -> auto-approve
t_hi = np.quantile(cal_scores[y_cal==1], 0.25)    # above -> auto-block

decision = np.where(scores < t_lo, "auto_approve",
           np.where(scores >= t_hi, "auto_block", "defer"))

Layer 3: The investigation agent

The deferred cases are the ones worth spending money on, because they are the cases where a good decision actually changes an outcome. For each of them, an LLM agent assembles a case file. It gathers recent history for the account, reads the feature attributions that explain why the score landed where it did and writes a short, plain-language brief with a recommended action and a stated confidence.

SHAP, short for SHapley Additive exPlanations, assigns each input feature a contribution to a single prediction, grounded in cooperative game theory. In practice it answers the question "which features pushed this particular transaction toward the fraud side and by how much." Those attributions are what let the agent explain a score in words rather than leaving it as an opaque number.

The investigation step is itself a short function. It hands the model output and the feature attributions to a language model and asks for a structured brief, in a fixed shape the analyst can scan.

# The agent turns numbers into a decision-ready brief
import anthropic
client = anthropic.Anthropic()

def investigate(tx, score, status, top_features):
    prompt = (f"You are a fraud-investigation analyst. Write a short case file.\n"
              f"Transaction: {tx}\n"
              f"Model risk score: {score:.3f}  ({status})\n"
              f"Top contributing factors (SHAP): {top_features}\n"
              f"Return: (1) one-line summary, (2) why flagged, (3) risk "
              f"factors, (4) recommended action, (5) confidence.")
    msg = client.messages.create(model="claude-opus-4-8",
            max_tokens=500, messages=[{"role":"user","content":prompt}])
    return msg.content[0].text

Here is the kind of artefact the agent produces for a deferred transaction. It is written for a human reviewer and it makes the model reasoning legible.

Investigation brief (example agent output for a deferred transaction)

Summary
A mid-value transaction whose feature pattern deviates sharply from this card's normal behaviour but not decisively enough to block outright.

Why it was flagged

• Features V14 and V17 sit far into the fraud-associated range, the strongest historical signals in this portfolio.
• V12 and V10 reinforce the same direction.
• The amount is modest, which is consistent with a card-testing probe rather than a large cash-out.

Risk factors
The pattern resembles early-stage testing of a compromised card. It is isolated for now but the feature signature is concerning.

The agent is deliberately not the decision-maker. It prepares the decision. It turns a vector of anonymized features into something a person can read and act on in seconds rather than minutes and it does so consistently, at three in the morning, on the thousandth case of the shift.

Layer 4: The human decision

The analyst receives a case that is already framed. They read the brief, agree or disagree with the recommendation and choose to approve or block. That choice triggers the operational action: settle and release the transaction or freeze it and raise an alert. The human sits exactly where the stakes and the ambiguity are highest and nowhere else.

Two benefits follow from this placement. The first is throughput. Analysts spend their time only on hard, pre-packaged cases, so a small team can cover a large volume without the usual fatigue and error that come from wading through raw alerts. The second is accountability. Every escalated decision carries a written rationale and a named decision-maker, which is the difference between a defensible process and a black box.

Layer 5: The learning loop

Every human verdict is a fresh, high-quality label. More than that, it is a label on precisely the cases the system currently finds hard, which is the most valuable kind of training data there is. Each decision is written to a label store, drift is monitored continuously and the model is recalibrated and retrained on a schedule. The system improves exactly where it is weakest and it does so as a natural by-product of running.

The economics: What it costs and what it saves

The cost structure inverts in a way that is easy to miss. Because scoring is cheap and only the uncertain minority consumes an LLM call and a human minute, total cost scales with genuine ambiguity rather than with volume. Doubling transaction volume does not double investigation cost. It only adds work in proportion to how many new cases are actually hard.

These figures are illustrative and the real numbers depend entirely on your portfolio, your risk appetite and the quality of your data. That dependence is the whole point. The architecture is sound but the outcome is decided in the details of the build, which is where careful engineering earns its keep.

Applied: Where this pattern fits your business

Although the walkthrough uses payment fraud, the pattern generalizes to any high-volume stream of decisions where the positives are rare, costly and constantly shifting. Anti-money-laundering alert triage, insurance claims review, KYC and onboarding risk, account-takeover and abuse detection and lending decisions all share the same shape and all benefit from the same discipline of calibrated deference, machine-prepared investigation and human judgment reserved for the genuinely hard cases.

Building it well takes more than a good model. It takes clean data foundations and dependable feature pipelines, a calibrated scorer with an honest uncertainty gate, an investigation layer grounded in real context rather than generic prompts, a human-in-the-loop design that respects both throughput and accountability and the operational plumbing to watch for drift and retrain without drama. This is the work we focus on at Neulaxy: taking a promising model and turning it into a production system that a business can rely on and a regulator can audit.

The takeaway: Wrapping it up

The classifier is becoming the cheap part of fraud detection. The durable advantage is the system around it: calibrated deference so the machine acts only when it is sure, an investigator that prepares each hard case, humans placed exactly where judgment matters and a loop that turns every verdict back into learning. None of that needs a bigger model. What it needs is engineering discipline and an honest view of where the uncertainty and the cost actually live.

If there is one idea to carry away, it is this. Stop asking how accurate your model is and start asking what your organization does with each score. Once the second question is answered well, the first one mostly takes care of itself.