A Prompt Is Not a Security Function

How one finding illustrates the need for people, process, technology — and whether we can automate the bigger picture.

What does it mean for a finding to be a "False Positive" to your team?

I watch a lot of metrics on the SAST+SCA triage harness we build at Pixee. Among them are jitter (how often the system changes its classification when given materially identical evidence), technical accuracy, and team alignment. People want these systems to be as deterministic as possible, but they also want the decisions to match how their security program actually operates.

A basic model-plus-prompt implementation (the "DIY harness") will often classify the same vulnerability differently across runs: True Positive, False Positive, Won't Fix, Accepted Risk.

Your team would probably have a strong, philosophy-informed opinion about how the finding should be classified. Letting the AI run on it twice and getting two different answers leaves you in a state of confusion and mistrust.

But why does that happen? How can the same vulnerability be defensibly classified in several different ways?

Let's look at an example — consider this java/insecure-randomness reported by CodeQL on the OSS project RavenDB:

// User.password does not allow null, so generate one
if (authMethod.equals(AuthMethod.OPENID) ||
        (authMethod.equals(AuthMethod.DB_OPENID) && !StringUtils.isEmpty(bean.getOpenIdUrl()))) {
    String randomString = RandomStringUtils.randomAlphanumeric(255);
    existingUser.resetPassword(randomString);
}

So, when authentication is set up not to use passwords, the app still sets one up — but using an insecure source of randomness. It has to set a value, because there's a non-null constraint in the database, so the developer thoughtfully put in a random-ish guard value.

This single example reveals a lot about AI in the enterprise.

So... how can this be a true positive and a false positive?

The stable technical facts are:

• A non-cryptographic random number generator creates a password-like value.
• That value is stored on the database user model.
• The configured authentication mode does not currently use the password.
• The password does not appear to be returned to the user.
• The risk could become concrete if the authentication mode later becomes password-based.

Those facts do not change from run to run, but the classification can!

A strict AppSec program might want this called a True Positive. The code uses an insecure primitive to generate a credential, and the program does not want latent risk sitting around waiting for a configuration change.

An exploitability-driven program might call it a False Positive. The password is unused in the current authentication mode, and there is no apparent path for an attacker to obtain enough output to predict future values.

Both of these are perfectly normal responses that you'd see in enterprises. Many teams will have a bias towards tolerating or removing abstract risk, but they'll also have directives around secret handling and every other dimension of every other type of security issue.

An AI agent, on the other hand, will arbitrarily pick one set of facts to argue, and emit plausible text and a defensible conclusion. But it won't be your program's conclusion.

A real harness turns a raw model capability into a usable, reliable, and verifiable program

Personally, I might be less tolerant of abstract risk, and I might not want any more scanner noise in the future, so I might decide I want this to be called a True Positive.

But with our basic DIY harness, this will not be classified the way I want.

The first step on the road to doom is to start cramming guidance into the prompt, and supporting skills. Then you'll figure out you want different rules minimally for different repos, and potentially for different BUs, etc.

This will all spiral out of control, and you'll end up with a sprawling mess of markdown files. How do you audit and get reporting on this system? How do you know the "system" is doing what you want at scale? LLMs are still really bad at multi-instruction following, and this entire system is built on multi-instruction following, and has high compounding derangement and high agentic risk.

Eventually, you end up at the need for a use-case-specific, purpose-built harness.

As my buddy Jeff likes to say, security, like many things, is a People + Process + Technology problem. Without a harness, you'll never "prompt" your way out of not having the People and Process parts.

A great harness requires:

• Deterministic integrations with the systems it interfaces, so as to reduce the likelihood of compounding derangement in just detecting events, moving information around, etc.
• AI controls that enable maximally efficient model routing, data sovereignty, cost accounting and governance, backpressure management, etc.
• Customization tools that help tune the system's behavior (e.g., uploading secure coding guidelines).
• Memory that compounds, and can attribute influenced behavior back to user feedback.
• Caching and other state management that saves you from redundant or unnecessary analysis, and allows for detailed reporting on key points in the system.
• Reporting that allows people who own the high-level business function to pop their head into key parts and audit the detailed trace behavior of the system, as well as understand insights across different parts of the business.

You cannot prompt your way out of missing the People and Process parts — the model is only one component. The security function is the surrounding system: policies, evidence requirements, precedent, controls, reporting — and the people who remain accountable for its decisions.

A prompt can make a model sound like your AppSec team, but the harness is what makes it behave like one.

How does the RavenDB example above change in the presence of a real harness? Let's see:

• With AI controls, we can use our own models and protect our IP, prioritize high-severity findings first, decide behavior in moments of token scarcity, etc.
• With memory, we can see how the team has previously felt about latent risk issues, in order to make sure we are in alignment on classification, with a link to the past feedback that led us to this position.
• With caching + state, we could see this vulnerability has already been adjudicated yesterday, and fetch the analysis from yesterday given no pertinent code changes have been made.
• With context (like your secure coding guidelines), we can honor what your tribal knowledge has to offer about this rule specifically.
• With reporting, we can audit the memory entries that are influencing the most results, and make sure they're trustworthy.

I believe we'll need a purpose-built harness for every intelligence-based "use case" or "cognitive task" we do, to minimize cost, get reporting, etc. The argument here generalizes fairly well to just about any human step — but I'll write more about that later!