How to evaluate a vendor's technical claims without being technical

Executives evaluating vendors are usually told they have two options: trust the vendor, or find an engineer to interrogate them. There is a third, and it is the one professional buyers actually use. You assess the quality of the evidence rather than the technology itself, and that is a transferable skill that has nothing to do with understanding algorithms.

The four questions that do most of the work

Every technical claim can be pressed with the same four questions. They are not adversarial and a good vendor will welcome them.

1. What exactly was tested? A claim about an application is not a claim about the infrastructure it runs on, and a claim about one product is not a claim about the one you are buying.
2. Who tested it? The vendor, a tool the vendor runs, or an independent party. All three are legitimate; they are not equally strong, and the difference should be stated rather than blurred.
3. When? Security claims decay. A result from eighteen months and four hundred deploys ago describes a system that no longer exists.
4. Against what standard? A named framework such as OWASP ASVS, NIST SSDF or SLSA means someone else defined the bar. Without one, the vendor set their own.

You will learn more from how these are answered than from the answers. Precision is a signal. A vendor who says a named standard, a date, and who ran the test is operating a real process. Warmth in place of specifics is the finding.

Answers that should slow you down

• Military-grade or bank-level security. Neither term means anything and both are used exclusively by people avoiding a specific claim.
• We are fully compliant, with no framework named and no report offered.
• We use encryption, offered as though it were a conclusion rather than a component.
• Our provider handles security. Cloud providers secure their platform; what you build on it remains yours, and confusing the two is common and consequential.
• A certificate with no scope statement. The scope is where the entire meaning lives, and it is often narrower than the logo implies.

What to ask a vendor to send you

Rather than asking whether they are secure, which invites a yes, ask for artefacts. What you get back is the real answer.

• A current report or attestation with its scope and date visible on it.
• The list of controls tested and, importantly, which ones failed or were not applicable. A document with no failures anywhere describes a test that was not difficult.
• Their process for known vulnerabilities in dependencies, and how quickly critical ones are patched.
• What happens to your data if you leave, and who else can access it while you stay.
• A named contact for security questions. Absence of one tells you the function does not exist.

Calibrating for the size of the vendor

The same evidence bar applied to every vendor produces bad decisions. A twelve-person startup will not have a mature certification, and demanding one selects for vendors who bought a badge rather than those who do the work.

Ask instead for evidence proportionate to the risk you are taking. If the vendor handles no sensitive data and is easy to replace, a recent independent check of technical controls is a reasonable ask. If they will hold customer records or sit in a critical path, the bar rises accordingly. The question is always whether the evidence matches the exposure, not whether it matches a template.

Where executives actually learn this

There is no single qualification for this, and anyone selling one should be treated with the scepticism described above. In practice the skill is taught in a few places: executive education programmes at business schools that cover technology governance for boards, director training run by institutes of directors, and the internal frameworks large firms use to structure diligence. Standards bodies also publish their frameworks openly, and reading the control list of OWASP ASVS once is more useful than most courses, because it shows you what the questions are.

The honest summary is that the method above is most of it. Evidence, source, date, standard. Apply it consistently and you will out-perform a great deal of formal training.

What good looks like from the vendor's side

It is worth knowing what a strong answer looks like, so you recognise one. A vendor who is doing this properly can show you which controls were tested, what the result was for each, how strong the evidence behind that result is, when it was gathered, and which named standard the control maps to. They will also be willing to show you the ones that did not pass.

That is what StackAttest produces for the companies it validates, in a page a buyer can read without being technical. If a vendor you are evaluating has one, it will answer most of this article in about a minute. If they do not, the four questions still work.