What an OWASP ASVS assessment can and cannot automate
ASVS is a list of properties somebody has to verify, not a list of things that are already wrong. Almost every tool that claims to cover it blurs that distinction, because a scanner enumerates what it recognises and a verification standard asks whether a property holds. Below is which of its requirements a machine can settle at all, which ones will always need a person, and the exact clauses this platform maps and tests.
Which layer settles which clause
Every clause named on this page is one the control pack really carries, and each one is settled by a named layer rather than by the report in general. A clause no layer reached is uncovered, which is a different result from passing and is shown as a different result.
Repository validation
E2Reads the source, which is where the coding clauses live. How queries and commands are assembled answers v5.0.0-1.2.4 and v5.0.0-1.2.5, where credential material sits answers v5.0.0-13.3.1, the resolved dependency graph checked against public advisories answers v5.0.0-15.2.1, and a record fetched from a request identifier with no ownership check is the source half of v5.0.0-8.2.2. Findings cite the file they came from, so the reader can disagree with one.
Runtime validation
E4Builds and runs the application in an isolated environment and asks it. Type-violating and oversized payloads against the endpoints it advertises answer v5.0.0-2.2, which nothing static can settle. Asking for one account's record with another account's session answers v5.0.0-8.2.2 as a demonstration rather than an inference. Forced failures answer v5.0.0-16.5, and the health surface is what the pack maps to v5.0.0-16.2 and v5.0.0-16.3. A clause the run could not reach is reported uncovered rather than passed.
Public URL scan
E4Reaches five clauses and no more. Transport security answers v5.0.0-12.1 and v5.0.0-12.2, the response headers answer v5.0.0-3.4, a small bounded burst against a public endpoint answers v5.0.0-6.3.1, and a request for a path that cannot exist answers v5.0.0-16.5, but only when the application really returns an error: a catch-all route that answers 200 means the error path was never reached, and the clause stays unverified rather than passing. Everything it reports it observed against the running system, so it is graded as runtime evidence. Its weakness is reach, not strength.
Every ASVS 5.0.0 clause the pack carries
Ten of the thirteen controls carry an ASVS 5.0.0 clause. Each row gives the clause, how the result is established, and what passing requires. The other three carry none: database indexes at scale, verified backups and a deployment rollback path are reliability and operations controls, and ASVS is a security verification standard, so their absence here is correct rather than a hole.
What ASVS is, and the six things people get wrong about it
None of these are subtle points of interpretation. They are the mistakes that make an ASVS claim worthless, and most of them are made by the tooling rather than by the reader.
It is a verification standard, not a vulnerability list
The difference decides everything downstream. A vulnerability list names things that are wrong, so a tool covers it by recognising them. A verification standard names properties that must be shown to hold, so covering it means proving a positive. That is a harder job and a much more useful one, because the result tells you what is true rather than what happened to be recognised. It also means a clean scan is not a passing verification: nothing was proven, only nothing was matched.
The levels are a scope you choose, not a grade you are awarded
Requirements are sorted into levels so an application can be verified against a bar proportionate to what it holds. Choosing a level happens before any work is done, and reaching one is a claim about that entire scope having been verified. No run awards a level, ours included. A product that tells you it has certified you at a level has redefined a word that already had an owner.
Most of the standard cannot be automated, and that is deliberate
A large part of ASVS asks whether a decision was correct: whether the trust boundary is in the right place, whether a business rule holds under the sequence a real user can perform, whether a key was generated and stored the way its threat model needs. A machine can settle whether a header is present and cannot settle whether a design is sound. Anyone selling full coverage of the standard is selling either a different standard or an opinion dressed as a result.
The automatable part is small, repeatable and worth doing properly
The clauses a machine can settle are the ones where the answer is the same every time and comes from observing the system rather than reasoning about it. That is a minority of the standard and it is the minority worth automating hard, because those are the clauses that quietly regress between releases. A person reviewing a design once a year is expensive and correct. A machine rechecking transport, headers, authorisation behaviour and dependency state on every deploy is cheap and never gets bored.
Version 5.0.0 renumbered the standard, and most writing still describes 4.0
The pack here carries both vocabularies, so the drift is visible in one place. Transport security is chapter V9 under 4.0 and v5.0.0-12.1 under 5.0.0. Injection-safe data access is V5.3 under 4.0 and v5.0.0-1.2.4 under 5.0.0. Server-side authorisation is V4 under 4.0 and v5.0.0-8.2.2 under 5.0.0. A report that quotes a 4.0 chapter number beside a 5.0.0 requirement is quoting two different documents at you, and one of the two is not the one it claims to be testing.
Coverage quoted without clause numbers is a claim you cannot check
A percentage, a rating or the phrase aligned with the standard tells you nothing, because you cannot tell which requirements were in the denominator. The only version of this claim that survives contact with a reader is the specific one: these clauses, established this way, and everything else untouched. That is the shape of the table above, and it is the shape to demand from anyone else who raises the subject.
How to use ASVS when there are four of you
The standard was written to be used by a verification team. It is still the best list in the field for a team that does not have one, provided the first two steps happen in this order.
Pick the level before you pick the tooling
The level decides which requirements are in scope, and the scope decides what any tool is being measured against. Doing it the other way round produces the familiar bad outcome, where the tool's coverage silently becomes the definition of the standard and everything it cannot see stops existing.
Split the in-scope list in two
Go through the requirements once and mark each one machine-settleable or person-settleable. It takes an afternoon and it is the highest-value afternoon in this whole exercise, because from then on you know exactly what you are buying from a tool and what you are still on the hook for.
Automate the machine half and run it on every deploy
These are the clauses that regress quietly: a header dropped in a config change, a redirect lost behind a new edge rule, a dependency that acquired an advisory since Tuesday. Checking them once tells you about one moment. Checking them on every deploy is the only version that stays true.
Book the person half as work rather than filing it as a gap
Design review, business logic, cryptographic choices and anything about trust boundaries need someone thinking. Put them on the roadmap with a date. A requirement recorded as not applicable because nobody had time is the same requirement, with a worse label.
Record how each answer was established, not just the answer
Six months later, the useful part of the record is not that a clause passed. It is whether it passed because someone read the code, because a probe observed the running system, or because a person said so. Those are three different strengths of claim, and the first buyer who reads your evidence will want to know which one they are looking at.
What the result looks like
An illustrative extract, not a real customer result. Every row carries the evidence grade behind it, and a control nobody could establish says so.
| Control | Status | Evidence | Note |
|---|---|---|---|
| Transport security (TLS) is enforced | Verified | E4 | v5.0.0-12.1 and v5.0.0-12.2, confirmed against the running deployment |
| Injection-safe data access | Verified | E2 | v5.0.0-1.2.4 and v5.0.0-1.2.5, from reading how queries are assembled |
| Object ownership is enforced server-side | Failed | E4 | v5.0.0-8.2.2, one account's record returned to another account's session |
| Security response headers are present | Partial | E4 | v5.0.0-3.4, no content security policy on the document response |
| Verified backups and restore procedure | Uncovered | Not graded | The pack maps no ASVS clause to it, and no layer that ran could establish it |
Mapped to published standards
Each control is tested against a named clause, so a result means something outside our own vocabulary. Mapping is not certification, and none of these bodies endorse StackAttest.
What this does not tell you
- It is not a penetration test. A deterministic check and a runtime probe are not an adversarial human engagement, and there are classes of weakness only a person hunting for them will find.
- It is not SOC 2 and does not replace it. SOC 2 attests to organisational controls over time; this validates technical controls in the software. They answer different questions for different buyers.
- It does not replace human due diligence. It gives a reviewer repeatable evidence to start from, which is a different thing from being the reviewer.
- Controls the run could not reach are reported as uncovered rather than passed. A result you did not earn is worse than no result.
- It is not a certification and there is no such thing to award. OWASP and NIST publish these documents openly. Neither one endorses, accredits, reviews or has any relationship with StackAttest, and a result produced here is a result against named clauses that you can check yourself, which is the only kind worth having.
- It does not cover the standard. Ten controls carry ASVS 5.0.0 clauses between them, which is a small fraction of the requirements in any level. Everything outside that set is untouched rather than passed, and a report never counts an untouched requirement as a good result.
- It does not award a verification level. A level is a scope decision plus the work of verifying that entire scope, most of which needs a person. Nothing in a run can reach one, so nothing in a run claims one.
- It does not tell you whether the requirements you chose were the right ones. The level, the applicability decisions and anything about your threat model are yours. A tool can tell you what holds; it cannot tell you what should have mattered to you.
Questions
Can an OWASP ASVS assessment be automated?
Part of it, and it is worth knowing which part before you buy anything. Requirements that can be settled by observing the system, such as transport security, response headers, authorisation behaviour under a real request, and dependency state, automate well and should be rechecked continuously. Requirements that ask whether a design decision was correct cannot be automated by anyone, because the answer depends on a threat model a machine was never given.
Which ASVS clauses does StackAttest actually test?
Ten of the thirteen controls carry an ASVS 5.0.0 clause: v5.0.0-12.1 and v5.0.0-12.2, v5.0.0-3.4, v5.0.0-6.3.1, v5.0.0-8.2.2, v5.0.0-2.2, v5.0.0-1.2.4 and v5.0.0-1.2.5, v5.0.0-13.3.1, v5.0.0-16.5, v5.0.0-16.2 and v5.0.0-16.3, and v5.0.0-15.2.1. The table above says how each is established and what passing requires. Nothing outside that list is claimed.
Is this ASVS 4 or ASVS 5?
Both are in the data and the pages here quote the 5.0.0 identifiers. Controls also carry the older 4.0.x chapter references so historical results stay interpretable, but the current mapping is the versioned one. This matters more than it sounds: the chapters were renumbered, so a 4.0 chapter number and a 5.0.0 requirement number that look similar are usually not about the same thing.
Does passing these clauses make us ASVS certified?
No, and there is nothing to be certified against by us. OWASP publishes the standard and does not endorse, accredit or review StackAttest or anyone assessed with it. What a run produces is a result against named clauses, with the evidence grade that records how each was established, which you or a buyer can check line by line.
What is the difference between ASVS and the OWASP Top 10?
The Top 10 is an awareness document listing broad risk categories, and it is deliberately short. ASVS is a verification standard with hundreds of specific requirements, meant to be worked through. A team that has read the Top 10 knows what tends to go wrong. A team that has verified against ASVS knows what is true of their own application, which is the harder and more useful position.
We are four people. Is ASVS realistic for us?
The standard is realistic, the way most teams use it is not. Pick a level, split the in-scope requirements into the ones a machine can settle and the ones a person must, automate the first half so it reruns on every deploy, and schedule the second half instead of quietly dropping it. That is a few days of thinking and then a background cost, rather than a project nobody has staff for.
What does a run leave uncovered?
Everything outside the clauses in the table, plus any clause in it that the layers you ran could not reach. A URL scan reaches five of them and nothing else, so a report from a URL scan alone names the rest as uncovered rather than passed. Three of the thirteen controls carry no ASVS clause at all, because they are reliability and operations controls rather than security verification.
Related
Find out what is actually true of your application
Start with the layer you can run today. The report says which layers ran and what they could not reach, so the result is honest about its own limits.