Key takeaways

  • A vulnerability scan asks what known weaknesses exist across many systems. A penetration test asks whether a person can break in and how far they get. A red-team exercise asks whether your people and tools notice and stop a determined adversary.
  • Scans are cheap to repeat and should be routine. Penetration tests are scoped and periodic. Red teams suit organizations that already have detection and response to test.
  • PCI DSS 4.0.1 asks for both: internal and external vulnerability scans at least every three months, and penetration testing at least annually and after significant changes.
  • If you have never scanned, do not start with a red team. You will pay to be told things a scan would have found.
  • Every engagement needs written authorization, a defined scope, agreed hours and a way to stop at once. Our article on written authorization lists what to include.
  • Fix, then retest. A report is only useful if the findings are closed and the closure is checked.

Three tools, three questions

Buyers often ask for a penetration test when they mean a scan, or a red team when they want a penetration test. The words matter because the cost, the duration and the result are very different. The way to choose is to start from the question you need answered.

Vulnerability scan: what known weaknesses do we have?

A vulnerability scan is mostly automated. A tool checks systems against a database of known flaws and misconfigurations, such as missing patches, outdated software, weak settings and exposed services, and produces a long list. It covers many systems quickly, and it can be run every week or every month. It does not prove anything is exploitable, and it cannot find flaws in your own business logic, such as a customer being able to read another customer's invoice. Scans have false positives and false negatives, so a person should triage the results. Our article on scans versus penetration tests goes into this in detail.

Choose a scan when you want a baseline, a regular health check, evidence for an insurer or auditor, or a list for your patching team.

Penetration test: can someone get in, and how far?

A penetration test is a goal-driven exercise in which skilled testers try to exploit weaknesses in a defined scope, such as an application, an external network, an internal network, a wireless network or a cloud tenant. They use tools and manual judgment, chain small findings into larger ones and prove impact: this account can read these records. NIST SP 800-115 describes the phases of planning, discovery, attack and reporting. The OWASP Web Security Testing Guide and the OWASP Mobile Application Security Testing Guide set out what to test in applications. The result is a ranked list of confirmed findings with evidence, fixes and a retest.

Choose a penetration test before a launch, after a major change, once a year for systems that matter, when a customer or regulator asks for it, or when you want to know the real impact of the scan's findings. Our article for executives covers scope and frequency.

Red team: would we notice, and would we stop it?

A red-team exercise emulates a specific adversary with a specific objective, such as reaching the finance system or exfiltrating a customer database, using any realistic route that is in the rules: phishing, stolen credentials, a weak vendor connection, and sometimes physical entry. Unlike a penetration test, the goal is not a complete list of weaknesses. It tests the defenders, namely monitoring, alerting, response and decision-making. Typically only a small control group knows it is happening, so the real response is observed. Techniques are commonly mapped to the MITRE ATT&CK knowledge base, so the findings read as which behaviors were detected and which were missed. A variant called a purple team has attackers and defenders work together in the open to tune detections.

Choose a red team when you already have a monitored environment, a security operations function and a response plan, and you want to find out whether they work. If you do not, the result is the first sentence of the report: nobody noticed, because nobody was watching.

A quick comparison

  • Question: scan, what known flaws exist; penetration test, what can be exploited and what is the impact; red team, can we detect and respond.
  • Method: scan, automated; penetration test, tools plus human skill; red team, human-led, stealthy, objective-driven.
  • Scope: scan, broad; penetration test, defined systems; red team, defined goal, wide range of routes.
  • Frequency: scan, weekly to monthly; penetration test, yearly and after big changes; red team, occasional, when detection is mature.
  • Awareness: scan and penetration test, the IT team usually knows; red team, defenders often are not told.

What standards already require

If you take cards, PCI DSS 4.0.1 sets minimums. Requirement 11.3 calls for internal vulnerability scans at least once every three months and external scans by an approved scanning vendor at least every three months. Requirement 11.4 calls for penetration testing at least annually and after significant changes, internal and external, with a documented methodology, and for retesting after corrections. Red teaming is not a PCI requirement. Banking examiners and insurers ask similar questions in different words, and our article on penetration testing for community banks covers what they expect.

How to choose

  • No regular scanning yet: start with scans and fix the top of the list.
  • Scanning in place and a new application, a new network or a major change: order a penetration test of that scope.
  • An annual test already, and a monitored environment with response procedures: consider a red team or a purple team.
  • A particular worry, such as whether a stolen laptop reaches your finance data: scope a penetration test around that one question.

Before you sign

  • Written authorization from someone who has the authority over every system in scope, including cloud and hosted systems whose providers have their own rules.
  • A scope, hours, excluded systems, a contact who can stop the test and a plan for data the testers might see.
  • Clear reporting: ranked findings with evidence, a fix for each and a retest.
  • Fix capacity. A test is wasted if no one has time to act on it.

If you are not sure which fits, describe the worry in plain words and our team will recommend one. Start with our Cybersecurity service, or the Security category of our shop for fixed-scope assessments.

Sources

This article is general information. Scope, standards and requirements depend on your systems, contracts and regulators. Last reviewed .

Keep reading

Need help with this?

We find out whether everything you run is safe from attackers, then we fix it.

Cybersecurity