Key takeaways

  • A penetration test attempts to breach your systems with the tools and techniques an adversary would use. It gives assurance that your own vulnerability management works; it is not a substitute for it.
  • Defensive security prevents, detects and responds. Offensive security tests whether those defenses hold. You need both, and they should talk to each other.
  • Scope is the most important decision. It sets the boundaries, test type, timing, rules of engagement and reporting requirements before anyone touches a system.
  • A test only shows what was vulnerable on the days it ran. Test after significant change, fix what you find, and retest.

What a penetration test is

A penetration test, or pen test, is an authorized attempt to break into your systems. The UK National Cyber Security Centre defines it as a method for gaining assurance in the security of an IT system by attempting to breach some or all of that system's security, using the same tools and techniques as an adversary might. The U.S. National Institute of Standards and Technology describes it as security testing in which assessors mimic real-world attacks to identify ways of circumventing the security features of an application, system or network. It often involves launching real attacks on real systems and data.

The word that matters most is authorized. Every action is agreed in writing in advance, performed by named testers, and limited to defined systems and times. The same techniques without that agreement are a crime. That is also why executives, not only the IT team, should understand and approve the engagement.

NIST also notes what makes a penetration test valuable: most tests look for combinations of weaknesses that together give an attacker more access than any single flaw would. A scanner might report ten medium-severity findings. A tester might chain three of them into full control of a server. That chain, not the individual finding, is what business leaders need to see.

Offensive and defensive security

Defensive security is the daily work of protecting the organization. It includes asset inventory, patching, secure configuration, identity and access management, multi-factor authentication, endpoint protection, network monitoring, logging, backup and incident response. Its purpose is to prevent attacks where possible, detect the ones that get through and recover quickly.

Offensive security tests those defenses from the attacker's side, and only ever as authorized, contracted testing under a written scope and rules of engagement. A penetration test is its most common form. A related practice, scenario-driven testing of detection and response, goes further and asks whether your people and tools notice an attack and react to it. NIST lists this among the things penetration testing can reveal: how well a system tolerates real-world attack patterns, how sophisticated an attacker would need to be, which additional countermeasures would help, and whether defenders can detect and respond appropriately.

The two sides work best together. The defenders learn how attackers actually behave and fix the specific gaps. The testers get context about how the organization is built. After a test, the useful question is not only what was found but also why the existing process did not find it first.

A penetration test is not a vulnerability scan

Executives often hear the terms used interchangeably. They are different tools. A vulnerability scan is largely automated. It checks systems against a database of known weaknesses and produces a list. NIST notes that scanners check only for the possible existence of a vulnerability, while the attack phase of a penetration test verifies a vulnerability by attempting to exploit it. Manual processes can find new or obscure issues that scanners miss, though they are slower.

The NCSC offers a helpful comparison: a penetration test should be thought of as similar to a financial audit. Your finance team tracks income and spending every day, and an outside auditor confirms that your processes are sound. Likewise, your team should scan and patch continuously, and a penetration test checks that this work is effective. The NCSC goes as far as saying that in an ideal world you should know what the testers will find before they find it. If a test uncovers a long list of basic issues, the real finding is that internal vulnerability management is not working yet.

Types of tests

Tests vary along a few lines, and your goal should choose among them.

  • External testing starts outside your perimeter, as an internet attacker would, using public information and scanning to find reachable systems and weaknesses.
  • Internal testing starts from inside the network. Testers are often given the access of an ordinary user and try to escalate privileges, up to administrator level, to see how far an intruder or a malicious insider could go.
  • Web application and API testing examines bespoke software for flaws such as broken access control and injection. It gives developers feedback on coding practices.
  • Open-box (transparent) testing gives the testers full information about the target. It checks that known vulnerabilities and misconfigurations are controlled. Closed-box (opaque) testing gives them none. It more closely models an unknown outsider, but the lack of information may leave issues undiscovered in the time available.
  • Scenario-driven testing explores a specific situation, such as a lost laptop, an unauthorized device plugged into the office network or a compromised server in a DMZ.
  • Social engineering tests how staff respond to phishing, phone pretexts or physical access attempts. NIST notes that penetration testing often includes non-technical methods.

How to scope a test

Scope is the contract between you and the testers. Weak scoping is the most common reason a test disappoints. The NCSC recommends that scoping involve the risk owners, technical staff who know the system, and a representative of the testing team. Risk owners explain their areas of special concern, technical staff describe the boundaries of the environment, and the testers recommend what testing will give a true picture.

The output should be a written plan covering the following.

  • The technical boundaries: which networks, applications, cloud accounts and sites are in scope, and which are explicitly out of scope
  • The types of testing expected and the amount of effort, usually given in tester-days
  • The timeframe, including any need for out-of-hours testing and any critical systems that need special handling
  • Scenarios or use cases to test, plus preparation such as test accounts
  • Compliance or legal requirements the test must satisfy, and any specific reporting requirements, such as including CVSS scores
  • Who can be reached at all times during the test, and how a critical finding is escalated

NIST SP 800-115 adds the rules of engagement. This document sets out the purpose and scope of the test, its assumptions and limitations, the risks and how the testers will mitigate them, the logistics and the people involved. It is where you decide, for example, whether testers may attempt denial of service, what they may do with data they reach, and when to stop. Both NIST and the NCSC acknowledge that testing cannot guarantee that no unexpected reaction will occur on a live system. NIST states that systems may be damaged or rendered inoperable during testing. Plan for that with backups, change windows and an agreed way to pause.

Scope can also change during a test. If testers find a connected system that was not included, they can either propose a scope change, which usually changes cost and timing, or record the exclusion as a limitation. The risk owner makes that decision, which is another reason a decision-maker should be reachable.

The phases of a test

NIST describes a four-stage methodology: planning, discovery, attack and reporting. Planning sets the rules and goals and gets approval. Discovery has two parts, which are gathering information and scanning, and then vulnerability analysis, comparing what was found with known weaknesses and the testers' own knowledge. The attack phase verifies likely vulnerabilities by trying to exploit them. Successful attacks often lead back to discovery, because each foothold reveals more to examine. Reporting runs alongside the other phases and concludes the engagement.

What a good report contains

The report is the product. Insist on seeing a sample before you buy. According to the NCSC, a test report should include the security issues uncovered, an assessment by the testers of the level of risk each one creates, a method for resolving each issue, an opinion on the accuracy of your own vulnerability assessment, and advice on improving your internal process. Reports are commonly accompanied by a debrief meeting where the testers walk through the findings and answer questions.

In practice, ask for these elements.

  • An executive summary in plain language that states the overall risk, what the testers achieved and what it would mean for the business
  • The scope, dates, method and limitations, so a reader knows what was and was not tested
  • Findings with a severity rating, evidence such as screenshots and steps to reproduce, the affected systems and the likely impact
  • A clear remediation recommendation for each finding, ranked by priority
  • Attack narratives that show how individual findings chained together
  • Positive observations, meaning controls that worked, because they tell you what to keep

On severity, the NCSC notes that testers commonly use the Common Vulnerability Scoring System, and that reports in its CHECK scheme must state risk as high, medium, low or informational. Whatever scale is used, any deviation from the standard rating should be justified, for example when other controls reduce the practical impact or when two issues together are worse than either alone.

NIST adds that results can be used as a reference point for corrective action, to define mitigation, and to track progress against security requirements, and it recommends a plan of action and milestones so that each vulnerability is addressed with specific, measurable steps. A report that sits in a drawer has no value. Assign each finding to an owner and a date, and have the testers confirm fixes with a retest.

How often to test

There is no single correct interval, and standards differ. The NCSC points out that a penetration test can only validate that your systems are not vulnerable to known issues on the day of the test, and that it is not uncommon for a year or more to elapse between tests. In that gap, new vulnerabilities and configuration changes accumulate. Treat that as a reason to test after significant change as well as on a schedule.

A practical approach is the following.

  • Test on a regular cycle, at least annually for systems that hold sensitive or regulated data or face the internet
  • Test after major change: a new application, a cloud migration, a merger, a new remote access method or a significant network redesign
  • Retest after remediation to confirm the fixes work
  • Check whether a regulation, contract or insurer sets a frequency for you. For example, HHS's proposed update to the HIPAA Security Rule, published in January 2025 and not yet final, would require penetration testing of relevant systems and asks for comment on a 12-month frequency. Customers and insurers may also set their own requirements.
  • Keep continuous scanning and patching between tests so that the test confirms rather than discovers

Questions to ask before you hire a tester

  • Who will actually perform the test, and what are their qualifications and experience with systems like ours?
  • What is the written scope and rules of engagement, and who signs them?
  • How will you protect our data and the findings, and how long do you keep them?
  • What happens if you find an active attacker or a critical issue during the test?
  • Can we see a sample report and speak to a reference?
  • Is retesting included?

The quality of a test is closely tied to the ability of the testers, as the NCSC stresses, because a test cannot be reduced to a fixed script. Clayton Cybernetics performs penetration testing and security assessments through its Cybersecurity (including penetration testing) service. Testing is always authorized and contracted: we scope each engagement in writing, test within the agreed rules of engagement, and deliver reports that executives and engineers can both act on.

Sources

This article is general information, not legal advice. Penetration testing must always be authorized in writing by the owner of the systems tested. Regulatory and contractual testing requirements vary; confirm yours with your compliance counsel. Last reviewed .

Keep reading

Need help with this?

We find out whether everything you run is safe from attackers, then we fix it.

Cybersecurity