Pentesters and Decision Making

So my first post of the year is going to be non-technical and just a little bit of a rant. As the infosec industry matures, more companies have entered the market selling things from security appliances to SaaS platforms that do lots of fancy things. I worry that the high demand for services and infosec people is driving the idea of a "penetration test" much closer to the idea of "vulnerability scan" considering little if any post exploitation and that a lot of the value that can be gained from a high-value penetration test(er) may be lost due to ugly market pressures.

Depending on what industry or company you work for you might use different terms to describe what I think of as a penetration test:

  • If you were from a military / DoD background, you were likely to call it red-teaming or tiger-teaming
  • If you were commercial industry, you'd call it ethical hacking or penetration testing
  • If you were one of the audit companies, it might come under the terms risk assessment or vulnerability assessment

I like to think that value of any delivered service in the infosec world is the information produced and how it helps an organization drive EFFECTIVE strategic security decision making. To utilize any infosec service or product there needs to be a very clear definition of the activities and objectives. With a solid understanding of the how and what you can accomplish with a product, it becomes much easier to decide when or if you need it and how best to utilize it.

Having a clearer understanding of what these terms should mean and what sort of scope of activity each should include is an important thing. I can't help but wonder how many security services are sold to customers with the details of the work to be performed purposefully obfuscated behind technical buzzwords just get a contract signed.


Vulnerabilty Assessment

Vulnerability assessments are essentially vulnerability scans with varying levels of analysis. In practice this means you define your scope, run a scan, and review the results. The last step might include various additional pieces such as manual verification of the discovered vulnerabilities or a review and modification of the risk rating presented by the scanning solution.

There seems to be a real issue happening in the VA space.  It appears that traditional vulnerability scanning is dying, largely due to authenticated scanning. With these changes, the results of any scan and the resulting vulnerability assessment essentially becomes a patch gap analysis. and how often do you use anything that shows up on a VA to hack a network?

Penetration Test

If you read publications from organizations like NIST you get a definition of penetration testing that leaves too much open to interpretation to ensure a quality deliverable. Of all the definitions I could find in search engine results I really like the SANS definition of a high-value penetration test. The idea comes down to working with a client to model threat scenarios involving a hacker, discovering vulnerabilities, and exploiting them to achieve a pre-defined objective. The penetration test should only model things that have fair representation in the real-breach space. Can this include running a vulnerability scan? Yes. Sometimes. I do find valuable information for a penetration test from Nessus results but just as often I don't run a vulnerability scan at all.


To me, red-teaming involves an experienced team that can replicate long-term attacks by highly-resourced adversaries attempting to breach protected network resources. Where a penetration test involves more modeling the scenario the red-team engagement more closely mimics the full real-world social, physical, network and application attacks. In my experience there are a very limited number of organizations with a mature enough infosec program to warrant this type of engagement.

There will never be a CVE for this

So why bring all this up? The penetration test gains the most value in identifying those problems which might not even be 'vulnerabilities' in the first place. These are the "forever-vulnerabilities" live in a little sweet spot of "it's not a patch, and no one's at fault" but can still be leveraged to gain access to a resource. It's almost hard to call them vulnerabilities in the way some people frame the word. They are security issues inherent in the architecture of a system and the network it's attached to. When a business process demands that a group of laptop users is given local administrator privileges on their system and I abuse that to get shell the definition of that 'vulnerability' becomes fuzzy.

It's like saying an airplane is vulnerable to crashing while driving on the interstate.
well...yeah...I guess it is but if you want to drive it on the interstate you need to make some different design decisions.

So I think that maybe is where some value of a penetration test and pentester is lost. The real answer is in the business process itself being analyzed. How do you use the airplane? Does it need to drive on the interstate? Can we make it smaller? What is your flexibility in only using it on large empty interstates?

A Model

So where do these things fall in providing value to an organization?

  • VA is necessary but after a while you being to lose ground on the VA value proposition
    • Your increased security per dollar ratio starts shifting
  • You need pen-testing to get you past the basic hygiene issues
  • When do you make the argument to go to full red-teaming?
    • Maybe in the case where you kill the pen-tests every time
    • You built up your response team and you want to test them every once in awhile.