Why Do Bugs Suddenly Appear, Every Time....You Are Near?

An exploration of key differences between good and great penetration testers.

If you understood the Carpenters reference, congratulations.....You're old.

Introduction

The amount of hackers I've met at this point that have made me question my technical relevance in the security industry is crazy. There are some extremely talented individuals out there that have this innate ability to consistently identify and exploit beautiful bugs in the most hardened targets, which, even to those of us in the know seems like absolute wizardry.

To that point, I've been thinking a lot lately about the key differences between a good penetration tester, and a great one. What are the actual separators between the two? Is it strictly technical ability? If so, what are the metrics on which we measure that? Amount of critical vulnerabilities identified throughout their career? Amount of CVEs to their name? Their contributions to the wider security community i.e. conference talks, blog posts, research etc.?

To me, it's never been solely about the technical. The testers that continue to impress me the most are the ones that can hold their own against developers / IT staff AND the executives. Being able to bring an organisation to their digital demise, whilst educating developers / IT staff in the errors of their ways and converting the technical risk into business risk for decision makers. That's my idea of a great tester.

With that in mind, I wanted to explore several key traits that I think separate the penetration testers / security researchers I respect, from us common folk. Needless to say, this isn't an exhaustive list and I'm sure you could think of many more, it's just an observation of attributes I've identified in talented testers over the years.

For some unknown reason this is a controversial topic amongst the security community. The question is always posed "Do I need to know how to code to be a penetration tester" and the answer 98% of the time is something like "It depends, but probably not". To me it's a no brainer. Yes, you need to learn how to read and write code (mainly read though) to get ahead of the rest and be an asset for your company.

I can tell you now, I really dislike black box testing. I liken it to throwing spaghetti at a wall to see what sticks. For some people that's exciting, not for me though. Would you rather fuzz an endpoint for 20 minutes or just look at the code and live debug the application to see what it's doing? I know which one I'd prefer. The blog posts I love reading the most are about these super obscure business logic bugs that would never have been found without source code. Assetnote and Source Incite are great for that.

My advice? Go pick an Object-Oriented Programming language (any one) and do a course on it (if the course involves building web applications and APIs, even better). Once completed, you'll understand the OOP paradigm and will be able to pick up other languages instinctively as long as you stay within the same paradigm.

Note: This point is more towards the AppSec people.

Am I saying you had to have been a professional developer in a past life to be a great penetration tester? No (It helps though). But the people I look up to all understand the importance of understanding code flow within applications. Most of that comes from having built applications either personally or professionally. For fun, or for profit.

An ex-dev colleague of mine always used to say "How can you break something if you don't know how it's built?" which really resonated with me. And after seeing his pentest reports, I now believe that developers make the best web app testers (Although I know a few security professionals that would likely disagree with me). It really is just about shifting the mindset from making to breaking, all the fundamentals are already in place.

Just because an application / network has had security testing conducted on it before, doesn't mean it's not littered with bugs. Good testers sometimes fall into the trap of thinking something is secure because it's had a lot of eyes on it (I know I certainly do).

The greats seem to remove that bias from their head and instead approach it with an open mind. Even if XYZ tester / researcher has looked at it before. They also have a tendency to go significantly deeper in their testing than your average hacker, so that probably helps with the bias removal.

I really like this tweet from Louis around secure code review:

The biggest mistake I made was trying to be a master of multiple disciplines within the offensive security domain.

• Reverse Engineering;
• Mobile;
• Active Directory;
• Web Applications;

I wanted to be the best at all of them. What I quickly realised was that it was impossible to become a master of everything. I found myself having an ok understanding of everything but not really understanding anything in depth. A jack of all trades, master of none, if you will.

The greats pick something and rarely deviate. They become experts in one discipline and have passable knowledge in others.

Great testers are renowned for having an alternate way of looking at a problem. Which is often why they're so consistent in identifying issues in targets that have had a lot of eyes on them. They don't think like the normal tester. Like I said before, they go deeper.

How many times have you read a blog post where the researcher / tester chained three bugs together to gain RCE? (Absolute *chefs kiss* by the way). This often requires a huge amount of creativity.

Personally, I think a testers lateral thinking starts to evolve the better they become. You don't know what you don't know, so the more attack vectors a tester is exposed to, the better the chances of them thinking outside the box.

It's impossible to become an expert at something if you don't invest the time and effort. Michael Jordan didn't become Michael Jordan because he half-assed it. He became the best in the world because of his commitment to the game. Being a great tester is no different.

Learning new things is hard, it's not meant to be easy. Especially in this domain. That's why I think having an unrelenting passion for what you do and learn is vital to success. It takes a certain kind of person to persevere through the constant roadblocks and headaches that a foreign security topic has to offer you. The amount of times I've contemplated moving to Canada to snowboard after having sat through an x86 exploit development module that cooked my brain is frightening. I'm sure the greats have had similar thoughts at various points in their life. But much like me, I'm sure they couldn't see themselves doing anything else for a career.

Final Thoughts

We all know testers that are absolute wizards, either directly or indirectly. It's often helpful to take a step back and pinpoint the attributes that make them different. Why are they so good at what they do?

Often it's as simple as:

Passion + Time =  Extreme Competence

Work on your craft long enough and you'll soon find that people are trying to deconstruct why you're so good at what you do.