The term "vulnerability assessment" holds varied meanings for individuals and lacks a precise definition in official regulatory bodies. However, a common interpretation suggests that a penetration test is typically conducted manually by a human adversary against an organisation within a defined scope.
When we first launched our external vulnerability scanner, we wrote this blog:
https://www.roboshadow.com/blog/certify-your-vulnerability-assessment-penetration-tests
The following is the continuation of our original blog to put a bit more meat on the bone around the nuances between vulnerability assessments and penetration tests.
Role of RoboShadow in Cybersecurity
A frequently asked question concerns the role of RoboShadow in conducting vulnerability assessments and penetration testing. RoboShadow encompasses both realms but leans more towards vulnerability assessments. While many online platforms claim to offer "automated penetration testing", we advocate that a manual or human component is needed in defining a proper penetration test.
Notably, at least 70% of a penetration test involves a vulnerability assessment, which effectively matches software and infrastructure attributes against global vulnerability databases. Where we think the "actual" Penetration Testing comes into the fray is when a penetration tester (human) tries to exploit vulnerabilities which are found as part of the scanning "reconnaissance" phase. However, the usefulness of the exploitation part is often called into question, which we discuss below in this blog.
Elements of Penetration Testing and RoboShadow's Features
The following highlights the core areas of the RoboShadow platform, which make up the core constituent parts of a vulnerability assessment/penetration test:
1. External Vulnerability Assessments:
These assessments target IPs exposed to the raw internet. They scan all 65,535 standard ports, report on the software behind them using fingerprint databases, and reference these findings against global vulnerability databases, primarily hosted in the United States. The RoboShadow technology combines industry-standard Open Source and proprietary technology.
Locking down all your external services which face the raw internet is a high priority in any network security exercise, not only for the apparent reason but also to keep out the organised crime groups who just routinely scan the internet IP ranges for vulnerabilities, fire AI-based bots and exploitation tools at random. To try their luck, they also build malware to do this from unsuspecting PCs all over the world.
Even nation-state actors will use tools like Shodan and Twitter to track brand-new exploits and people who might have these particular exploits (just waiting to be hacked). So, long story short, get your external IP addresses locked down tight.
2. Internal Device Vulnerability Assessments and Penetration Testing:
This modern approach, often termed 'zero trust testing' , evaluates devices for antivirus presence, operating system updates, application vulnerabilities, and configuration settings for firewalls and anti-ransomware.
Post-Windows 10, such assessments typically require agents, as scanning from the outside using domain administration credentials has become nearly impossible. The most common way you get hacked is via a user's desktop; it's a lot easier to get someone to click a link or socially engineer them than trying to brute force through a firewall.
3. Internal IP Network Scans:
This involves auditing internal device scans by scanning internal IP ranges, performing extensive port scans on discovered devices, and comparing the results to global vulnerability databases, similar to external penetration testing but with internal IP-connected devices.
This becomes important and a backstop, effectively, once a hacker gets access to one of your machines because Susan or Derek in accounts clicked on a link they shouldn't (and were working on a machine with known vulnerabilities); then they will scan the internal network for something that can get a "remote shell" on, this is usually an old printer or an out of date switch on the network. They do this because you will always assume a PC will be patched, so you need to find something you can hide in the network to launch more attacks.
4. Multi-Factor Authentication Assessment:
Penetration testing teams commonly audit environments like Google Workspace or Microsoft 365, ensuring each account is secured with multi-factor authentication. This also extends to verifying multi-factor authentication statuses on various organisation platforms. The RoboShadow platform will connect to your Office 365 / Azure AD environment and assess the multi-factor authentication posture within the 365 / Azure AD tenancy. Not having an MFA means you are very susceptible to Phishing style attacks, and this is not an excuse in this day and age.
5. Primary User Store Reconciliation:
This step involves reconciling all data against a primary user store (like on-premise Active Directory or Azure Active Directory) to identify targets for penetration tests or vulnerability assessments. Ensuring you have full coverage and can account for all your devices that access your data is a keen part of an internal vulnerability assessment/penetration test. It's the machines that are feral which will cause you to fail a penetration test (or worse, suffer an actual attack). Finding a good reconciliation point from your "primary user store" is the springboard needed to reconcile your risk.
What is not "In Scope" as part of a RoboShadow vulnerability assessment?
There are many other areas within a Penetration Test / Vulnerability assessment which are not currently available within the RoboShadow platform; these include but are not limited to:
1. Line of business application testing: whereby actual bespoke or line of business applications are penetration tested directly as an application suite. RoboShadow does have OWASP Zap Scanning, which has a lot of web application testing in its makeup; however, it would not be considered a complete SaaS / Line of Business application penetration test. We will suggest other National Cyber Security Centre Alumni companies that can help you do this.
2. WIFI Scanning: actually having someone physically come to your offices and scan your WIFI networks and see if they can be penetrated by using devices like the Pineapple WIFI hacking tools.
3. Dark Web Scans: although this is coming, the RoboShadow platform doesn't conduct assessments on the Dark Web to see if any of your accounts have been owned (this is coming soon, though).
4. Phishing Exercises: we don't currently conduct phishing exercises on the platform, whereby fake phishing scams are created to trick staff members into giving up their credentials. We can suggest platforms to use of this nature if you get stuck with options. This is also true for other types of “Social Engineering” tests, which are typically not available on security platforms as they often involve human adversary-type scenarios.
5. Mobile Device Scanning: The RoboShadow platform does not currently conduct mobile device scanning and security; many mobile device management platforms perform this role wonderfully that we can recommend you to.
The above list is not to discredit our excellent platform at all; our mission statement is to Demystify Cyber Security and level the gap for everyone. We can't do this without a sincere relationship with the truth; there is a lot of hype in cyber security and "over marketing", and we want to help people cut through the noise and protect themselves correctly.
Being part of the National Cyber Security Centre Alumni means we have a lot of contacts in various cyber technologies, and we would happily recommend you to other vendors for areas we do not cover.
Defining Vulnerability Assessment and Penetration Tests further
As discussed, the distinction between vulnerability assessments and penetration testing can be nuanced. Vulnerability assessments typically identify ports, devices, or services with known vulnerabilities. Penetration testers often exploit these vulnerabilities, which can be time-consuming to gain access to the target system. However, there's debate about charging for using known vulnerabilities, highlighting a controversial aspect of penetration testing.
This effectively can be money for old rope, but we want to stress that there is value in this "exploitation". However, the problem with this is one of budget and value. Take the following scenario:
Suppose you scan an external environment, and it reports a particular vulnerability is discovered. In that case, you can check the attack vector, quickly Google how to exploit the vulnerability online, and then watch a few YouTube videos. Now granted, actually gaining access to the system could take hours or even days in a detailed but fun "capture the flag" style approach; either way, the client can rest assured that they got something for their money (i.e. the Penetration Tester gained access), and the actual tester got to sharpen their arsenal with a new skill they learned online.
This is wonderful, with the only nuanced exception. If the vulnerability is reported, and it is well-known online how it is exploited (with clear instructions and guides), then should a client pay for the penetration tester to have fun with a cat-and-mouse game to prove a known vulnerability can be hacked in the way that lots of other people have hacked it before.
This would not be a problem if penetration testing were cheap, but they are not (there is a massive skills shortage). The problem harks back to "scope". If the penetration tester charges two days to exploit known exploits (tremendous but can be pointless), but then doesn't do Mobile testing or a Cloud Assessment, then this could mean the client spends money on paying for the penetration tester to play with exploits as opposed to adding more value across the tech stack.
We certainly do not want to be unpopular by making this point, but we must stand by our principles and try to lift the Cyber Market's lid and demystify it for everyone.
RoboShadow's Approach and Market Dynamics
For RoboShadow users or service providers, transparency with clients about what is scanned and tested is crucial. Explaining that a significant portion of penetration testing is, in fact, vulnerability assessment, without the exploitative component, helps maintain client trust and understanding.
The cyber insurance market also plays a role, as vulnerability assessments and penetration testing documentation can aid in obtaining better cyber insurance; though this varies by insurance vendor in terms, overall, they do seem to accept a well-placed vulnerability assessment.
Future of Vulnerability Assessment and Penetration Testing
The ongoing debate in the cybersecurity field is about defining and regulating the scope of these assessments. The details of the chosen scope often determine whether an activity is classified as a penetration test or a vulnerability assessment.
We always support our clients and customers seeking third-party scoping advice, echoing Warren Buffett's advice: "Don't ask the barber whether you need a haircut". That is, when companies are trying to charge you for a penetration test, you should question whether it's prudent to also let them determine the scope, given that they ultimately profit from the work.
Any Questions?
For more information on RoboShadow's capabilities and the dynamics between penetration testing and vulnerability assessments, feel free to get in touch. You can send us an email at hello@roboshadow.com.
Additionally, for our current users, there's a convenient 'Support' option within the RoboShadow console, ensuring you get timely and effective responses. We're here to help and ensure your experience with RoboShadow is seamless and beneficial.
I’m lucky to have worked in technology all over the world for large multi-national organisations, in recent years I have built technology brands and developed products to help make technology that bit easier for people to grasp and manage. By day I run tech businesses, by night (as soon as the kids have gone to bed) I write code and I love building Cyber Security technology.
