Updated April 13, 2025
Penetration testing serves as a foundational part of your cybersecurity strategy. With regular pen test exercises, you can identify and resolve vulnerabilities before hackers exploit them. However, not all pen test vendors deliver the same results. As a result, they may not be as effective as you expect.
If you find inconsistencies or identify issues post-test that the security firm didn’t, it may be time to evaluate your partner. In this post, we’ll define some red flags that are signs you should seek a new pen test vendor.
The Basics: What You Should Expect from Penetration Testing
Before discussing those potential red flags, let’s discuss pen testing basics. At its core, a pen test is a simulated cyberattack carried out by ethical (white hat) hackers. A third party conducts these, and many organizations specialize in them.
Testers employ the same tools, techniques, and strategies as real cybercriminals. They can mimic a variety of attacks, depending on what they’re testing, which could include:
- Web applications: Assesses your overall security and risk by testing the architecture, design, and configurations of web applications, which consists of anything delivered over the internet via a browser interface.
- Network security: Identifies the exploitable issues on networks associated with routers, switches, or network hosts. The approach uses weak assets or misconfigured assets to breach.
- Cloud security: Confirms your cloud deployment security is accurate and evaluates overall risk and the likelihood that a breach could occur in cloud properties. You can test public, private, and hybrid clouds.
- IoT security: This approach leverages layered methodology to analyze IoT devices and their interactions. Since hackers prefer these assets, assessing their security is critical.
- Social engineering: Uses phishing techniques to determine if your network can defend, detect, and react to these. It also gauges if employees are applying security training learnings.
In addition to what you can test, there are multiple methods:
- External testing: Testers target your visible assets (e.g., web applications, company website, email, and domain name servers) to attain access and extract data.
- Internal testing: This method occurs behind the firewall to simulate what could happen after an incident of human error, such as credentials stolen through phishing.
- Blind testing: A blind test only makes the tester’s target company name available. In this scenario, your IT team experiences a real-time perspective of an application assault.
- Double-blind testing: In a double-blind test, internal security teams don’t know it’s happening. Your personnel would have to respond immediately to the threat.
- Targeted testing: Testers and internal security professionals work together. This provides a training experience for your team and gives them feedback from the hacker’s perspective.
Finally, there are different access levels in pen testing:
- Black Box Penetration Testing (Opaque Box): Testers do not know the target system’s internal structure. They operate like real hackers and look for any weaknesses to exploit.
- Gray Box Penetration Testing (Semi-Opaque Box): Ethical hackers have some information about the target system. Often, they receive insights on data structure, code, or algorithms. The penetration strategy is different and may include specific test cases.
- White Box Penetration Testing (Transparent Box): The third option involves giving pen testers access to systems and artifacts. They may also be able to enter servers running the system.
In reviewing pen test types, methods, and access levels, you’ll see that pen tests have many layers and options. You may not need every kind of pen test available, and not all firms will offer all these approaches. That’s the first potential red flag. If you’ve hired them in the past only to do web application pen tests and now want to execute a cloud security one, you may run into an obstacle here. They might not have the expertise for this, leaving you vulnerable in the cloud.
More Red Flags: Proceed with Caution
Is it time for a new pen test vendor? These signs would point to yes:
1. They Only Use Automated Tools for Pen Tests
There’s nothing wrong with using automated tools in pen tests. They can be a great complement to human testers. Many organizations depend on them entirely to execute the test. The problem with relying solely on automation is that it is notorious for too many false positives and negatives. Scanning yields low observational findings, which are unlikely to be actionable. As a result, the accuracy of the tests falls into question.
It’s critical to ask providers how they pen test and what technology they employ. Again, automation can be a good start, but it will never equal the quality of human testers.
2. Their Experience Has Gaps
As discussed earlier, not all security firms can provide every type, method, or access level. Much of that has to do with their internal capabilities. Those testing need to have specific expertise and hold specific credentials like CISSP, CSSLP, OSWE, OSCP, ECSA, LPT (Master), CEH, etc.
These gaps may not have been apparent in an initial engagement. As you desire to go deeper and assess more parts of your cyber footprint, they may never have completed an IoT pen test, for example. If these assets are part of your network, testing their security posture is crucial. Such a need is very critical for medical device security. Threats are rising exponentially in IoT malware attacks, which hit 112 million in 2022.
In this case, along with others, your pen test vendor must be an expert in understanding the nuances. A standard pen test won’t accomplish your goals.
3. Complacency Can Create Blind Spots
Complacency is never a good thing in cybersecurity. The environment is too dynamic, with new threats occurring daily. Testers, however, can find themselves here, blind to vulnerabilities they overlooked. They aren’t purely inept. Rather, they’ve become comfortable about what they know and may not feel they need to brush up on skills. That puts you in danger.
One way to determine if that’s happening is to have a new potential vendor review the findings from your current provider. They won’t officially launch a new test, but they may point out some results they question.
4. Reports Are Confusing and Not Transparent
At the end of a pen test, you receive a report of what they found and recommendations for fixes. Ideally, the report should be concise and easy to understand without high levels of technical knowledge. It should also lay out the priority level for remediation and be an actionable list.
That’s the best-case scenario, but rarely what vendors deliver. Pen test reports can be overly complex and hard to understand. You’d need an “interpreter” to get any value from it.
Why do some vendors do this?
Several reasons can lead to this type of reporting. Some firms want to be too technical and complicated, so you have to depend solely on them for interpretation. Technical folks can be condescending and out of touch with what clients expect. If this is the exchange you’re receiving, it’s a big red flag.
5. They Don’t Provide RVTs
A remediation validation test (RVT) should be part of a firm’s deliverables. RVTs come after you’ve fixed the vulnerabilities identified in the pen test. It’s a final check to ensure you’ve resolved the issues properly. If your current provider doesn’t include this, you’ll have questions about remediation until they conduct the next round. During this time, you may be more susceptible to an attack.
6. They Aren’t Familiar with Regulatory Requirements
If you’ve been using pen tests to support regulatory requirements, your vendor must have experience with them. Some examples include:
HIPAA
HIPAA doesn’t explicitly require pen testing but does state in HIPAA Evaluation Standard § 164.308(a)(8) that a covered entity or business associate is required to “perform a periodic technical and nontechnical evaluation.”
A technical evaluation would be a pen test.
Information Access Management: § 164.308(a)(4)27 references the requirement to assess “security measures related to access control” and confirm how effective authentication practices are in preventing unauthorized access to PHI (protected health information) and other assets containing protected information.
NIST 800-66 for HIPAA cites this recommendation: “Conduct penetration testing (where trusted insiders attempt to compromise system security for the sole purpose of testing the effectiveness of security controls), if reasonable and appropriate.”
Thus, your pen test vendor must have deep knowledge of HIPAA and healthcare cybersecurity.
PCI DSS
PCI DSS (Payment Card Industry Data Security Standard) is an information security standard related to credit card information. The PCI SSC (Payment Card Industry Security Standards Council) oversees it, and card brands mandate it.
The standard includes four levels of compliance, depending on the transaction dollar amounts. All require a PCI scan.
SOC 2 Type 2
SOC 2 Type 2 is a System and Organization Controls (SOC) framework that provides a report about security, confidentiality, integrity, privacy, processing, and availability of data controls to organizations.
Any organization that uses or transmits protected data should undergo a SOC 2 Type 2 pen test. It applies to SaaS and tech companies that hold customer data. Conducting these ensures you comply with the scheme and have the correct security controls to safeguard data.
FDA Guidance for Medical Devices
FDA Cybersecurity Guidance for Medical Devices outlines the regulatory expectations for managing cybersecurity risks in connected and software-driven medical technologies. The guidance emphasizes the need to address security throughout the product lifecycle, including premarket submissions, design controls, software updates, and postmarket vulnerability management.
Any medical device connecting to a network, storing or transmitting electronic health information, or relying on software should comply with the FDA’s cybersecurity framework. This includes preparing a robust cybersecurity risk management plan, conducting threat modeling, maintaining a software bill of materials (SBOM), and validating security controls through testing. Adhering to this guidance ensures your device is not only regulatory-ready, but also equipped to protect patient safety and data integrity in today’s evolving threat landscape.
Is It Time to Find a New Pen Test Vendor?
If any of these red flags sound familiar, you should evaluate other options. Ask many questions about capabilities, expertise, certifications, and testing methods. Don’t let an ineffective vendor put you at greater risk.
Contact our pen test experts today to learn more about our solutions.
Penetration Testing FAQs
Please schedule a 30-minute Discovery Session with us so we can best understand your objectives.
Penetration testing, also known as security testing, should be conducted on a regular basis to ensure the protection of organizations' digital assets. It is generally recommended that all organizations schedule security testing at least once a year. However, it is essential to conduct additional assessments in the event of significant infrastructure changes, prior to important events such as product launches, mergers, or acquisitions.
For organizations with large IT estates, high volumes of personal and financial data processing, or strict compliance requirements, more frequent pen tests are strongly encouraged. Such organizations should consider conducting penetration testing with a higher frequency to continually assess and strengthen their security measures.
To further enhance security practices, organizations can adopt agile pen testing or continuous pen testing. Unlike traditional pen testing, which occurs at specific intervals, agile pen testing integrates regular testing into the software development lifecycle (SDLC). This approach ensures that security assessments are conducted consistently throughout the development process, aligning with the release schedule of new features. By doing so, organizations can proactively address any vulnerabilities and mitigate risks to customers, without significantly impacting product release cycles.
Penetration Testing as a Service (PTaaS) is a dynamic approach to cybersecurity where regular and systematic penetration tests are conducted to assess the security of an organization's IT infrastructure. Unlike traditional penetration testing, which is typically performed as a one-time assessment, PTaaS offers ongoing testing and monitoring, allowing for continuous identification and remediation of vulnerabilities.
Key aspects of PTaaS include:
Regular Testing Cycles: PTaaS involves conducting penetration tests at predetermined intervals, such as monthly or quarterly. This regularity ensures that new or previously undetected vulnerabilities are identified and addressed promptly.
Updated Threat Intelligence: As cyber threats evolve rapidly, PTaaS providers stay abreast of the latest threat landscapes. This ensures that each test is relevant and effective against the most current types of attacks.
Continuous Improvement: By receiving regular feedback and insights from these tests, organizations can continually improve their security postures. This process includes patching vulnerabilities, updating security policies, and enhancing defense mechanisms.
Comprehensive Reporting and Support: PTaaS typically includes detailed reporting on the findings of each test, along with expert recommendations for remediation. Ongoing support and consultation are often part of the service to help organizations respond effectively to identified issues.
Cost-Effectiveness and Budget Predictability: With an annual contract and monthly payment options, PTaaS allows organizations to budget more effectively for their cybersecurity needs, avoiding the potentially higher costs of one-off penetration tests.
Cloud penetration testing is a specialized and crucial process involving comprehensive security assessments on cloud and hybrid environments. It is crucial to address organizations' shared responsibility challenges while using cloud services. Identifying and addressing vulnerabilities ensures that critical assets are protected and not left exposed to potential threats.
Cloud penetration testing involves simulating real-world attacks to identify and exploit vulnerabilities within the cloud infrastructure, applications, or configurations. It goes beyond traditional security measures by specifically targeting cloud-specific risks and assessing the effectiveness of an organization's security controls in a cloud environment.
The importance of cloud penetration testing lies in its ability to uncover security weaknesses that might be overlooked during regular security audits. As organizations increasingly adopt cloud services, they share the responsibility of ensuring the security of their data and assets with the cloud service provider. This shared responsibility model often poses challenges regarding who is accountable for various security aspects.
Cloud penetration testing not only helps in understanding the level of security provided by the cloud service provider but also provides insights into potential weaknesses within an organization's configurations or applications. By proactively identifying these vulnerabilities, organizations can take necessary steps to mitigate risks and strengthen their security posture.
These terms refer to the amount of information shared with the testers beforehand. Black box testing is like a real-world hacker attack where the tester has no prior knowledge of the system. It's a true test of how an actual attack might unfold. Gray box testing is a mix, where some information is given - this can lead to a more focused testing process. White box testing is the most thorough, where testers have full knowledge of the infrastructure. It's like giving someone the blueprint of a building and asking them to find every possible way in. Each type offers different insights and is chosen based on the specific testing objectives.
When choosing a pen test provider, you'll want to consider several important factors to ensure your organization's highest level of cybersecurity.
Selecting the right pen test provider is crucial for your organization's security. It's about identifying vulnerabilities and having a partner who can help you remediate them effectively. To make an informed decision, here's what you should look for:
Expertise and Certifications: One of the key factors to consider is the expertise of the pen testers. Look for providers with a team of experts holding certifications such as CISSP (Certified Information Systems Security Professional), CSSLP (Certified Secure Software Life Cycle Professional), OSWE (Offensive Security Web Expert), OSCP (Offensive Security Certified Professional), CRTE (Certified Red Team Expert), CBBH (Certified Bug Bounty Hunter), CRTL (Certified Red Team Lead), and CARTP (Certified Azure Red Team Professional). These certifications demonstrate a high level of knowledge and competence in the field.
Comprehensive Testing Services: The cybersecurity landscape constantly evolves, and threats are becoming more sophisticated. To stay ahead, you need a provider with expertise and resources to test your systems comprehensively. Look for a pen test provider like Blue Goat Cyber that offers testing across various areas, including internal and external infrastructure, wireless networks, web applications, mobile applications, network builds, and configurations. This ensures a holistic evaluation of your organization's security posture.
Post-Test Care and Guidance: Identifying vulnerabilities is not enough; you need a partner who can help you address them effectively. Consider what happens after the testing phase. A reputable pen test provider should offer comprehensive post-test care, including actionable outputs, prioritized remediation guidance, and strategic security advice. This support is crucial for making long-term improvements to your cybersecurity posture.
Tangible Benefits: By choosing a pen test provider like Blue Goat Cyber, you ensure that you receive a comprehensive evaluation of your security posture. This extends to various areas, including internal and external infrastructure, wireless networks, web and mobile applications, network configurations, and more. The expertise and certifications of their team guarantee a thorough assessment.
We follow a seven phase methodology designed to maximize our efficiency, minimize risk, and provide complete and accurate results. The overarching seven phases of the methodology are:
- Planning and Preparation
- Reconnaissance / Discovery
- Vulnerability Enumeration / Analysis
- Initial Exploitation
- Expanding Foothold / Post-Exploitation
- Cleanup
- Report Generation
An External Black-Box Penetration Test, also known as a Black Box Test, primarily focuses on identifying vulnerabilities in external IT systems that external attackers could exploit. This testing approach aims to simulate real-world attack scenarios, mimicking the actions of adversaries without actual threats or risks.
During an External Black-Box Pen Test, ethical hackers attempt to exploit weaknesses in network security from an external perspective. This form of testing does not involve internal assessments, which means it may provide a limited scope of insights. However, it is crucial to note that the absence of identified external vulnerabilities does not guarantee complete security.
To gain a comprehensive understanding of the network's resilience, it is recommended to complement the External Black-Box Pen Test with an Internal Black-Box Penetration Test. By combining both approaches, organizations can evaluate the effectiveness of their security measures from both external and internal perspectives.
It is important to acknowledge that external-facing devices and services, such as email, web, VPN, cloud authentication, and cloud storage, are constantly exposed to potential attacks. Therefore, conducting an External Black-Box Pen Test becomes imperative to identify any weaknesses that could compromise the network's confidentiality, availability, or integrity.
Organizations should consider performing External and Internal Black-Box Penetration Tests to ensure a robust security posture. This comprehensive approach allows for a thorough assessment of external vulnerabilities while uncovering potential internal risks. Organizations can strengthen their security defenses by leveraging these testing methodologies and proactively addressing identified weaknesses.
Blue Goat Cyber employs a comprehensive approach to gather intelligence for a penetration test. We begin by actively seeking out relevant information about the targets. This includes identifying the devices, services, and applications the targets utilize. In addition, Blue Goat Cyber meticulously explores potential valid user accounts and executes various actions to uncover valuable data. By conducting this meticulous information-gathering process, Blue Goat Cyber ensures we comprehensively understand the target's infrastructure and potential vulnerabilities for a successful penetration test.
Compliance penetration testing is specially designed to meet the requirements of various regulatory standards. For SOC 2, it's about ensuring that a company's information security measures are in line with the principles set forth by the American Institute of CPAs. In the case of PCI DSS, it's specifically for businesses that handle cardholder information, where regular pen testing is mandated to protect against data breaches. For medical devices regulated by the FDA, pen testing ensures that the devices and their associated software are safe from cyber threats. This type of testing is crucial not just for meeting legal requirements but also for maintaining the trust of customers and stakeholders in industries where data sensitivity is paramount.
