In the increasingly complex and rapidly changing landscape of cybersecurity, the decision to invest in a vendor or platform is one of the most critical challenges facing security leaders. The key to navigating this is moving beyond reliance on peer insights or industry hype and adopting a data-driven, requirements-based approach, while constantly adapting to emerging threats like automation and supply chain attacks.
We spoke with Richard Stiennon, Chief Research Analyst at IT-Harvest and author of Security Yearbook 2024, about his insights on vendor selection, the true value of analyst reports, and the evolution of network security concepts like Intrusion Detection Systems (IDS).
You can read the complete episode transcript here >
How Should Organizations Select a Cloud Security Vendor?
While many organizations select vendors based on peer recommendations, value-added resellers (VARs), or industry analyst endorsements, the most effective approach is to treat the process like buying a car: start with a full market view and your specific, data-backed requirements.
Common, Less Ideal Selection Methods
- Peer Insight: CISOs often look for peer insights, selecting a product if somebody they respect in a bigger organization uses it (e.g., “went with CrowdStrike, then let’s do that”). This is a surprisingly common practice.
- Reseller Influence: A value-added reseller may recommend a product they believe is best, though they likely selected it based on what is selling the best.
- Industry Analysts: Analysts may ask questions that elicit what the client is already leaning towards, often giving a “stamp of approval” to the client’s decision rather than offering a truly definitive choice, as their answer is typically “it depends”.
The Requirements-First Approach
- Define Requirements: Start by defining your requirements, even if you are unsure what they are initially. Determine what layers of defense you need to cover, such as network access control, workload protection, and identity protection (PAM, customer, user, or third-party).
- Evaluate Cloud-Native First: Before purchasing a third-party product, check if the cloud provider already offers the solution—it is often close to free. Third-party vendors often put a “nice front end” on the cloud provider’s offerings and charge a lot for better management.
- Proof of Concept (POC) is Easy, but Be Skeptical: POCs for cloud security products are often “mind-blowing” and can be set up in minutes, quickly revealing critical misconfigurations. Don’t immediately say “yes” to the first vendor; get at least two other quotes and try them out.
- Vendor Assessment: Evaluate the vendor’s viability and health. Consider their headquarters location (e.g., Chinese or Russian) and whether they are in the process of being acquired, which can introduce instability.
The Multi-Cloud Consideration
In a multi-cloud setup, the selection process changes because you generally cannot rely on one cloud provider to service the others. You lose the benefit of the native, integrated capabilities that the individual cloud providers offer.
- Tool Agnosticism: You must look for tools that can handle multiple environments, which most multi-cloud solutions do.
- Question the Value: Richard often questions the value of multi-cloud until later stages, suggesting that initial “multi-cloud” should be one cloud provider plus your own data center (doing bare-metal cloud stuff).
- Open Source: Small companies with technically savvy people often turn to open source for multi-cloud solutions.
- Native Capability Loss: You lose the benefit of the native capability that individual cloud providers offer.
Do Gartner’s Reports Actually Matter, and Can Vendors Buy Their Ranking?
Industry analyst reports, such as Gartner’s Magic Quadrant, hold significant weight, but their influence and integrity are often misunderstood.
- Ranking Integrity: Vendors absolutely cannot purchase their ranking in Magic Quadrants or other reports. Analysts are typically insulated from knowing how much a vendor spends. Gartner remains independent because vendors are only 10% of their revenue; the majority comes from selling advice to CISOs and executives.
- Purpose: Gartner customers tend to be late adapters who do not buy the latest and greatest. Gartner serves a good purpose by giving these CIOs and CISOs really good advice about what other large companies are doing, providing approaches and even pricing information.
- The Failing: The primary failing is that analysts give their opinions rather than the data needed for a client to make a decision themselves. They may miss smaller, regional, but perfectly viable vendors that would offer excellent customer support (e.g., a UTM vendor in Perth, Australia).
How Can Organizations Avoid Checkbox Buying and Overcoming Leadership Constraints?
A major mistake is checkbox buying—selecting a product simply to meet a compliance or customer requirement.
Avoiding Checkbox Buying
- Dual Purpose Tools: Ensure that the product, while fulfilling a compliance requirement (e.g., “we got logs”), actually makes you more secure. The tool should also give you more visibility, recoverability, and resilience.
- Use Real Frameworks: A mature organization will already have adopted a framework like NIST and will know what security gaps they have (e.g., lack of coverage in one of NIST’s 23 areas).
Dealing with Leadership Constraints
When leadership has non-security motivations (e.g., “we have to use CrowdStrike” because of a marketing partnership), security teams must work around the constraints.
- Pad the Solution: If forced to use a specific vendor (e.g., CrowdStrike), you can “pad it” by adding other solutions (e.g., using Defender to catch viruses, which is often already paid for) to beef up and work around the constraints.
- Desktop Exercises: Use tabletop exercises (desktop exercises for incident response) to find gaps in process and visibility. This is the most important thing you can do, and it can save you so much during an actual incident.
What Implementation and Vendor Management Practices are Essential?
Contracting and Phased Rollouts
- Payment Terms: Try to include terms in the contract that you will not pay the vendor until the product is up and running. This incentivizes the vendor to prioritize implementation.
- Training & Support: Ensure that professional services, training, and the ability to extend support are included in the contract, though cloud security often needs very little configuration.
- Phased Rollout: Do not use a “big bang” approach. Use phases of implementation and roll out endpoint solutions incrementally (e.g., on IT people’s desktops first). This allows you to learn and prepare users for the change.
Zero Trust for Updates
This is an application of Zero Trust: Do not trust the vendor to have perfect updates.
- SaaS Risk: With SaaS solutions, the vendor can change the solution all day without telling you.
- Build in Delays: Do not have a process where the vendor can auto-update software (especially agents on your endpoints) and you just accept it. Build in delays (e.g., 24-hour delay) to let other organizations do the initial testing for you.
- Testing: Organizations with good testing processes would have discovered the issue and not implemented the update.
Where Should Organizations Start Their Zero Trust Journey?
Zero Trust is quickly becoming the new normal in cybersecurity. While it is a little more “ephemeral” than a layered defense model, it is a framework that helps organize security thought.
- The Engineer’s Mindset: An engineer (like Richard) thinks in terms of a layered defense model: stop attackers, identify what they will attack (endpoint, network, data), and secure each layer.
- Starting Point: You can adopt a Zero Trust framework to guide your existing work and shoehorn it into the model.
- Graduated Trust: Zero Trust is more accurately “graduated trust” or “dynamic trust”. You give more trust based on credentials, and less trust based on context (e.g., logging in from China).
Is Intrusion Detection System (IDS) Dead? The Evolution of Network Security
Richard Stiennon is known for his 2003 pronouncement that IDS was worthless. His argument was rooted in the lack of action taken on the millions of alerts generated.
- The Problem with IDS: IDS was designed to look at stuff that does get in by generating alerts when a signature of bad stuff was seen. However, in two years at Gartner, Richard never met one team that had 24x7 coverage for IDS—they were capturing alerts but not doing anything with them.
- The IPS Solution (Action vs. Alert): If you have a signature for an attack, why create an alert when you can stop those packets and shut off the connection?. This ability to stop the packets is Intrusion Prevention System (IPS).
- The Legacy: The market for selling IDS evaporated. This created two new industries:
- MSSPs (Managed Security Service Providers): A way to outsource the problem of ignoring logs to somebody else.
- SIEM (Security Information and Event Management): A place to store alerts (data management platform) that people can also ignore. The ultimate takeaway is that action versus alert is the key: if you’re not taking action, the system is pointless.
What are the Most Critical Emerging Threats, and How Should CISOs Handle Burnout?
Emerging Threats: Automation
The most critical emerging threat is more automation on the part of the attackers.
- Speed: Attackers will not take weeks to break into systems; they will take minutes using AI tasked with picking from a library of exploits. Two minutes later, they could have everything they were after from a critical resource.
- Defense: Organizations must start being prepared to turn on automation now, pushing the boundaries of SOAR solutions. They must become comfortable with a tool using AI to reset a TCP/IP connection or shut off an API call, as debugging that is less onerous than dealing with a mass data theft.
Burnout for Security Leaders
With 73% of CISOs and security leaders reporting burnout, Richard’s advice is to think big picture and step out of the current situation.
- Perspective: Don’t let a specific thing that’s happening consume you. Have a backup plan and know that it’s not the end of the world for you personally.
- Preparation: You should not be making new decisions during an emergency. Pre-planning (e.g., knowing who to call at the FBI or SEC) and practicing/rehearsing the incident response plan through tabletop exercises will save so much stress.
- Leadership Role: A leader’s job is to get the resources so the team isn’t stressed or burned out.
Final Conclusion: The Path to a Resilient Security Posture
The modern security landscape demands that leaders abandon outdated practices like checkbox buying and embrace a data-driven approach to vendor selection. Resiliency is not achieved by trusting a single analyst opinion or relying on automatic updates. It is achieved through pragmatic Zero Trust—using real requirements, vetting vendor health, enforcing phased rollouts, and building delays into software updates. As attackers increasingly leverage automation, security teams must prioritize SOAR capabilities and accept that rapid, automated response is the only defense against minute-scale breaches. Ultimately, the ability to think big picture and be meticulously prepared is the key to minimizing stress and navigating constant change.
