A Discussion on the Number of Vulnerabilities as a MetricA common security metric is a count of the number of vulnerabilities found.
Many organizations measure it and many say that it is a common mistake as a measure of security effectiveness. Sometimes -- this is a necessary metric. But most times it is indeed a mistake. Why? Because there is no business context.
We will show how not to use this metric.
And show how to use it with four examples:
- Metric 1: Cumulative vulnerability counts with the types of vulnerabilities, over time
- Metric 2: Cumulative vulnerability counts with the types of vulnerabilities, stand-alone
- Metric 3: Vulnerability counts with how the vulnerability was found, over time
- Metric 4: Defect Removal Efficiency, over time
When this Metric Does Not Work
Consider these ambiguities.
- What would a time series graph of the number of vulnerabilities found demonstrate?
- An ability to mine for vulnerabilities?
- What happens when it goes up? Celebrate that you can find issues?
- Or when it goes down? Does it mean you are losing the ability to find vulnerabilities? Or that developers are improving?
In most situations, this metric is a mistake. But there is one situation when this might work. When you are in the very beginning of a security turnaround.
When it Works
A Turnaround Situation
What is a turnaround?
“STARS” is an acronym for the five common situations leaders move into: start-up, turnaround, accelerated growth, realignment, and sustaining success. Thus, the model outlines the challenges of launching a venture or project; saving a business or initiative that’s in serious trouble; dealing with rapid expansion; reenergizing a once-leading company that’s now facing problems; and following in the footsteps of a highly regarded leader with a strong legacy of success.1
Examples of turnarounds
- Microsoft in the year 2000
- A product that has had security neglect either from the very beginning or for years. No active security testing and customers were testing the product for the software vendor by reporting security issues. You were hired to not only perform a turnaround, but also start-up a security program. This hits two categories of "STARS"
How to Use it
There are limited situations when vulnerability counts can be used as a metric. You can't use it stand-alone and its use is generally limited to turnaround situations.
In a turnaround situation, you need to demonstrate, as part of a larger story, that the security program is working.
Metric 1: Cumulative vulnerability counts with the types of vulnerabilities, over time
|Metric 1: Cumulative vulnerability count with vulnerability type over time|
What story does this tell?
- Identifies systemic and persistent vulnerabilities for remediation investment. In this case, the immediate focus area should be in resolving a systemic and persistent problem around "cwe-79," also known as Cross-site Scripting. It should be obvious that all vulnerabilities entered into a bug tracking system should be tagged with a standard Common Weakness Enumeration (CWE) identifier.
- Historical view of the types of vulnerabilities that have appeared in the system under test. No particular executive action to be taken here, other than informational. Demonstrates the system is clearly not free from vulnerabilities -- it would be good to compare this to the overall population of software defects. At best, this would pull heads from the sand that the system does have issues requiring attention.
Be sure to convert the CWE identifiers to a common name that the Executive may understand. Please add comments to this page for corresponding terms useful for Executives.
|CWE Identifier||Layman's Term|
|cwe-79: Cross-site Scripting||Missing sanitization allowing script injection|
|cwe-89: SQL Injection||Missing sanitization allowing db script injection|
|cwe-352: Cross-site Request Forgery||Missing transaction authenticity check|
|cwe-287: Missing Authentication||Missing authentication check|
|cwe-285: Improper Authorization||Missing authorization check|
Metric 2: Cumulative vulnerability counts with the types of vulnerabilities, stand-aloneCatch attention - wake up Executives with a big number
See the Wall Street Journal's Number of the Week column for examples on how to apply this concept to other security metrics.
Follow-up with context
|Metric 2: Cumulative vulnerability count by types of vulnerabilities|
What story does this tell?
- Identifies systemic and persistent vulnerabilities for remediation investment. A time element as seen in Metric 1, is not always needed. This shows the bulk of problems in the application is from "script injection." It's likely that spot-fixing the problems will not solve the problem and that a larger framework solution is required to stop this class of vulnerabilities. After all, 62% of all vulnerabilties found are related to this vulnerability class.
Metric 3: Vulnerability counts with how the vulnerability was found, over time
What story does this tell?
- Our customers tested our products for us. To an Executive, it clearly shows that in 2010, all reported security issues were from customers. The company's security testing was extremely inadequate.
- A turnaround is in effect. That's the point of this entire article. To demonstrate you are righting the ship. Security defects leaking to customers are disappearing and the other detection methods you have brought in are beginning to show promise in 2011 and have been a break-through in 2012 and 2013.
Metric 4: Defect Removal Efficiency, over time
Defect Removal Efficiency measures your ability to detect a defect before it gets to your customers. What's a good number? Depends on your industry and the criticality of your product. A common number to beat is between 85% and 95% for medium risk systems. Obviously, if this is medical software or military grade software, you probably want to be near 100%.
|Metric 4: Defect Removal Efficiency|
Depending on your goal percentage (85-95% is a common goal -- see Capers Jones for benchmarks by industry), similar to Metric 3, this shows how far off your organization is from your peers.
0% says a great deal -- all vulnerabilities were detected by customers - this is terrible for your organization's reputation. This also easily shows how your security program has turned things around. Evidence that a security turnaround is in effect.
Defect Removal Efficiency (DRE) =
(Total # of Third Party-reported Issues) / (Grand Total of Issues)
Corresponding Data Table
Product Release Dateb
Internal - Dynamic Analysis
Internal - Static Analysis
Defect Removal Efficiency
- Watkins, Michael D. "January 2009." Picking the Right Transition Strategy. Harvard Business Review, Jan. 2009. Web. 07 Sept. 2013.
- MITRE's Common Weakness Enumeration: http://cwe.mitre.org/
- Number of the Week: http://blogs.wsj.com/economics/category/number-of-the-week/
- Capers Jones' Applied Software Measurement book: http://www.mcgraw-hill.com.au/html/9780071502443.html