How to measure security? NIST maps out the emerging field of IT metrology
Connecting state and local government leaders
A NIST computer scientist outlines the challenges and charts a course for future research in IT security metrics in a report that examines the state of this nascent art.
Information technology security is a hot topic, but attention usually focuses on the lack of it. What is missing is an objective, quantifiable way to effectively measure it.
“Security can be looked at in different ways by different people,” said Wayne Jansen, a computer scientist at the National Institute of Standards and Technology’s IT Laboratory. There is quality control for code developers, the process of deploying a system, and its maintenance by users. “These are all different aspects,” and they do not lend themselves to traditional methods of measurement used in physical science, he said.
Jansen has examined the status of efforts to develop security metrics, identified challenges and suggested a course for future research in a recent NIST report, "Directions in Security Metrics Research."
There have been a number of efforts to establish metric systems for security, including the international Common Criteria, the Defense Department’s Trusted Computer System Evaluation Criteria, the European Communities’ Information Technology Security Evaluation Criteria, and the International Systems Security Engineering Association’s Systems Security Engineering Capability Maturity Model.
“Each attempt has obtained only limited success,” Jansen wrote. “Compared with more mature scientific fields, IT metrology is still emerging.”
The issue is complicated because security means different things to different people and organizations. “Security is risk- and policy-dependent from an organizational perspective; the same platform populated with data at the same level of sensitivity, but from two different organizations, could be deemed adequate for one and inadequate for the other,” he wrote. “The implication is that establishing security metrics that could be used for meaningful system comparisons between organizations would be extremely difficult to achieve.”
There is no standardized terminology for discussing or describing security, Jansen said. The Federal Information Security Management Act's criteria for rating systems as low, medium or high impact is subjective, and assigning them numerical rankings can blur the distinction between qualitative and quantitative measures.
It is difficult to remove subjectivity from IT security. Security measures can be correctly implemented yet still not be effective. “Effectiveness requires ascertaining how well the security-enforcing components tie together and work synergistically, the consequences of any known or discovered vulnerabilities, and the usability of the system,” the report states. In other words, what is effective for one system might not be for another.
Are meaningful security metrics even achievable?
“The answer is yes,” Jansen said, “but they might not be as satisfying as you want.”
He identified two broad areas of research — process and organizational maturity — that focus on the care and maintenance of IT systems, and the intrinsic characteristics or properties of the systems. “I think we can make good progress on the maturity aspect,” he said. Research on security characteristics is not as far along.
There is not likely to be a single system of security metrics anytime soon because of the need to address different elements of security separately. Jansen cited the Federal Information Processing Standard 140 for cryptographic modules as a workable metric “because it bites off a manageable chunk.” The much broader Common Criteria, on the other hand, is less effective, he said.
“The issue of how to do this is going to be with us for the foreseeable future,” he said.
Challenges to effective security metrics identified in the report include:
- The lack of good estimators of system security.
- The entrenched reliance on subjective, human, qualitative input.
- The protracted and delusive means commonly used to obtain measurements.
- The dearth of understanding and insight into the composition of security mechanisms.
Promising lines of research for improved metrics include:
- Formal models of security measurement and metrics.
- Historical data collection and analysis.
- Artificial intelligence assessment techniques.
- Practicable concrete measurement methods.
- Intrinsically measurable components.
NEXT STORY: New cybersecurity services available to agencies