Is there a number for that?

"When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge of it is of a meager and unsatisfactory kind" --- Sir William Thompson, Lord Kelvin

We live in a society obsessed with numeracy but work in a field where meaningful numbers are hard to come by. It is particularly challenging for security professionals because a perfectly effective security program produces no visible results as nothing untoward happens. There are no virus outbreaks, no hacker intrusions, no disclosures of confidential information and business pretty much just runs the way it's supposed to.

This leaves the Chief Information Security Officer (CISO) in a quandary when she approaches the corporate coffers with a budget request only to find herself in the position of the fellow who religiously beat a gong for two hours outside his suburban home at 6:00AM every morning much to the irritation of his neighbors. When local law enforcement arrived to inquire regarding his reasons for disturbing the peace in this raucous fashion, he replied that he was keeping tigers away. The puzzled patrolman replied that "There aren't any tigers around here" only to be met with a beatific smile and "See how well it's working!!"

Equally troublesome issues arise when the CISO looks at her own operations to assess effectiveness, proper allocation of resources and needs for new technologies. Without a realistic way to measure effectiveness, she is left with vague (and largely indefensible) notions of how well the program is working and where efforts should be focused next. The subject of this month's review offers a roadmap for moving from a "meager and unsatisfactory" knowledge of our security efforts to one backed by meaningful metrics.

Your humble correspondent admits to an instinctive cringe when anything crosses his desk with the words "complete guide" in its title but within its 800+ pages and some 900 metrics, this book makes a noble effort at surveying the entire landscape of security and privacy metrics.

Laudably and immediately , Dr. Herrmann points out in her introduction a glaring limitation to many books on IT Security: it doesn't function in a vacuum and must embrace the related domains of physical, personnel and operational security. Embracing that single insight would better many organizations' efforts at securing their information. The introduction continues to map out her approach to organizing the metrics into compliance, resilience and ROI metrics culminating in a 4-page table that organizes the 900+ metrics into those categories (with numeric designations that are carried though in later chapters to the individual metrics). This table provides an excellent starting point for reading the book and also allows quick identification of the particular sections most relevant to a reader's interests.

However, under no circumstances should you skip chapter 2, which provides valuable directions on how measurement works and what goes into producing a useful metric. The discussion covers such basics as the measurement scales (nominal, ordinal, etc), the types of operations appropriate to measurements on each scale and the characteristics of a useful metric (accuracy, precision, validity and correctness). Paying attention to these matters will help avoid the more egregious errors such as those inflicted on us by the marketing departments of the various security product vendors. She introduces Victor Basili's useful GQM (Goal, Question, Metric) approach which will appear again in the following two chapters. The chapter concludes with a useful survey of terms and their meanings as they will be used throughout the book.

On the down side, this chapter is replete with acronyms. Most are decoded in the glossary, but a wise reader will keep a cheat sheet of their meanings and avoid a great deal of flipping back and forth.

The next three chapters form the real meat of the book. Each chapter follows the form of an introductory exposition on the topic (Compliance, Resilience or ROI), the chapter's overall GQM (except for chapter 4) and then an in depth development of the metrics for that topic. Metrics are not just listed in endless tables but are developed within the framework of the discussion so that there is little question as to where that metric came from or its relevance to the topic.

Chapter 3 on compliance metrics contains one of the most complete and concise expositions of the bewildering landscape of legal and regulatory mandates (and their implications) that I have ever encountered. If we awarded meritorious service medals for authors then I think Dr. Herrmann deserves one for wading through the arid text of the laws and regulations to distill their meaning into something understandable and usable by mere mortals. However, an evolving regulatory landscape may rapidly date the details with the value remaining in how the approach developed meaningful metrics to match the current regulatory environment.

Chapter 4 addresses measuring resiliency (the ability of an infrastructure to maintain essential services and protect assets while repelling attacks and minimizing loss of integrity) across all four domains and develops a solid basis for each metric as it is developed. Again, the user of these metrics will have a clear understanding of their relevance and the reasons for their choice in assessing a particular program.

Chapter 5 addresses the thorny issue of ROI for security investments and is the shorter of the "meat" chapters. Herrmann immediately makes the important point that ROI does not always equate to increasing profit but often needs to measure increased efficiency, reduction/avoidance of costs and prevention of losses. She provides clear guidance on troubling issues such as how to merge best/worst case estimates into a single number when evaluating costs, asset values and losses. An added bonus is that the illustrations are in the nature of tables that can be easily customized and mapped into a spreadsheet application for immediate use.

In summary, this is not a book that you'll be likely to read from cover to cover, but when faced with the necessity of developing a metrics program to measure the effectiveness of some aspect of your security efforts, this rather imposing tome is one I would heartily recommend as a way to jumpstart your efforts. The master table in the introduction provides a quick guide to the particular section most relevant to the reader's needs but I earnestly recommend that the first two chapters ("Introduction" and "The Whats and Whys of Metrics") be reviewed before diving immediately into the details.

Richard Austin recently retired from his position as storage security architect for a Fortune 25 company but his curmudgeonly nature has survived the transition and will serve him well in his future efforts as a university instructor and consultant. He can be reached at rda7838@kennesaw.edu and welcomes comments on this review as well as suggestions for future reviews.