Author: Dr. Philip Cao

For the past decade, I have had this notion that there must be a Cybersecurity Canon: a list of must-read books where the content is timeless, genuinely represents an aspect of the community that is true and precise and that, if not read, leaves a hole in cybersecurity professional’s education. I’ll be presenting on this topic at RSA 2014, and between now and then, I’d like to discuss a few of my early candidates for inclusion. I love a good argument, so feel free to let me know what you think.

Security Metrics: Replacing Fear, Uncertainty and Doubt (2007) by Andrew Jaquith

I have been interested in cybersecurity metrics and how to visualize them since before we were connecting the Internet with strings and soup cans. In 2011, I had been looking for somebody to put some rigor to the idea when I stumbled upon a strong, positive review of Andrew Jaquith’s book on Amazon. A little more digging told me this was a book I really should check out.

From the beginning, Jaquith attacks the security community’s sacred cow of applying annualized loss expectancy (ALE) to convince management that the security program it is paying for is working. I have to say that I loved this attack. I remember first learning about ALE when I was studying for the Certified Information Systems Security Professional (CISSP) exam back in the day. I thought then that ALE sounded well and good when you said it fast, but in reality, you were just making up the numbers to plug into a formula that sounded scientific.

According to Jaquith, and most every CISSP preparatory exam book on the planet, “ALE is the monetary loss that can be expected for an asset due to a risk over a 1-year period and is calculated by multiplying the single loss expectancy (SLE) by the annualized rate of occurrence (ARO).”

Doesn’t that sound precise and mathematical? Indeed it does. But it turns out that there are lots of problems with this formula. The biggest problem is that we don’t know what the probabilities are. How can we possibly know what the probability is that an advanced-persistent-threat-style attack will compromise the computer that your chief of counsel’s secretary uses? This is not the insurance industry; we do not have actuary tables derived from decades of data collection that can tell us precisely what these adversaries will do, how often they will do it and how much it will cost us when they do it.

So what, Jaquith and others have asked, do ALE practitioners do in the absence of hard data? They guess. They estimate. They fudge. And when they do this, they undermine the veracity of the very process that they are trying to convince management is so exacting. What good is a scientific formula if all you do is fill it with garbage data?

Jaquith’s thesis is that, instead of using imprecise models like ALE, security professionals should use metrics instead. He says that “[this change in thinking] requires practitioners to think about security in the same way that other disciplines do – as activities that can be named, and whose efficiencies can be measured with key indicators.”

Coincidentally, the first time I read Jaquith’s book, I just happened to listen to the Patrick Gray Risky Business podcast from April 2011 where he interviewed Brian Snow. Snow is a former NSA information assurance technical director, and he had a lot to say then about the folly of using probabilistic risk assessments, like ALE, to improve the cost-effectiveness of securing nuclear facilitates and government information assurance programs.

Snow made the point that these models are fine for standard risks that routinely occur—like what is the mean time to failure of the hard drive in your laptop—but that they fail miserably when trying to predict cases that have high impact to an organization but are not likely to occur. These cases that Snow referred to are called “black swan events.”

Black Swan Events

The “black swan event” term was made famous by Nassim Nicholas Taleb in his 2007 book “The Black Swan: The Impact of the Highly Improbable.” For some organizations, computer breaches are black swan events that Taleb describes as “outliers that carry extreme impact.” They are outliers because the chances of something like that happening to your network are pretty small, but when it does, the cost to your organization is extreme.

Jaquith’s solution is to “… quantify, classify, and measure information security operations in a modern enterprise environment” and to provide “… a set of key indicators that tell customers how healthy their security operations are.”

He spends a good portion of his book, two entire chapters actually, explaining what some of these metrics might be. Your organization might not have a use for all of them, but you will appreciate the thoroughness that Jaquith uses to explain why they should be considered.

As a bonus, he spends a chapter reviewing the fundamentals of statistics. If you are like me and slept through your probability and statistics course in college, you will welcome this refresher. Jaquith’s simple explanation alone about what a standard deviation is and what correlation really means is worth the price of admission.

As an extra bonus, he spends a chapter on visualization. I am a fan of Dr. Edward Tufte, who is in my opinion the world’s leading expert on how to visually display complex data. Tufte devotees will learn nothing new here but will appreciate how Jaquith reduces Tufte’s four seminal books on the subject to six rules:

• It’s about the data, not the design
• Just say no to three-dimensional graphics and cutesy chart junk
• Don’t go off to meet the (Microsoft) wizard
• Erase, erase, erase.
• Reconsider Technicolor
• Label honestly and without contortions

The only real fault I have with the book is the last chapter, “Designing Security Scorecards.” Here, Jaquith had the opportunity to show some practical security dashboards that perhaps some real organization used and found useful. Instead, he spends the entire chapter explaining what goes into making a scorecard.

As I got closer to the end of the book, I just knew that I was going to see some dazzling examples that I might use in my own organization. When I turned to the last page and found nothing but the index, I was dumbfounded. He provided no examples of real-world security dashboards. D’oh! So close to being perfect!

Why It’s Worth It

That one caveat aside, Jaquith’s book is well worth the read. I recommend it highly. I dare you to get to the end of that book without learning something that will help you in your current job, and even if security metrics are not your thing, then statistics and visualization will make you a more well-rounded business person.

But for you security professionals out there, this book is for you. It will help you unshackle yourself from the chains of probabilistic risk assessments. It will turn you away from the dark side and toward a more meaningful process to assess your enterprise’s security. You should have read this by now.

Rick Howard

For the past decade, I have had this notion that there must be a Cybersecurity Canon: a list of must-read books where the content is timeless, genuinely represents an aspect of the community that is true and precise and that, if not read, leaves a hole in cybersecurity professional’s education. I’ll be presenting on this topic at RSA 2014, and between now and then, I’d like to discuss a few of my early candidates for inclusion. I love a good argument, so feel free to let me know what you think.

Breakpoint (2007) by Richard Clarke

In an earlier entry I looked at Richard Clarke’s Cyber War, and this time around I’ll look at how Clarke jams a boatload of cutting edge cybersecurity ideas into this slim, Michael Crichton-esque political thriller. Clarke wrote it in 2007 but set it in the near future of 2012 and when I say there is a boatload of information, I am talking about yacht-sized, not dingy-sized.

The bad guys in this novel execute most of the cyber fantasy attacks against the United States that any group of cybersecurity geeks (including myself) could conjure up after a few beers sitting around a bar at the annual Black Hat / DEFCON conventions in Vegas (incidentally, one of the settings in the book).

Clarke gives us bombings of US beachhead routers on both coasts that reduce inbound and outbound internet traffic to just 10 percent, buffer overflow attacks against a communications satellite that sends it reeling out to space, SCADA attacks that blow up a research institution with a live nuclear reactor and a well-coordinated SCADA attack that takes out all power west of the Mississippi. Of course, in the novel, US government leadership, specifically the Intelligence Community (IC), thinks the Chinese are behind everything and they put all of their efforts into proving it.

All of these “fantasy” attacks are quite possible in the real world and the cybersecurity community has been talking about them for at least the last decade. At Palo Alto Networks, for example, we spend a lot of time looking at SCADA security and the challenges in securing such systems. (One of our experts, Del Rodillas, is speaking on the topic at an ISC-ISAC event on January 22.)

Clarke definitely knows the landscape. Before he retired from government service, he served three different Presidents as the Special Assistant to the President for Global Affairs, the National Coordinator for Security and Counterterrorism and the Special Advisor to the President for Cybersecurity. The political theory behind these acts is known as Escalation Dominance. It is the idea that China, or any government really, would launch some kind of attack against the US that would hurt the country in an effort to prove that they could launch a much larger attack that would really hurt if the US did something that the opposing government did not like.

Why Read It

In the afterword to this novel, Clarke said that it was easier to talk about these issues in a fictional setting then it was to talk about them in dry, academic and political journals. I concur – and that’s one reason why I’ve included novels and “lighter” books in my selections the Cybersecurity Canon. The truth is that many of these things are much more exciting and frightening when splashed across the fictional page.

This is a good read. Clarke’s story races across 10 days in March of 2012 as our heroes, Susan Connor – an agent for the Intelligence Analysis Center (IAC) – and Jim Foley – an ex-marine on loan to the IAC from the NYPD — try to out-think the US Intelligence Apparatus and Law Enforcement community and track down the real culprits behind the Internet attacks. Critics have taken Clarke to task for his wooden characters in the story, but I found that not to be true. I liked his portrayal of the misguided Internet billionaire especially and I liked the way he portrays New York and Boston cops.

The bottom line here is that this is book is a fun political thriller that gets the cybersecurity stuff right. I recommend it.

Rick Howard