Bad stats sink cyber crime costs claims

Microsoft researchers find that estimates of damages caused by cyber crime are wildly inflated -- and increase the danger

If you follow computer security and have a good memory, you might remember a story from early 2009 that claimed cyber crime costs businesses as much as $1 trillion in just one year -- that's "trillion" with a "t." The version I saw was by Cnet writer Elinor Mills, whom I've always considered quite reliable. Somehow, her reporter's BS detector didn't go off, and she regurgitated that wild assertion by McAfee, a company that makes a living selling security products and services.
I had forgotten about that story until I came across a study by two Microsoft researchers who took the trouble to look hard at the facts behind the cyber crime scare stories, which persist to this day. Their paper, with the appealingly sensational title of "Sex, Lies and Cybercrime Surveys," is a rigorous debunking of the wildly inflated claims spread by security companies, law enforcement, and credulous journalists.
[ The newest security scaremongering has hit users of Mac OS X and confused IT managers of mobile devices. | Keep up with the key tech news and analysis with the InfoWorld Daily newsletter. ]
I don't mean to pick on McAfee or Mills, but as I've written more than once, neither IT nor the public benefit from security scare stories. Indeed, the more security companies cry wolf, the less likely it is that well-founded warnings will be heeded.

Consider how much money we're talking about when McAfee claims that cyber crime costs $1 trillion a year. The requested federal defense budget for the United States for fiscal year 2013 is just (!) $525.4 billion. The total profits derived from the global trade in illegal drugs were pegged at $600 billion by the International Monetary Fund in 2010.
Is cyber crime really a bigger source of revenue than the drug trade? Hard to believe.
Enter Dinei Florencio and Cormac Herley, the authors of the Microsoft study, who say, "One recent estimate placed annual direct consumer losses [from cyber crime] at $114 billion worldwide. It turns out, however, that such widely circulated cyber crime estimates are generated using absurdly bad statistical methods, making them wholly unreliable."
You'll notice that the figure they call wholly unreliable is just one-tenth the size of the McAfee assertion.
The researchers make the point that most estimates of damage are reached via surveys. Using surveys seems like a good strategy until you realize that researchers start with what appears to be a hard number provided by respondents, then extrapolate to a larger population: "Suppose we asked 5,000 people to report their cyber crime losses, which we will then extrapolate over a population of 200 million. Every dollar claimed gets multiplied by 40,000. A single individual who falsely claims $25,000 in losses adds a spurious $1 billion to the estimate. And because no one can claim negative losses, the error can't be canceled" through averaging, as happens somewhat when people choose from ranges.
They go on to say the cyber crime surveys they've examined "exhibit exactly this pattern of enormous, unverified outliers dominating the data. In some, 90 percent of the estimate appears to come from the answers of one or two individuals," Florencio and Herley state.
If you've ever done testing, you know it often makes sense to discard outliers in your results -- a practice you should've learned in introductory statistics classes. I have to assume that the people who conduct the self-servingly skewed surveys probably know it too, but choose not to bother. As we used to joke in our newsrooms: Never let facts spoil a good story.
How common is this upward bias in surveys? "Among dozens of surveys, from security vendors, industry analysts, and government agencies, we have not found one that appears free of this upward bias. As a result, we have very little idea of the [actual] size of cyber crime losses."
In case you're wondering why "sex" appears in the report title, it wasn't just to sensationalize the survey. Florencio and Herley liken the reporting of cyber crime to the reporting of the number of sexual partners claimed by survey respondents. "Cyber crime, like sexual behavior, defies large-scale direct observation and the estimates we have of it are derived almost exclusively from surveys," they say. And both topics lend themselves to exaggeration.
None of this is to suggest that cyber crime is not a problem. It is, of course. But the researchers note that in most cases, stolen passwords and other data are sold for pennies on the dollar, which is to say they're hard to monetize.
Even if they don't translate into the losses claimed by the self-interested security industry, there is a real price to be paid from the misuse of these surveys: Exaggerated stories of the size of profits dervived from cyber crime not only scare users unnecessarily, they fool novice hackers into to thinking they'll get rich quick. So they try.

infoworld.com