This is mostly griping, but I wanted to have this conversation.
I’ve observed a lot of “statistical dark patterns” in reports I get from commercial threat intel. Things like telling me how attacks against my industry grew year-over-year without giving me the total increase in attacks or any delta in my industry’s share of the total. Or providing raw scalar counts of “attacks” without any context.
I’m curious what kinds of practices like that you’ve observed, and how/if you push back against it?
As someone who writes threat intel for a commercial provider, I don’t often get to see private reports from other vendors (as the saying goes: Information wants to be proprietary and paywalled). It’s interesting to know that these dark patterns aren’t just in the stuff they put out publicly.
I have to wonder how much of it is people not knowing how to properly present results and how much of it is intentional. Probably an unhealthy mix of both.
From informal conversations with others in the industry, I get the impression that the industry-specific and periodical reports are fairly often compiled by people not involved in producing the actual intelligence. This could partially explain the lack of context around the numbers.
An annoying behavior I’ve noticed in public reporting is vendors only providing queries for their own EDRs instead of YARA or Sigma rules (sometimes in lieu of IoCs. Looking at you Microsoft…)
Wow do I hate industry-focused attack reports.
It is SO dependent on the threat actor and their intentions that it is kind of a useless metric for most industries.
Do North Korean TAs disproportionately target tech and crypto companies? Yes and you should be aware of that and take action.
Do “spider” or opportunistic ransomware groups wake up one day and say “I’m going to spend the next month only targeting retail companies with 10,000 employees and with logos that contain pretty colors”? No.
I have the same feelings about the MITRE ATT&CK matrix when talking about detection coverage.
Oh absolutely. The targeting reports are mostly nonsense unless you’re talking about politically motivated groups. Most everything else is vuln-driven or opportunistic, imo.
This is such a common thing, I didn’t realize that anybody noticed it. “You get what you pay for” doesn’t seem to count for much when it comes to data.
I keep spreadsheets of metrics from feeds and do the math myself. They’re not going to and I never have the budget for services that will, so an hour or two up front with GSheets saves hours of questions during security meetings.
As for hard numbers, if we’re not paying for that provider to keep accurate numbers on the crap that hits our infrastructure at $dayjob they’re not going to. So, I do. I write some utilities that go through relevant logs and do numerical breakdowns.
To put it another way, “This is what they tell us. This is what we actually see hitting us.”