Ask IFIN: If patching can't save us, what can?

General
,
1

Starting a new topic based on @Shecky’s excellent question here:

I believe there are three primary strategies defenders can pursue to meaningfully respond to a reality with this many 0-days.

  1. Reducing attack surface
  2. Improving post-exploitation detection
  3. Active deception

I’m pretty passionate about 1 and 3. Widescale traditional VPN usage, for example, is a huge attack surface that could be removed with some creative networking and tools like Tailscale. If Fortinet, Ivanti, et al have such trouble keeping their devices secure, why rely on them?

Active deception takes many forms, but the objective is the same: make every move for the attacker after initial access as dangerous as possible. Use their own techniques against them to reveal their presence. We don’t have to play fair, and we shouldn’t.

2

If y’all think you’re pissed off and tired, try approaching it from the systems/architect side. It’s been “happy $day, vendor just dumped a mountain of updates, have fun picking out which you need. p.s. update all the images with those patches immediately” for years. (Which is even fucking worse if you have any sort of regulatory scrutiny.)

Which of those patches is Oracle going to say voids support or breaks production? :person_shrugging:
Did the vendor document any of it adequately? :see_no_evil_monkey::hear_no_evil_monkey::speak_no_evil_monkey:
Well at least they told you if a reboot’s required, right? :rofl:
It’s fine, I’m sure we can schedule it quickly and easily. :headstone:

And it’s all down to a fundamental lack of craft and giving a shit. Not skill - craft. Notice how all of these problems are not “oh yeah if you jiggle the bits precisely this way when precisely these conditions happen you can sometimes fail through a handler” or “… crap, I put the order wrong 10 years ago” and instead keep coming up as “basic logic bug”?

Lack of craft. This is what happens when you have an entire generation of people who think they are god’s gift to coding, and have no need to look at history or listen to older programmers.

These are not the result of unexplained, novel, or unknown behaviors. It’s quite simply people who decided they didn’t need to study or think because they knew better and everything they were doing was new and novel. (It is not, and in fact, is why most reasonable operating systems don’t co-mingle.) It’s why ../ is the meme that will never die, despite everyone knowing better in 1995. You know, back when /etc/passwd actually contained passwords and you could telnet in.

And when this has the oh so predictable results? The people responsible don’t give a shit. Companies certainly don’t - why should or would they? They’re never going to suffer any real consequences. The people who wrote it don’t give a shit. “I made an oopsie, it’s fine, just patch it.” It doesn’t cost them more than maybe 30 minutes writing a patch. It’s trivial! Maybe if it’s colossally embarrassing, they have to go work for somebody else.

It’s the normalization of failure; the standardization of unsafe operation; setting the baseline as this_is_fine.jpeg except the monitor displaying it is actively on fire. It is not the question people think it is.

“How do we deal with being unable to keep up with patching?” is entirely the wrong question and wrong framing.

The actual question that is being asked is basically “what self-flagellation can we perform indefinitely as penance for using shit that isn’t fit for purpose?”

Yes, that should sound exactly as unappealing and borderline insulting as it does.

The question we should be asking as an entire industry - not just security practitioners, fucking everyone - is: “how do we hold these assholes that have made reliability a mystery, security a joke, and any attempt to protect systems a true Sisyphean feat responsible for it?” Especially the executives that insist they take security very very seriously.

Run this by an executive: what if instead of hiring a new batch of security professionals and incident responders and crisis communication consultants every year because the vendor keeps screwing up, you stop doing business with a vendor that is directly responsible for seven digits of HR cost and climbing? What if when the same vendor keeps doing the same thing and making the same meaningless apologies and making the same mistakes year after year after year and costing you millions, you just stop doing business with them? Hell, maybe you even sue for a refund on breach of contract or misleading you as to their capabilities or whatever!

Imagine if everyone after reading the ‘root cause’ from ClownStrike said “you guys could not more obviously be :clown_face::running_shoe:, you clearly lied about your own internal processes, so get the fuck out.” ‘But, but then all those people lose their jobs when George yanks the plug so he can cash out!!’ Yeah, and? How many millions of dollars did it cost you trying to recover from that? Everyone who said “it’s okay, we all make mistakes” both failed to comprehend the RCA and is also eager to reward the dog that’s biting them. It’s fine, I’m sure giving Fido another pound of bacon while he rips open my artery will teach him to stop doing that.

A mistake is when you have a - instead of a -> somewhere in 2M LOC, the unit test couldn’t possibly catch it, and if you hit things just right any user can panic() the box. ‘I don’t need bounds checking’ is a complete lack of craft and caring. ‘It’s no big, everyone can just patch it’ is a complete lack of craft and caring. ‘We had another ../ but it’s fine, here’s a patch’ - you get the idea. It’s not people being better at finding bugs, it’s not better tooling. It’s worse and worse crap being shoveled out the door, or just the same crap from 10 years ago without having learned anything, or just saying “what’re they gonna do, buy from the other guy?! LOL!”

We are not the little Dutch boy with his finger in the dike. We’re not Superman stopping a bus from hitting that adorable puppy. At this point, we’re a guy who hasn’t slept in 6 months or had any sustenance but Monster(R) energy drinks wearing absolutely nothing but a pair of swim floaties insisting we can hold back a literal tsunami of sewage. (Please feel free to draw this image.) Especially as, hey look, they found a way to make even more and worse sewage! How grand. Which is why this is, god, the eighth? Ninth? Time I’ve had the ‘well how can we cope with vendors being shit?’ go-round. (It ain’t just security.) And you can guess how those all went by browsing your way to Palo’s or F5’s or Cisco’s or Red Hat’s or IBM’s or (your vendor here)'s security advisory page. Or product defect page. Or hot knowledge base topics. Or APARs. Or ‘hey Surveillance Machine, search for (your vendor here) sucks.’ Or just look at their share price.

So yeah. I hold that the question isn’t what to replace a broken process with. It absolutely should not be any of our job or responsibility to cover for the same repeated fuck ups, same vendors, same Bat-channel, same Bat-fucking-time. The question is how to break what is basically a cycle of abuse.

And I can tell you that I’ve thrown vendors out on their ass when it became clear they weren’t going to get it together. And so far the total result of that’s been two acquired by Oracle, and one Symantec merger-demerger. So if anyone’s got the right answer, it doesn’t appear to be me. But if people don’t start figuring out what I got wrong and getting it right, burnout with a side of nervous breakdown’s going to be the second most widespread thing in the industry, right behind self-destructive existential nihilism.

Yeah. This stuff puts me in a bit of a mood. (Can’t imagine why. Hey, check out this picture of Anna Kournikova I sent you on Discord.)

(edit: curse you, Discourse formatting.)

3

On point and eloquent as always.

I had a senior engineer at a very well known security shop basically shrug while telling me:
“Well, the Chinese have gotten very good at popping edge devices, so basically the best we can do is not connect edge devices directly to the Internet.”

So the answer is I need a firewall for my firewall.

I-heard-you-like-firewalls.gif

4

I’m just going to point out the always present urge to descend into despair and suggest we avoid it for this topic, and instead attempt to seek viable solutions.

5

I’m likely to be asked to speak to this soon, and I’ve been contemplating how I’ll answer. My first thought doesn’t scale well, because we can’t all be offgrid goat farmers.

And while @rootwyrm is right in that lots of this should be laid at the vendors feet, and we should be holding them more accountable; we are where we are, and we can’t fire vendors fast enough to change that today.

Your three are a good start, we should be doing all of those, but probably aren’t.
I don’t know that I’ve seen/heard of that many places doing active deception beyond maybe some passwords.xlsx type canaries on an open share.
And most that I’ve seen have minimal detection capabilities – either they have enough for compliance theatre, or a bunch of people, with paper certification training, drowning in alert fatigue.
As for reducing attack surface, it should already be minimal. It could be reduced further by geo-blocking, ASN-blocking, etc; but I’ve seen few people willing to risk accidentally locking an officer out of checking email while they’re on vacation in Spain.
We need to do all of your three, and do them well.

We also need to seriously look at basic architectures from a consequence-driven engineering standpoint.

-- Better network segmentation, using tools like isolated pVLANs to cut lateral movement, and layer-cake models that aren’t just Public/DMZ/Internal. With strict limitations on traffic flows between layers…no more default outbound established on anything!

-- Actually do inventory, not just on hardware, but on data and spend time thinking about how much of that can be deleted, archived, or moved to a lower layer. If it needs to be exposed close to the Internet, spend more time thinking about how that data needs to be accessed and restrict access to just those uses. A good example here is don’t put read/write Domain Controllers anywhere near an entry point, use RODCs for authentication at those layers.

-- Quit putting all the eggs in a single basket. Sure, it’s cool that we have all customer data in a single massive database, and a single pane of glass to control all our devices enterprise wide.
Maybe rethink that just a little: if it can be decentralized without impacting operations, decentralize — it’ll decrease the blast radius.
In the same vein, diversify the vendor list; particularly for security controls. It ups the management hassle some, but I know of several recent incidents where one of the biggest problems was a network segmented by the same vulnerable firewall at every layer. Just putting a different vulnerable firewall in somewhere would have slowed the attackers down.

-- Quit connecting non-mission essential stuff to the enterprise network. Those IoT cameras are neat and maybe even provide additional layers of security, get a totally isolated network drop for them.

6

Descend into despair?
This is the good side of bouncing back after researching the grid incident in Poland, while watching videos by the CTO of major vendor explaining why grid hardware doesn’t need basic security controls.
I’m actually vaguely hopeful that we may be able to dig ourselves out of this hole without the entire Internet falling down, largely because we’re finally waking up and realizing we’re in a hole.

Communities like this are going to be the only way we fix things.

7

If that’s “descending into despair” and “nonviable solutions,” then there’s no point in me sticking around for further discussion. I’m just calling a spade a spade.

8

I am in strong favor of 1 as early as it happens that ideas transform to plans, architectures or implementations..

I believe it is - for every design and development paradigm - necessary to add a dedicated black hat cycle where participants switch their hats to figure out what the change puts at stake and which countermeasures are available at that point.

For me and now: the combination of reducing attack surface and patching will be it.

I can’t remember where but i read a post about Dirty Frag and one wrote roughly “when did people stop compiling their own kernels?”.

And that made me think - yeah why not starting to reduce the attack surface at that point. Because i delegated that work (and time) to the software provider in good faith. As i trust the assembler of my hardware, as i trust the cpu manufacturer and so on and so on.

And that’s a point where i think the global delegation of trust comes to it’s limits. People with different values, goals and skill do my work. And i have to make hard decisions which risk i am willing to carry.

Patching - at least - gives me an impression how they deal with mistakes.

9

RootWyrm isn’t wrong. A lot of this does lie at the vendors feet, but an equal amount of it lies at ours as well.

I’ve seen it said in other infosec spaces countless times: “Cybersecurity is largely a solved problem.” But everyone refuses to elaborate on that. The most you will ever get out of someone is a bullet point response of:

  • Asset inventory
  • patching
  • monitoring
  • attack surface reduction

Which is all well and good, but the real answer is so much more nuanced than that, and both the industry and the community are fucking awful at elaborating on it without either shelling out thousands of dollars (USD in most cases, which is its own set of problems) or finding niche communities like this one where the knowledge is there, but asking questions about it outright is a social faux pas.

I’d like to think that everyone else here holds a special place in their heart when The Mentor wrote:

My crime is that of curiosity

Or when Stewart Brand said:

Information wants to be free

And yet, as a community we’re really shit at disseminating that information into something actually actionable instead of making memes about “ignoring security basics, lol”

If we actually want change, we have to be better at feeding that curiosity from newcomers. We can scream “Asset Inventory! Monitoring!” all we want, but if we don’t follow that up with “Here are some asset inventory best practices I’ve learned over the last X years” or “Here are the important things to monitor, here are the things that use more disk space than they’re worth, and here’s how you don’t get bogged down” then all we’re doing is leading people into the arms of shitty vendors who claim to solve all of these problems for them. Those vendors then get popped, and we’re back here writing posts about why patching can’t save us.

10

You’ve identified IFIN’s raison d’être.

12

Challenge issued :wink: