Spectre and Meltdown: What We’ve Done to Secure the gBlock Cloud

Unless you’ve been living under a rock or aren’t in the IT field at all, by now you’ve likely heard about the widespread Spectre and Meltdown vulnerabilities affecting an enormous swath of processors manufactured by Intel and AMD, the industry leaders, leading to security vulnerabilities and performance problems.

Green House Data staff have been hard at work patching systems as fixes have come available this week. Here’s a quick summary of the vulnerabilities, their effects on cloud and general computing performance, and what we’ve done to fix them so far. We also provide a few links for users who need to patch their own operating systems or investigate further.

What’s the Big Deal?

If you’re already familiar with Spectre and Meltdown, go ahead and skip below for more details on our patching strategy. If you weren’t aware of this far-reaching set of vulnerabilities, well, it’s a doozy.

Meltdown affects basically every Intel processor manufactured since 1995, while Spectre affects AMD and ARM processors as well as Intel. Together that means that these vulnerabilities are extremely likely to exist on the vast majority of computers, servers, cloud servers, and smartphones across the entire globe. These vulnerabilities are known by the Common Vulnerabilities and Exposures project as CVE-2017-5715 and CVE-2017-5753. 

Both vulnerabilities work by using side channels to access stored information in your computer’s memory. Meltdown breaks the isolation mechanism between the operating system and system memory, allowing an application to access arbitrary locations within your memory and OS that it otherwise would not have access to. Spectre breaks down isolation between different applications, allowing one app to access memory that it otherwise would not be able to see.

Basically, without patching your OS for these vulnerabilities, an attacker can hijack your existing applications to gain access to sensitive information stored in RAM, like passwords, personal details, credit card numbers, or anything else existing in your memory.

You can read more about the issues here: https://spectreattack.com

What Has Green House Data Done to Secure Its Systems?

Spectre and Meltdown stretched all the way from servers hosting our cloud platform down to user endpoints and even mobile devices. As vendors released tested and compiled fixes, we tested them within our own lab before pushing them out ASAP.

Beginning as soon as the news broke, we took steps to mitigate and patch for both vulnerabilities, with the following strategy:

1. Identify the systems under our management susceptible to Spectre and/or Meltdown
2. Begin testing available patches in our lab to help identify as many unforeseen impacts to applications as possible
3. Start staging our staff to begin patching systems

Green House Data customers received frequent updates on the progress of patching for systems under the management of Green House Data staff, including timing preferences. For customers managing their own environments, we encouraged (and continue to encourage, for those dragging their feet), patching of all physical and virtual operating systems.

As of today, January 12, 2018, all systems are believed to be patched. Below are the milestones we checked off on the way.

The situation continues to develop, however, and vendors are still releasing patch updates as performance hiccups and other problems have reared their heads. Green House Data continues to evaluate, test, and install or rollback patches as they are released or updated. For example, some AMD systems were still waiting on an updated patch from Microsoft, who pulled their AMD patch on Tuesday due to additional performance issues.

1/4/18: Incident team formed with tasks/milestones identified

1/5/18: All operations and management clusters patched at hypervisor level

1/7/18: All customer compute clusters patched at hypervisor level

1/8/18: All customer facing portals patched unless vendor has not supplied patch

1/12/18: All remaining systems patched, and exceptions noted for further analysis

Ongoing: Work with support vendors and partners to validate, test, and install new patches for all critical systems

We are also monitoring systems with close attention to watch for any signs of malicious activity, such as access from suspicous locations. Unfortunately, neither attack leaves much of a trace for digital forensics to track.

What Can I Do?

If your systems are not managed by Green House Data staff, we strongly urge you to patch all physical and virtual operating systems. We also recommend that all customers patch their systems individually, strengthening the protections within the OS that isolate software currently running on the virtual machine. Contact any other relevant vendors for guidance as necessary.

Microsoft offers some guidance for handling both Windows clients and servers:

https://support.microsoft.com/en-us/help/4073119/windows-client-guidance-for-it-pros-to-protect-against-speculative-exe

https://support.microsoft.com/en-us/help/4072698/windows-server-guidance-to-protect-against-the-speculative-execution

https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/ADV180002  

RedHat offers some guidance for handling RedHat Enterprise Linux versions 6 and 7:

https://access.redhat.com/security/vulnerabilities/speculativeexecution

Wait, What About the Performance Issues?

I did mention performance problems, and those come from the patches themselves. This gets pretty technical, pretty fast, so for a deep dive I recommend taking a look at this Ars Technica Article about the performance hit from patching Meltdown and Spectre.

Generally, the hardest hit processors will be older ones. As patching continues, it appears that newer hardware has only taken performance hits of a few percentage points. Most of the impact has come from the actual process of patching itself, which, even if it takes place outside of usual working hours, can negatively impact application and server performance.

With these vulnerabilities, the patching is more intensive as well, as it affects deep OS operations within the kernel. Updating a few hundred or thousand VMs with a major OS patch takes some serious time and some serious computing effort.

Processors that are a few years old may be hit with a 20% or more negative performance toll for the patches, which is hardly insignificant. At this point in time, it’s hard to definitively know exactly how major the performance hit will be, as many of the patches are also causing glitches, reboots, and slowdowns, with new versions of the patches sometimes helping to reduce these. Case in point: the AMD patch that Microsoft pulled on Tuesday.

The situation will no doubt continue to evolve over the next few weeks and even the next few years, as vendors refine their patches and attackers find new ways to exploit the vulnerabilities. In the meantime, make sure your computers have the most recent OS updates applied — as you should anyway.