Dell EMC updates VxRail software to address Spectre

So Dell EMC have finally released the patches for their VxRail appliances, I know many of my customers were asking about these patches – in a way it’s good it was slightly delayed given how many normal VMware customers experienced issues when patching and how one patch was pulled by VMware!

The good thing about VxRail is that any software patches or updates released have been tried and tested by the Dell EMC CPSD engineering team, so they should be ready for roll out with minimum disruption!

Updates 4.0.401 and 4.5.150 are now available to download from Dell EMC’s support portal.

Release notes can be found here:
https://support.emc.com/docu80740_VxRail-Appliance-Software-4.0.x-Release-Notes.pdf?language=en_US
https://support.emc.com/docu86659_VxRail-Appliance-Software-4.5.x-Release-Notes.pdf?language=en_US

It’s worth noting that at present this patch only contains 2 of the 3 required fixes for Intel to address the Speculative Execution vulnerability (Spectre – Meltdown doesn’t really affect VMware and hence VxRail). The 3rd fix has not yet been released by Intel and Dell EMC basically decided they couldn’t wait any longer as Intel drag their heels!

Advertisements

Spectre & Meltdown Update

So it seems that the microcode patches released by VMware associated with their recent Security Advisory (VMSA-2018-0004) have been pulled….
https://kb.vmware.com/s/article/52345
So that’s ESXi650-201801402-BG, ESXi600-201801402-BG, or ESXi550-201801401-BG.

The microcode patch provided by Intel was buggy and there seems to be issues when VMs access the new speculative execution control mechanism (Haswell & Broadwell processors). However, I can’t seem to find much around what these issues are…

For the time being, if you haven’t applied one of those microcode patches, VMware recommends not doing so and to apply the patches listed in VMSA-2018-0002 instead.

If you have applied the latest patches you will have to edit the config files of each ESXi host and add in a line that hides the new speculative execution control mechanism and reboot the VMs on that host. Detailed information can be found in the KB above.

 

Finally William Lam has created a very handy PowerCLI script that will help provide information about your existing vSphere environment and help identify whether you have hosts that are impacted by Spectre and this new Intel Sighting issue: https://www.virtuallyghetto.com/2018/01/verify-hypervisor-assisted-guest-mitigation-spectre-patches-using-powercli.html

Spectre & Meltdown Vulnerabilities

So at the beginning of the new year, news broke via The Register that there could be a potential security vulnerability to Intel processors (Meltdown) and how it was a problem which couldn’t be easily fixed by a microcode update because of how the Intel architecture does speculative execution of code (in a nutshell this is how modern processors try to ‘predict’ the code it needs to execute next, before the current executing code produces a result – all modern processors do this to some extent in order to fill its internal pipeline and speed up processing)…. this quickly blew up into a storm where additional vulnerabilities were identified (Spectre) which affects Intel, AMD and ARM processors!

Three closely related vulnerabilities involving the exploit of speculative execution in CPUs were made public:

Variant 1 & 2 have been branded as Spectre, with Variant 3 known as Meltdown.

The fallout is spectacular…. lawsuits being filled against Intel…. videos of exploits (proof of concepts) already on youtube….. customers going crazy that Russians/North Koreans could be stealing data from their systems….. all this because chip manufacturers tried to outdo each other by putting speed of processing above security.

The best article I’ve read that explains how Speculative Execution works and how these vulnerabilities could be exploited can be found here: http://frankdenneman.nl/2018/01/05/explainer-spectre-meltdown-graham-sutherland/

It seems that at the moment the only way to minimise your exposure to potential exploits is to patch the OS or Hypervisor, however this isn’t without issues as people have started reporting that it adds an overhead to performance. In all honesty, I doubt personal users will notice a performance hit on their day to day usage (home/office applications or games), it will however impact anyone that undertakes high IO or system-call intensive applications (such as DBs, email, Big-data/data-mining)… a performance hit of between 5-30% depending on application!!

VMware have stated that at present they don’t believe Meltdown to be an issue to their products because ESXi does not run untrusted user mode code, and Workstation and Fusion rely on the protection that the underlying operating system provides. For Spectre, they have released an article detailing their response to the issues and 2 Security Advisories which addresses the vulnerabilities and how they can be mitigated, VMSA-2018-0002 has been superseded by VMSA-2018-0004.

From what I can see, the first Security Advisory consists of security patches to ESXi that addresses the vulnerability to mitigate against leakage from the hypervisor or guest VMs into a malicious guest VM – these were patches made available late last year before the news broke (which makes you wonder how long the industry have known about it).

The second Security Advisory is a full minor update to vCenter (5.5, 6.0 and 6.5) in order to support both newer vSphere ESXi patches and Microcode/BIOS patches to hardware. This seems to be what they call “Hypervisor-Assisted Guest mitigation” which virtualises the new speculative-execution control mechanism for guest VMs so that a Guest OS can mitigate leakage between processes within the VM – and this mitigation requires specific microcode patches from platform vendors which seem to introduce these new ‘speculative-execution control features’. More information on how to apply this Security Advisory can be found here: https://kb.vmware.com/s/article/52085.

Note: The update patches found in VMSA-2018-0004 will mean that these new CPU features will be exposed to Guest VMs and as such vMotion to ESXi hosts without the microcode or hypervisor patches applied will be prevented. However, if you have an EVC cluster, it looks like vCenter will suppress the new features from VMs to enable vMotion compatibility until all hosts have been upgraded (after which it will enable those features) – unpatched hosts will not be allowed to join an EVC cluster that has been patched.

It’s worth noting that Guest VMs should also have their OS updated with the latest security patches for effective mitigation of these known vulnerabilities!

Finally, VMware have released an article regarding these vulnerabilities and whether their virtual appliances are affected: https://kb.vmware.com/s/article/52264. It currently looks like vSphere Integrated Containers and vRealize Automation have not been patched yet.

Goodbye vCenter Server for Windows and Flash-based vSphere web client!

Hmm…. it’s not even VMworld yet and VMware decide to make 2 big-ish announcements.

Although tbh, since vSphere 6.5 was released these 2 announcements have long been coming!

Finally, after loads of speculation, VMware had announced that vCenter Server for Windows and the Flash-based vSphere web client is to be deprecated with the launch of the next version of vSphere. Updates to 6.5 will continue supporting the 2 features, but come vSphere 7.0 it will be no more….

https://blogs.vmware.com/vsphere/2017/08/farewell-vcenter-server-windows.html

 

“vCSA-exclusive capabilities such as file-based backup and restore, unified update and patching, native vCenter High Availability, and a significant performance advantage mean that the VCSA has become the platform of choice for vCenter Server.  Additionally, due to the integrated nature of appliance packaging, VMware is able to both better optimize and innovate vCenter Server at an accelerated pace.  Finally, with the VCSA, VMware can provide support for the entire vCenter Server stack including the vCenter Server application, the underlying operating system (Photon OS), and the database (vPostgres). By doing so, VMware can ensure that customers can focus on what matters most while having a single source for updates, security patches, and support.  The VCSA model is simply a better model for vCenter Server deployment and lifecycle management.”

That pretty much sums up why VMware are 100% behind the vCSA, although they miss out the whole “screw you Microsoft licensing!!” part! Plus given that 6.5 ships with a migration tool that helps you move/upgrade from a Windows vCenter to an Appliance vCenter, it’s no surprised that more and more people are moving over when it comes round to upgrade time!

In fact ever since 6.5 was released, I’ve not even deployed a single Windows vCenter Server – all my customers have been moved over to the vCSA.

https://blogs.vmware.com/vsphere/2017/08/goodbye-vsphere-web-client.html

With regards to the vSphere Web Client, loads of people found the flash-based version was frustratingly difficult to use – it was slow, it was notoriously prone to crashing and frankly it was based on in-secure Flash technology (not to mention that Adobe themselves are dropping flash). HTML5 is the way to go baby!

So with those announcements in mind….. I may think about changing some of my VMworld sessions to jump on the vCSA and Web Client update sessions!!

 

RIP…..

Nutanix to gobble PernixData?!?

Hmm, interesting rumours……
http://www.theregister.co.uk/2016/07/01/hyperconverged_nutanix_could_be_buying_vmcare_cacher_pernixdata/
http://www.theregister.co.uk/2016/07/06/nutanix_gobbling_pernix_why_not/

Anthony Spiteri sums up my thoughts about PernixData quite well in his blog: http://anthonyspiteri.net/nutanix-buying-pernixdata-critical-analysis/

Based on past experience with FVP and Architect and running both in a demo lab, I’ve always thought they’re pretty clever products…. FVP especially with the way it handles fault tolerance with their clever clustering functionality for the cache!

Anyways, it was always a hard sell…. trying to solve storage problems (which may not exist) with a software caching solution which – let’s be quite frank – was expensive!

Plus with the discounts you can now get for All Flash Arrays, and the price of SSDs dropping, it was always going to be hard to convince a customer to part with money to put in FVP and some local SSDs rather than purchasing a brand new storage array full of Flash!
Be interesting to see what Nutanix does with them!