Dell EMC VxRail Software Update – Spectre Guest OS leakage mitigation

I posted earlier in the year that Dell EMC had released a Security Advisory to address Spectre (Meltdown doesn’t really affect VMware and hence VxRail).

One of the items that wasn’t addressed in the original fix was Guest OS leakage mitigation between processes within the VM – this required CPU/BIOS microcode updates which were not yet available from Intel.

Those updates were made available from Intel at the beginning of April and it’s taken a while for it to filter through to vSphere and VxRail – the delay is down to VxRail being a fully turn-key appliance which means all software/firmware updates from Dell EMC are fully tested and validated before release.

Updates 4.0.402 and 4.5.152 are now available to download from Dell EMC’s support portal.

Release notes can be found here:
https://support.emc.com/docu80740_VxRail-Appliance-Software-4.0.x-Release-Notes.pdf?language=en_US
https://support.emc.com/docu86659_VxRail-Appliance-Software-4.5.x-Release-Notes.pdf?language=en_US

The accompanying Dell EMC Security Advisory is available here: DSA-2018-074: Dell EMC VxRail Security Update for Multiprocessor Side-Channel Analysis Attacks (Meltdown and Spectre)

VxRail Appliance software 4.0.402 and 4.5.152 contains the Intel microcode fix to complete the resolution of the speculative execution security issues.
VxRail Appliance software 4.0.402 includes fixes for the following security vulnerabilities:

  1. CVE-2017-5753 (Variant 1: bounds check bypass, also known as Spectre) – Complete fix in 4.0.401 and above.
  2. CVE-2017-5715 (Variant 2: branch target injection, also known as Spectre):
    • Mitigates leakage from the hypervisor or guest VMs into a malicious guest VM – Complete fix in 4.0.401 and above.
    • Guest OS leakage mitigation between processes within the VM requires BIOS or CPU microcode update released by Intel and included in this release – Complete fix with either BIOS or CPU microcode update automatically applied through the VxRail 4.0.402 automated software upgrade. No manual BIOS update required for any supported VxRail hardware platforms.
  3. CVE-2017-5754 (Variant 3: rogue data cache load, also known as Meltdown): Does not affect VxRail Appliance.

NOTE: Manual steps are required after the VxRail Appliance software upgrade to 4.0.402 to power cycle the VMs for branch target injection to take effect. More info available within this KB article: https://support.emc.com/kb/519601

Also note that this update does not patch Guest OS!

For more information about Spectre/Meltdown, have a meander to my original posts:
Spectre & Meltdown Vulnerabilities
Spectre & Meltdown Update

Advertisements

vExpert 2018 Award Announcement

So last Thursday/Friday the vExpert slack channel was awash with lots of nervous energy as people were eagerly waiting for the announcement to see if they had been accepted back into the vExpert program for 2018…. Strange, but to me it seemed that everyone was a little bit more nervous this year then previous years!

On a side note – my newly favourited key stroke on Slack is Shift+Esc which clears all unread messages and notifications! =P

What probably didn’t help the nerves was when someone posted up a tweet by Eric Nielsen (who helps run the community alongside Corey Romero) showing that 1366 were accepted into the 2018 vExpert program, 305 were rejected and 183 deferred!!
Definitely made me a bit more nervous when I saw that…. >_<”

I think some people take it for granted that they’ll be re-accepted, I for one am always nervous and never take these things for granted because I see a lot of other people around me who blog a lot more than me or help out in the community a lot more than me.

Nerves were finally settled close to midnight on Friday, just as I was getting ready to go to bed…. an email pinged through with some welcoming words:
vexpert

I’m obviously glad and honoured to be considered part of this amazing group for the 4th year running. =)

The new vExpert portal looks brilliant and the directory has even updated our profiles:
vexpert-profile

For those who don’t know, the VMware vExpert program is VMware’s global evangelism and advocacy program. It’s a select group held in high regards within the VMware community as a bunch of IT professionals who ‘give back’ to the community whether by sharing their VMware knowledge by blogging or by helping within the community forums.

 

As always, much thanks has to go to those in the background who help run the vExpert and VMTN communities…. Eric NielsenCorey Romero and Katie Bradley (to name just a few… apologies if I’ve missed anyone out).

 

Finally well done to all the new and returning vExperts for 2018.

https://blogs.vmware.com/vmtn/2018/03/vexpert-2018-award-announcement.html

 

MTI Secure Hyper-Converged Infrastructure Webinar

So last Thursday I was asked by the marketing peeps at my company, MTI Technology, to run a webinar with my colleague, Andrew Tang, around what Hyper-Converged Infrastructure is all about, why it’s suddenly become so popular within the industry, and how best to secure a HCI solution.

The webinar has now been uploaded for public consumption…. and since it kind of went ok – apart from me suffering from a runny nose throughout (sorry for all the sniffing) – I’ve decided to blog about the webinar for you all to watch.

I don’t really touch upon product in this webinar, as the last thing customers want is to be shoehorned into a certain vendor product… instead I hope the webinar gives enough information about what HCI is in general, why customers should be looking at HCI during their next infrastructure refresh, and more importantly what to consider when evaluating a HCI solution!

Feel free to pop along and access the webinar recording here: https://mti.com/secure-hci-webinar-page/ (sorry, you have to fill in your details to gain access….)

Finally, if you’re interested in talking more about HCI then feel free to contact me or register for one of MTI’s HCI Discovery Workshops: http://bit.ly/2C8vS14

End of General Support for vSphere 5.5 and other Products

So there’s 7 months left from today until vSphere 5.5 and complimentary VMware products go out of General Support. The official EoGS date for vSphere 5.5 is 19th September 2018.

The products going EoGS on that same date are:

  • Site Recovery Manager 5.5/5.8
  • vSAN 5.5
  • vCenter Server 5.5 (including Update Manager 5.5)
  • vSphere Replication 5.5/5.6/5.8
  • vSphere Data Protection 5.5/5.8

When products go End of General Support, this typically means those products stop getting updates/upgrades and patches – including any new security/bug fixes! The only support offered is web based Support Requests for assistance on existing patches and bug fixes. Fortunately any issues with Spectre/Meltdown will still be addressed post Sept 19th as they’re classed as existing security issues (phew).

My suggestion is that you start planning your upgrades to vSphere 6.5 now as typically from experience with my customers, an upgrade project tends to take 2-3 months to plan/design and another month to execute.

There are quite a number of differences with 6.5, and I’ve blogged about it previously here:

There are 2 VMware KBs I recommend people reading before planning an upgrade:

 

Finally the best website to visit to help with any upgrades is VMware’s own Upgrade Center: https://www.vmware.com/products/vsphere/upgrade-center.html

Dell EMC updates VxRail software to address Spectre

So Dell EMC have finally released the patches for their VxRail appliances, I know many of my customers were asking about these patches – in a way it’s good it was slightly delayed given how many normal VMware customers experienced issues when patching and how one patch was pulled by VMware!

The good thing about VxRail is that any software patches or updates released have been tried and tested by the Dell EMC CPSD engineering team, so they should be ready for roll out with minimum disruption!

Updates 4.0.401 and 4.5.150 are now available to download from Dell EMC’s support portal.

Release notes can be found here:
https://support.emc.com/docu80740_VxRail-Appliance-Software-4.0.x-Release-Notes.pdf?language=en_US
https://support.emc.com/docu86659_VxRail-Appliance-Software-4.5.x-Release-Notes.pdf?language=en_US

It’s worth noting that at present this patch only contains 2 of the 3 required fixes for Intel to address the Speculative Execution vulnerability (Spectre – Meltdown doesn’t really affect VMware and hence VxRail). The 3rd fix has not yet been released by Intel and Dell EMC basically decided they couldn’t wait any longer as Intel drag their heels!

Spectre & Meltdown Update

So it seems that the microcode patches released by VMware associated with their recent Security Advisory (VMSA-2018-0004) have been pulled….
https://kb.vmware.com/s/article/52345
So that’s ESXi650-201801402-BG, ESXi600-201801402-BG, or ESXi550-201801401-BG.

The microcode patch provided by Intel was buggy and there seems to be issues when VMs access the new speculative execution control mechanism (Haswell & Broadwell processors). However, I can’t seem to find much around what these issues are…

For the time being, if you haven’t applied one of those microcode patches, VMware recommends not doing so and to apply the patches listed in VMSA-2018-0002 instead.

If you have applied the latest patches you will have to edit the config files of each ESXi host and add in a line that hides the new speculative execution control mechanism and reboot the VMs on that host. Detailed information can be found in the KB above.

 

Finally William Lam has created a very handy PowerCLI script that will help provide information about your existing vSphere environment and help identify whether you have hosts that are impacted by Spectre and this new Intel Sighting issue: https://www.virtuallyghetto.com/2018/01/verify-hypervisor-assisted-guest-mitigation-spectre-patches-using-powercli.html

Spectre & Meltdown Vulnerabilities

So at the beginning of the new year, news broke via The Register that there could be a potential security vulnerability to Intel processors (Meltdown) and how it was a problem which couldn’t be easily fixed by a microcode update because of how the Intel architecture does speculative execution of code (in a nutshell this is how modern processors try to ‘predict’ the code it needs to execute next, before the current executing code produces a result – all modern processors do this to some extent in order to fill its internal pipeline and speed up processing)…. this quickly blew up into a storm where additional vulnerabilities were identified (Spectre) which affects Intel, AMD and ARM processors!

Three closely related vulnerabilities involving the exploit of speculative execution in CPUs were made public:

Variant 1 & 2 have been branded as Spectre, with Variant 3 known as Meltdown.

The fallout is spectacular…. lawsuits being filled against Intel…. videos of exploits (proof of concepts) already on youtube….. customers going crazy that Russians/North Koreans could be stealing data from their systems….. all this because chip manufacturers tried to outdo each other by putting speed of processing above security.

The best article I’ve read that explains how Speculative Execution works and how these vulnerabilities could be exploited can be found here: http://frankdenneman.nl/2018/01/05/explainer-spectre-meltdown-graham-sutherland/

It seems that at the moment the only way to minimise your exposure to potential exploits is to patch the OS or Hypervisor, however this isn’t without issues as people have started reporting that it adds an overhead to performance. In all honesty, I doubt personal users will notice a performance hit on their day to day usage (home/office applications or games), it will however impact anyone that undertakes high IO or system-call intensive applications (such as DBs, email, Big-data/data-mining)… a performance hit of between 5-30% depending on application!!

VMware have stated that at present they don’t believe Meltdown to be an issue to their products because ESXi does not run untrusted user mode code, and Workstation and Fusion rely on the protection that the underlying operating system provides. For Spectre, they have released an article detailing their response to the issues and 2 Security Advisories which addresses the vulnerabilities and how they can be mitigated, VMSA-2018-0002 has been superseded by VMSA-2018-0004.

From what I can see, the first Security Advisory consists of security patches to ESXi that addresses the vulnerability to mitigate against leakage from the hypervisor or guest VMs into a malicious guest VM – these were patches made available late last year before the news broke (which makes you wonder how long the industry have known about it).

The second Security Advisory is a full minor update to vCenter (5.5, 6.0 and 6.5) in order to support both newer vSphere ESXi patches and Microcode/BIOS patches to hardware. This seems to be what they call “Hypervisor-Assisted Guest mitigation” which virtualises the new speculative-execution control mechanism for guest VMs so that a Guest OS can mitigate leakage between processes within the VM – and this mitigation requires specific microcode patches from platform vendors which seem to introduce these new ‘speculative-execution control features’. More information on how to apply this Security Advisory can be found here: https://kb.vmware.com/s/article/52085.

Note: The update patches found in VMSA-2018-0004 will mean that these new CPU features will be exposed to Guest VMs and as such vMotion to ESXi hosts without the microcode or hypervisor patches applied will be prevented. However, if you have an EVC cluster, it looks like vCenter will suppress the new features from VMs to enable vMotion compatibility until all hosts have been upgraded (after which it will enable those features) – unpatched hosts will not be allowed to join an EVC cluster that has been patched.

It’s worth noting that Guest VMs should also have their OS updated with the latest security patches for effective mitigation of these known vulnerabilities!

Finally, VMware have released an article regarding these vulnerabilities and whether their virtual appliances are affected: https://kb.vmware.com/s/article/52264. It currently looks like vSphere Integrated Containers and vRealize Automation have not been patched yet.

VMware on Microsoft Azure….. interesting!

Earlier this week, Microsoft let slip that they were working with Premier VMware partners on a tech preview to deploy a full VMware stack on Azure bare-metal hardware, co-located with other Azure services.

Initially billed as a ‘stepping-stone’ to full Azure Cloud, Microsoft have made known that “sometimes there are specific VMware workloads that can be more challenging to migrate to the cloud” – and so customers may need the option to run these workloads on a VMware stack in Azure (for the time being). What I can’t quite work out yet is what these “workloads” would be… after all, nearly every workload I’ve ever deployed on VMware can be easily re-deployed on Hyper-V!

Microsoft have mentioned that this new VMware stack on Azure will GA in 2018. What they haven’t mentioned is who they’re working with, who will own and support the service and how it would be licensed…. for a start, it’s very interesting that it’s not being developed alongside VMware, and VMware have come out to say they’re not aware of any of their partners collaborating alongside VMware engineering to deliver this service – in fact VMware have stated it’s being developed independent of VMware and is “neither certified nor supported by VMware…. VMware does not recommend and will not support customers running on the Azure announced partner offering.” – which kind of makes you wonder what happens if a customer encounters problems with this Azure service?!? I highly doubt there will be any enterprise customers taking up this un-supported Azure service!!

I’m not sure why Microsoft have stated that “running your VMware stack in the cloud doesn’t address your hybrid requirements”… surely the fact that having a common framework on-prem and off-prem (ie VMware Cloud Foundation) is that “true consistency across your cloud and on-prem environment” that Microsoft say is missing….?!? Whilst it maybe true that Azure can provide a complete hybrid cloud package, let’s face it their Azure Stack offering is pretty limited – only a select few hardware vendors, no ability for customers to use their own hardware and lack of ability to expand/upgrade – plus I’m not aware of many customers jumping on board the Azure Stack on-prem platform! Also, when it comes to networking, Microsoft’s offering lacks the features of what NSX offers to VMware customers!

Should VMware start getting worried about this new announcement…..? On the contrary, they seem to have embraced the idea and even have the audacity to spin this announcement as Microsoft “recognizing the leadership position of VMware’s offering…. as a superior and necessary solution for customers over Hyper-V…..!!” TBH, they’ve never really seen much damage done to their vSphere install base when Microsoft started releasing tools to help people migrate off VMware, so I doubt this new announcement will trouble their new VMware Cloud on AWS offering.

It’s interesting that it was announce alongside the new Azure Migrate service which helps you discover and plan the migration of your on-prem VMware workloads and then execute the migration with Azure Site Recovery (ASR).

In my opinion, it’s all just a bit of hot air coming from Microsoft to try and take some of the plaudits before next weeks AWS re:Invent conference!

 

However, I do hope that Microsoft swallow some pride and reach out to VMware and start a combined engineering/development effort as that will go a long way to what every man and his dog wants to see – VMware Cloud on Azure! Only when Azure comes on board will VMware be able to say they are now a “broker of cloud” as only then will customers be given the option to migrate workloads seamlessly between the 2 biggest players in the public cloud market! (TBH given the relationship VMware has with GCP, I can see VMware Cloud on GCP happening first before Azure – although hardly anyone uses GCP!)

I mean, VMware and Microsoft already partner to offer VMware Horizon Cloud on Azure, surely they can put their differences aside and produce the one thing everyone is asking for!

…. watch this space….. (in eager anticipation!)

vSphere Central – new resource centre

A little while back I caught the vSphere blog about vSphere Central being launched and ended up bookmarking the portal to have a look at a later date. I had totally forgot about it till today when I needed to look up the PSC topology diagrams and Google sent me to the new vSphere 6.5 Topology and Upgrade Planning Tool (more on this later). Turns out this portal is exactly like Storage Hub (resource portal for everything vSAN, SRM and storage related)!

Everything technical you need to know about vSphere and vCenter can be found on this portal:

  • How to install vCenter and vSphere
  • How to migrate to vCSA
  • How to upgrade vCenter and vSphere
  • vCenter and PSC architecture
  • SSL certificate management
  • PSC Deployment Types
  • Product Interoperability Matrix
  • All the new features in 6.5 explained (vCenter HA, Backup/Restore, etc)

It really is a great resource portal, and even better you can download each section as a PDF! Beats the documentation site for vSphere as it’s far more easier to navigate!

The content is in a range of formats, most of it is text taken from the technical pdf documents, but there are videos and walkthrough demos also scattered throughout the topics.

One of the things launched with vSphere Central was the vSphere 6.5 Topology and Upgrade Planning Tool.

This tool aims to help customers plan and execute both upgrades to vSphere 6.5 as well as new deployments. With this initial release, the tool is focused on the most common upgrade paths and deployments of vCenter Server 6.5. The tool works by asking a series of questions while providing some guidance along the way to help answer those questions eventually making some recommendations on topology and upgrade and deployment steps.

In the past I used to refer to the VMware KB on deployment topologies: https://kb.vmware.com/kb/2147672

Some of the guys in the vSphere technical marketing team then came up with the PSC Topology Decision Tree which was a large poster – https://blogs.vmware.com/vsphere/2016/04/platform-services-controller-topology-decision-tree.html

This tool was inspired by the Decision Tree poster and extends its capability.

What I especially like about the tool is that after answering a series of questions regarding how I’m planning to design the vCenter/PSC deployment it gives me a recommended Topology diagram and then explains the steps to go about deploying the solution:

topology

Anyways, it’s a great tool…. and the portal is a brilliant collection of resources! Go use it! Bookmark it now…! =)

Horizon Cloud on Azure – GA

Interesting tie up between VMware and Microsoft…. is this the beginning of a new relationship? Have Microsoft woken up (post-VMworld) to the awesome VMware Cloud on AWS and realised they also want in on the party? (although if i’m honest this partnership has been bubbling in the background for a while now).

Anyways, after a round of beta testing in the US, Horizon Cloud has now gone GA on Azure: https://blogs.vmware.com/euc/2017/10/vmware-horizon-cloud-on-microsoft-azure-now-available.html

When Horizon Cloud was launched earlier this year, the concept of enabling end-user organisations the ability to deploy feature-rich VDIs and applications across multiple deployment options was very promising. From a single management console, end users are able to deploy virtual desktops onto on-premise infrastructure, to the Cloud, or a hybrid combination of both. TBH, some of this concept was already available in Horizon Air (which came out of the Desktone acquisition), but this is an evolution of that product.

Horizon Cloud is a cross-cloud architecture for VDI – much like how Cloud Foundation is for SDDC – however, in the case of Horizon Cloud, the Portal which acts as the control/management plane resides solely in the cloud (you get a choice with Cloud Foundation’s SDDC Manager), administrators log into this portal to deploy and manage their VDI sessions – whether on prem or cloud.

3 offerings currently:

  1. Horizon Cloud Hosted – so VDI infrastructure provided by VMware (IBM Cloud is currently the only provider), where you just choose the type of desktop and apps to deploy via the portal – DaaS. Infrastructure management/maintenance/SLAs are fully undertaken by VMware.
  2. Horizon Cloud On-Premise – based on HCI technology and acts like a stepping stone to Cloud VDI. VDI stored locally on prem, but management is all from the Cloud, perfect for data-residency issues, for end-users who require high performance VDI, and for IT admins who wish to have greater control over their VDI infrastructure.
  3. Horizon Cloud on Microsoft Azure – delivering RDS VDI and apps hosted in Azure datacentres. Connecting a customers Azure IaaS subscription to Horizon Cloud. So VMware manages the VDI aspects and Microsoft the underlying infrastructure. Also worth noting that currently only Azure deployments support vGPU-accelerated infrastructure.

Whatever the deployment option, customers will get a VDI infrastructure that’s easily scalable (whether cloud or on-prem) and easy to deploy. The best part is you get the flexibility of subscription based pricing.

With Horizon Cloud on Azure, you can import gold images from Azure marketplace which will then be configured and deployed for Horizon.

One key element of the Horizon Cloud technology is justin-time (JIT) provisioning of virtual desktops and applications. Using the configurations made in the cloud-control
plane, Horizon Cloud leverages VMware App Volumes, User Environment Manager, and VMware Instant Clone technologies to assemble personalised virtual desktop and application environments when an end user logs in, giving IT administrators high flexibility in leveraging the infrastructure.

I like the idea that if I have a persistent VDI deployed in Horizon Cloud, then I can access that VDI or hosted apps whether I’m in the office or on the move (as long as there is data connectivity). I can start writing a document in the office, then leave it open mid-sentence as I leave the office, or jump on a train – I can even power off my endpoint device – then I can re-establish the session and carry on without any interruption… an Always On desktop!

I also like the idea that with Azure, I could deploy a VDI session to the datacentre in the UK, then as I jump on a flight to the US I can re-deploy that VDI session to an Azure datacentre in the US. Although, I’m not quite sure you can migrate live VDI sessions between datacentres yet – I haven’t seen any articles that say you can live-migrate VDIs (but one would think this would be the ideal end-goal).

 

Hopefully this new VMware-Microsoft partnership will lead onto Microsoft accepting to run Cloud Foundation on Azure (VMware Cloud on Azure) which will then give end users the freedom to move their workloads from on prem to either AWS or Azure!! Almost Cross-Cloud (just need GCP to step up).