Spectre & Meltdown Update

So it seems that the microcode patches released by VMware associated with their recent Security Advisory (VMSA-2018-0004) have been pulled….
https://kb.vmware.com/s/article/52345
So that’s ESXi650-201801402-BG, ESXi600-201801402-BG, or ESXi550-201801401-BG.

The microcode patch provided by Intel was buggy and there seems to be issues when VMs access the new speculative execution control mechanism (Haswell & Broadwell processors). However, I can’t seem to find much around what these issues are…

For the time being, if you haven’t applied one of those microcode patches, VMware recommends not doing so and to apply the patches listed in VMSA-2018-0002 instead.

If you have applied the latest patches you will have to edit the config files of each ESXi host and add in a line that hides the new speculative execution control mechanism and reboot the VMs on that host. Detailed information can be found in the KB above.

 

Finally William Lam has created a very handy PowerCLI script that will help provide information about your existing vSphere environment and help identify whether you have hosts that are impacted by Spectre and this new Intel Sighting issue: https://www.virtuallyghetto.com/2018/01/verify-hypervisor-assisted-guest-mitigation-spectre-patches-using-powercli.html

Advertisements

Spectre & Meltdown Vulnerabilities

So at the beginning of the new year, news broke via The Register that there could be a potential security vulnerability to Intel processors (Meltdown) and how it was a problem which couldn’t be easily fixed by a microcode update because of how the Intel architecture does speculative execution of code (in a nutshell this is how modern processors try to ‘predict’ the code it needs to execute next, before the current executing code produces a result – all modern processors do this to some extent in order to fill its internal pipeline and speed up processing)…. this quickly blew up into a storm where additional vulnerabilities were identified (Spectre) which affects Intel, AMD and ARM processors!

Three closely related vulnerabilities involving the exploit of speculative execution in CPUs were made public:

Variant 1 & 2 have been branded as Spectre, with Variant 3 known as Meltdown.

The fallout is spectacular…. lawsuits being filled against Intel…. videos of exploits (proof of concepts) already on youtube….. customers going crazy that Russians/North Koreans could be stealing data from their systems….. all this because chip manufacturers tried to outdo each other by putting speed of processing above security.

The best article I’ve read that explains how Speculative Execution works and how these vulnerabilities could be exploited can be found here: http://frankdenneman.nl/2018/01/05/explainer-spectre-meltdown-graham-sutherland/

It seems that at the moment the only way to minimise your exposure to potential exploits is to patch the OS or Hypervisor, however this isn’t without issues as people have started reporting that it adds an overhead to performance. In all honesty, I doubt personal users will notice a performance hit on their day to day usage (home/office applications or games), it will however impact anyone that undertakes high IO or system-call intensive applications (such as DBs, email, Big-data/data-mining)… a performance hit of between 5-30% depending on application!!

VMware have stated that at present they don’t believe Meltdown to be an issue to their products because ESXi does not run untrusted user mode code, and Workstation and Fusion rely on the protection that the underlying operating system provides. For Spectre, they have released an article detailing their response to the issues and 2 Security Advisories which addresses the vulnerabilities and how they can be mitigated, VMSA-2018-0002 has been superseded by VMSA-2018-0004.

From what I can see, the first Security Advisory consists of security patches to ESXi that addresses the vulnerability to mitigate against leakage from the hypervisor or guest VMs into a malicious guest VM – these were patches made available late last year before the news broke (which makes you wonder how long the industry have known about it).

The second Security Advisory is a full minor update to vCenter (5.5, 6.0 and 6.5) in order to support both newer vSphere ESXi patches and Microcode/BIOS patches to hardware. This seems to be what they call “Hypervisor-Assisted Guest mitigation” which virtualises the new speculative-execution control mechanism for guest VMs so that a Guest OS can mitigate leakage between processes within the VM – and this mitigation requires specific microcode patches from platform vendors which seem to introduce these new ‘speculative-execution control features’. More information on how to apply this Security Advisory can be found here: https://kb.vmware.com/s/article/52085.

Note: The update patches found in VMSA-2018-0004 will mean that these new CPU features will be exposed to Guest VMs and as such vMotion to ESXi hosts without the microcode or hypervisor patches applied will be prevented. However, if you have an EVC cluster, it looks like vCenter will suppress the new features from VMs to enable vMotion compatibility until all hosts have been upgraded (after which it will enable those features) – unpatched hosts will not be allowed to join an EVC cluster that has been patched.

It’s worth noting that Guest VMs should also have their OS updated with the latest security patches for effective mitigation of these known vulnerabilities!

Finally, VMware have released an article regarding these vulnerabilities and whether their virtual appliances are affected: https://kb.vmware.com/s/article/52264. It currently looks like vSphere Integrated Containers and vRealize Automation have not been patched yet.

vSphere Central – new resource centre

A little while back I caught the vSphere blog about vSphere Central being launched and ended up bookmarking the portal to have a look at a later date. I had totally forgot about it till today when I needed to look up the PSC topology diagrams and Google sent me to the new vSphere 6.5 Topology and Upgrade Planning Tool (more on this later). Turns out this portal is exactly like Storage Hub (resource portal for everything vSAN, SRM and storage related)!

Everything technical you need to know about vSphere and vCenter can be found on this portal:

  • How to install vCenter and vSphere
  • How to migrate to vCSA
  • How to upgrade vCenter and vSphere
  • vCenter and PSC architecture
  • SSL certificate management
  • PSC Deployment Types
  • Product Interoperability Matrix
  • All the new features in 6.5 explained (vCenter HA, Backup/Restore, etc)

It really is a great resource portal, and even better you can download each section as a PDF! Beats the documentation site for vSphere as it’s far more easier to navigate!

The content is in a range of formats, most of it is text taken from the technical pdf documents, but there are videos and walkthrough demos also scattered throughout the topics.

One of the things launched with vSphere Central was the vSphere 6.5 Topology and Upgrade Planning Tool.

This tool aims to help customers plan and execute both upgrades to vSphere 6.5 as well as new deployments. With this initial release, the tool is focused on the most common upgrade paths and deployments of vCenter Server 6.5. The tool works by asking a series of questions while providing some guidance along the way to help answer those questions eventually making some recommendations on topology and upgrade and deployment steps.

In the past I used to refer to the VMware KB on deployment topologies: https://kb.vmware.com/kb/2147672

Some of the guys in the vSphere technical marketing team then came up with the PSC Topology Decision Tree which was a large poster – https://blogs.vmware.com/vsphere/2016/04/platform-services-controller-topology-decision-tree.html

This tool was inspired by the Decision Tree poster and extends its capability.

What I especially like about the tool is that after answering a series of questions regarding how I’m planning to design the vCenter/PSC deployment it gives me a recommended Topology diagram and then explains the steps to go about deploying the solution:

topology

Anyways, it’s a great tool…. and the portal is a brilliant collection of resources! Go use it! Bookmark it now…! =)

Horizon Cloud on Azure – GA

Interesting tie up between VMware and Microsoft…. is this the beginning of a new relationship? Have Microsoft woken up (post-VMworld) to the awesome VMware Cloud on AWS and realised they also want in on the party? (although if i’m honest this partnership has been bubbling in the background for a while now).

Anyways, after a round of beta testing in the US, Horizon Cloud has now gone GA on Azure: https://blogs.vmware.com/euc/2017/10/vmware-horizon-cloud-on-microsoft-azure-now-available.html

When Horizon Cloud was launched earlier this year, the concept of enabling end-user organisations the ability to deploy feature-rich VDIs and applications across multiple deployment options was very promising. From a single management console, end users are able to deploy virtual desktops onto on-premise infrastructure, to the Cloud, or a hybrid combination of both. TBH, some of this concept was already available in Horizon Air (which came out of the Desktone acquisition), but this is an evolution of that product.

Horizon Cloud is a cross-cloud architecture for VDI – much like how Cloud Foundation is for SDDC – however, in the case of Horizon Cloud, the Portal which acts as the control/management plane resides solely in the cloud (you get a choice with Cloud Foundation’s SDDC Manager), administrators log into this portal to deploy and manage their VDI sessions – whether on prem or cloud.

3 offerings currently:

  1. Horizon Cloud Hosted – so VDI infrastructure provided by VMware (IBM Cloud is currently the only provider), where you just choose the type of desktop and apps to deploy via the portal – DaaS. Infrastructure management/maintenance/SLAs are fully undertaken by VMware.
  2. Horizon Cloud On-Premise – based on HCI technology and acts like a stepping stone to Cloud VDI. VDI stored locally on prem, but management is all from the Cloud, perfect for data-residency issues, for end-users who require high performance VDI, and for IT admins who wish to have greater control over their VDI infrastructure.
  3. Horizon Cloud on Microsoft Azure – delivering RDS VDI and apps hosted in Azure datacentres. Connecting a customers Azure IaaS subscription to Horizon Cloud. So VMware manages the VDI aspects and Microsoft the underlying infrastructure. Also worth noting that currently only Azure deployments support vGPU-accelerated infrastructure.

Whatever the deployment option, customers will get a VDI infrastructure that’s easily scalable (whether cloud or on-prem) and easy to deploy. The best part is you get the flexibility of subscription based pricing.

With Horizon Cloud on Azure, you can import gold images from Azure marketplace which will then be configured and deployed for Horizon.

One key element of the Horizon Cloud technology is justin-time (JIT) provisioning of virtual desktops and applications. Using the configurations made in the cloud-control
plane, Horizon Cloud leverages VMware App Volumes, User Environment Manager, and VMware Instant Clone technologies to assemble personalised virtual desktop and application environments when an end user logs in, giving IT administrators high flexibility in leveraging the infrastructure.

I like the idea that if I have a persistent VDI deployed in Horizon Cloud, then I can access that VDI or hosted apps whether I’m in the office or on the move (as long as there is data connectivity). I can start writing a document in the office, then leave it open mid-sentence as I leave the office, or jump on a train – I can even power off my endpoint device – then I can re-establish the session and carry on without any interruption… an Always On desktop!

I also like the idea that with Azure, I could deploy a VDI session to the datacentre in the UK, then as I jump on a flight to the US I can re-deploy that VDI session to an Azure datacentre in the US. Although, I’m not quite sure you can migrate live VDI sessions between datacentres yet – I haven’t seen any articles that say you can live-migrate VDIs (but one would think this would be the ideal end-goal).

 

Hopefully this new VMware-Microsoft partnership will lead onto Microsoft accepting to run Cloud Foundation on Azure (VMware Cloud on Azure) which will then give end users the freedom to move their workloads from on prem to either AWS or Azure!! Almost Cross-Cloud (just need GCP to step up).

VMworld 2017 US General Session Day 2

….. This update is a bit late going up because Tuesday evenings is 5-a-side footie for me…. =)

So what was the General Session on Day 2 all about… well it kicked off with a fireside chat between Pat Gelsinger and Michael Dell, answering a few questions that were submitted the previous evening from attendees. In my opinion there wasn’t any major revelations or probing questions asked/answered, what we do know is that Michael Dell likes Peanut Butter & Chocolate… =P

It’s interesting that Dell thinks that we’re in for some exciting times with AI and machine learning…. although he didn’t quite pin his flag like Zuckerberg and Musk recently… =)
The amount of data created from IoT is stupendous, and the possibilities of using that data are endless – however, companies need to start thinking about how to use the vast amounts of data they have to try and improve processes, products and services – if they don’t then they could be left behind (Just like Elastic Sky Pizza were)!

However, one of the more memorable quotes from Pat was that “Today is the slowest day of technological evolution of the rest of your life!” Great quote, and how true it is…. In IT we live in an ever-changing world!

One thing I did pick up on was VMware Skyline – a new and innovative support technology which will offer pro-active support for VMware solutions. It will consist of a Collector appliance that end-users deploy, it then sits there securely collecting environmental-data from different VMware components (such as configuration, performance, and product usage) whilst performing machine-learning analytics to ensure the overall solution functions correctly. If it detects any changes, events or patterns that will cause a deviation from best practices or validate designs then it will alert the customer. Skyline is aimed at improving support experience through data analytics.

Both Pat and Dell were then joined on stage by Rob Mee (CEO of Pivotal) – it was really interesting to hear that Pivotal Cloud Foundry was being used in over 50% of the Fortune 500 – I wonder what the percentage is in the UK FTSE? Pivotal has be “pivotal” (excuse the pun) in helping enterprises and their digital transformation – how to run legacy production workloads alongside developing new cloud-native applications, yet still providing the availability and security whilst also reducing cost! Pivotal Cloud Foundry addresses all these issues.

The biggest announcement of the day was the unveiling of Pivotal Container Services (PKS) – a partnership between VMware, Pivotal and Google Cloud. Pivotal has been working with Google for a while – Project Kubo – and now with the partnership with VMware, PKS will enable enterprises to deliver production-ready Kubernetes on VMware vSphere and Google Cloud Platform (GCP), with compatibility to Google Container Engine (GKE) – all secured by – yup you guessed it – NSX. Pat went on to say that they’re not stopping there and will start to integrate other VMware products such as vRealize Automation and Operations, along with Wavefront (who VMware acquired in May – it’s a “real-time metrics monitoring and streaming analytics platform designed for developers to optimize their clouds and modern applications that rely on containers and microservices”)

PKS-Image

Pat, Dell and Rob were joined on stage by Google’s Sam Ramji (VP of Product Management – Developer Platforms). It’s interesting how Google are pouring in vast amounts of their knowledge on containers into Kubernetes – a way of giving back to the community! Sam also announced that Pivotal and VMware were to become as Platinum Members of the Cloud Native Computing Foundation – home of Kubernetes.

The rest of the General Session involved a fictitious company called “Elastic Sky Pizza” which was stuck in the past and needed assistance in transforming their business. Loads of demos and presentations showing how VMware’s suite of Cloud products work – including PKS, AppDefense, NSX, Pulse IoT (Edge LIOTA).

 

PKS looks amazing, but I do still think that containers is an enterprise play – which kind of goes against one of the questions during the fireside chat about looking after the SMB market… this is probably going to be priced above what SMBs can afford!

VMworld 2017 US General Session Day 1

If like me, you’re stuck in a sweltering London enjoying the bank holiday and watching the Game of Thrones season 7 finale, you may have forgotten that over in Vegas the city is just getting over the big fight of Mayweather vs McGregor and is now inundated with people looking to attend VMworld 2017 US.

It’s great that VMware live stream their keynotes, as it gives everyone an opportunity to hear first hand what VMworld will be about this year and also what is being announced!

And it’s of no surprise that VMware have continued to strengthen their vision on “Any Device, Any Application, Any Cloud” with the keynote by Pat Gelsinger. Whilst heterogeneous is a great thing that leads to the consumerisation of IT, it plays havoc with IT admins who’s key focus is to contain and secure a company’s data – and it’s worth noting how much emphasis is being placed on security within VMware – NSX is intrinsic to every solution that was mentioned during the keynote!

Vision

The first thing that was covered was how the digital transformation is affecting end users – the goal for any company is to ensure that their employees are well connected, yet the challenge is a complex one when you realise how many different technologies an end user has access to – smartphones, tablets, laptops – even smartwatches and cars now! So how do you deliver an unified workspace securely across multiple technologies?

Simple – Workspace ONE – piecing it all together to give companies a “consumer simple but enterprise secure” solution. Delivered in 3 areas:

  1. Apps and Identity – applications with a consistent feel across multiple devices. Secured by a common identity framework with a simple Single Sign-on experience.
  2. Management and Security – IT in control, delivering consistent management & security. Drastically improving  tasks that were previously costly, time consuming, and resource intensive, whilst still in control of data by combining identity and device management to enforce Data Security and Endpoint Compliance.
  3. Desktop and Mobile – Device Management and Compliance provided by AirWatch Unified Endpoint Management, protecting sensitive data as well as conditional access to how that data can be consumed by end-users.

workspaceone

Next Pat went on to explain that virtualisation has led to end-users deploying a private cloud within their own data centres, yet making such a transition is not an easy step – deployment isn’t straight forward, lifecycle management and day 2 operations isn’t always easy, and trying to secure different technologies of a private cloud is painful!

VMware’s goal is to “make Private Cloud Easy and that’s where Cloud Foundation comes along – a fully integrated SDDC stack that ‘just works’…. simple… agile… secure! version 2.2 was announced and is now GA.

Pat was then joined by Andy Jassy, CEO of AWS, to announce the General Availability of VMware Cloud on AWS. Announced as a tech preview at last years VMworld, it should be noted that it’s currently only available today in the US West Coast region Availability Zone, it will then be rolled out across the East Coast AZ before rolling out to the rest of the AWS global AZs by the end of 2018. So I guess we’re going to expect it in the UK late 2017/early 2018!

VMware Cloud on AWS allows you to seamlessly take a workload running on vSphere in your data centre and migrate it to AWS Public Cloud running a VMware stack – using the same tools (vCenter Server) to manage both your private and your public cloud workloads from a single pane of glass! A consistent feel no matter where your workload resides. What Andy Jassy said was correct – in the past customers hated the fact that if they wanted to consume public cloud, there was no easy way of migrating workloads across without some form of translation occurring. It was also painful and costly to manage as you couldn’t use a single tool to manage both private and public cloud.

VMware’s Cloud Strategy is as follows:

cloud

The first 7 VMware Cloud Services were announced as available for consumption.

VMware Cloud Services

NSX Cloud is an interesting service that addresses networking and security operational challenges inherent with using multiple public clouds. Unfortunately at launch it’s only available on AWS to protect EC2 workloads (ie native AWS workloads – not vSphere workloads which is what VMware Cloud on AWS gives). It differs from on-premise NSX as it is delivered as a service and managed by VMware.

As I previously said, NSX is a key foundation to every solution at VMware currently:

nsx

Security is hugely important… and Pat breaks it down into 3 components:

  1. the need to build it into the infrastructure
  2. the need to integrate with the current security vendor ecosystem
  3. the need to ensure good cyber hygiene and ensure security policies are in place. The 5 pillars of Cyber Hygiene are:
    • Least Privilege
    • Micro-segmentation
    • Encryption
    • Multi-factor authentication
    • Patching

Two years ago, VMware first began talking about the concept of the “Goldilocks Zone” where the hypervisor sits at the ideal location in the network to improve security. During the keynote VMware announced a new product named AppDefense which looks to be the fruition of Project Goldilocks.

AppDefense allows a virtual machine to learn its manifest and understand what is a good and secure process, it’s then able to determine whether the runtime behaviour of a VM or application deviates from its intended state. Finally it’s able to trigger an automated/orchestrated response to remediate or quarantine any detected anomalies.

appdefense

 

Strange that searching the VMworld Europe Content Catalog for AppDefense doesn’t bring up any sessions…. which is a shame as I was hoping to schedule a session after hearing the keynote and reading about it.

Roll on Day 2….

 

Goodbye vCenter Server for Windows and Flash-based vSphere web client!

Hmm…. it’s not even VMworld yet and VMware decide to make 2 big-ish announcements.

Although tbh, since vSphere 6.5 was released these 2 announcements have long been coming!

Finally, after loads of speculation, VMware had announced that vCenter Server for Windows and the Flash-based vSphere web client is to be deprecated with the launch of the next version of vSphere. Updates to 6.5 will continue supporting the 2 features, but come vSphere 7.0 it will be no more….

https://blogs.vmware.com/vsphere/2017/08/farewell-vcenter-server-windows.html

 

“vCSA-exclusive capabilities such as file-based backup and restore, unified update and patching, native vCenter High Availability, and a significant performance advantage mean that the VCSA has become the platform of choice for vCenter Server.  Additionally, due to the integrated nature of appliance packaging, VMware is able to both better optimize and innovate vCenter Server at an accelerated pace.  Finally, with the VCSA, VMware can provide support for the entire vCenter Server stack including the vCenter Server application, the underlying operating system (Photon OS), and the database (vPostgres). By doing so, VMware can ensure that customers can focus on what matters most while having a single source for updates, security patches, and support.  The VCSA model is simply a better model for vCenter Server deployment and lifecycle management.”

That pretty much sums up why VMware are 100% behind the vCSA, although they miss out the whole “screw you Microsoft licensing!!” part! Plus given that 6.5 ships with a migration tool that helps you move/upgrade from a Windows vCenter to an Appliance vCenter, it’s no surprised that more and more people are moving over when it comes round to upgrade time!

In fact ever since 6.5 was released, I’ve not even deployed a single Windows vCenter Server – all my customers have been moved over to the vCSA.

https://blogs.vmware.com/vsphere/2017/08/goodbye-vsphere-web-client.html

With regards to the vSphere Web Client, loads of people found the flash-based version was frustratingly difficult to use – it was slow, it was notoriously prone to crashing and frankly it was based on in-secure Flash technology (not to mention that Adobe themselves are dropping flash). HTML5 is the way to go baby!

So with those announcements in mind….. I may think about changing some of my VMworld sessions to jump on the vCSA and Web Client update sessions!!

 

RIP…..

vExpert 2017 Announcements

Congratulations to those who have been recognised as vExperts and will be joining the group for the 2nd half of 2017!

https://blogs.vmware.com/vmtn/2017/08/vexpert-2017-second-half-announcement.html

Also congratulations to existing vExperts who have been recognised as experts in the 2 sub categories – NSX and vSAN!

https://blogs.vmware.com/vmtn/2017/08/vexpert-nsx-2017-award-announcement.html

https://blogs.vmware.com/vmtn/2017/08/vexpert-2017-vsan-announcement.html

 

I’m fortunate enough to be once again considered as a vSAN vExpert for this year! =)

Top vBlog 2017 Full Results

And after a couple of weeks of counting, the results of the Top vBlog for 2017 (sponsored by turbonomic) has been released by www.vsphere-land.com:

http://vsphere-land.com/news/top-vblog-2017-full-results.html

Congratulations to all 273 bloggers that make up the Top vBlog table!

Big congratulations to William Lam who takes the top spot this year… a great website I always refer to when in need of some CLI scripts to get some work done!

Also congrats to Vladan Seget and Cormac Hogan who finish 2nd and 3rd…. Vladan’s blog is particularly useful for those who are just starting out on their VMware journey as he covers all aspects of the VMware portfolio – and as an independent blogger he dedicates quite a lot of time in writing concise articles for his readers! Cormac as always is a great source of everything storage related within VMware, particularly vSAN – I always flick between him and Duncan Epping for up-to-date info around vSAN and storage!

This year I fortunate to again rank in the Top vBlog list, dropping 16 spots to 177…. which is still mighty impressive given I don’t dedicate as much time as I would like to blogging! Thanks to everyone who voted for me! =)

However, there are so many quality blogs out there which didn’t make it into the top vBlogs lists for some reason…. the likes of:

  • Emad Younis, who has great content on vCSA and migration.
  • Chanaka Ekanayake, who like me works within the Channel and has great content on VMware as well as other vendors.
  • Simon Todd, who’s a VMware SABU SE aligned to EMEA… some good stuff on vSAN.
  • Marco van Baggum, who writes some interesting articles on NSX (and also helps run the vExpert slack space)

It’ll be wrong not to also mention some vExperts who aren’t on the list like Christian Parker, Mathew Bunce….

… and the list goes on…..

Anyways, congrats to all…. we’re all winners as they say!

HCIBench 1.6.2 – Testing vSAN performance

Over the past month or so I’ve been running a number of performance tests on VxRail and vSAN solutions.

HCIBench is a brilliant tool to help end-users understand the type of performance that they can achieve with their vSAN solution.

It’s essentially an automation wrapper around the popular Vdbench tool. Vdbench is an utility specifically created to help engineers and customers generate disk I/O workloads to use for validating storage performance and storage data integrity. Vdbench is a complex beast to run, with lots of different variables that can be configured via CLI… so the HCIBench wrapper helps simplify workload profiles and makes it so much easier to run benchmark tests!!

Please note, HCIBench is a VMware Labs Fling and so there’s limited support available and it shouldn’t be used in production environments (although the latter is just to cover themselves). If I’m honest, the creators of HCIBench are pretty good at replying to comments and feedback!

https://labs.vmware.com/flings/hcibench

It’s definitely worth remembering that as a benchmark tool, it can’t quite simulate real-world workloads! However, if you understand how your workload behaves (ie block size, read/write ratio, etc) then you can get pretty close to creating a workload profile that matches your workload (albeit running a test at max. workrate rather than the bursty rate we see in real-life).

 

HCIbench was updated 2 days ago in response to the recent release of vSphere 6.5u1, and in my opinion is even cooler now that it can utilise the new vSAN Performance Diagnostic feature of vSAN 6.6.1 (API integration with the new Performance Diagnostics part of vSAN Cloud Analytics).

You can now run an HCIBench test and view detailed results of the test in Performance Diagnostics with supporting graphs – you’re able to select a goal for the test based on “Max IOPS”, “Max Throughput” or “Min Latency”, and then get details on potential issues found in the analysed data which you can then use to improve the workload profile you’re using in HCIBench.

Point your browser here for more info:
https://blogs.vmware.com/virtualblocks/2017/07/31/what-to-expect-from-hcibench-1-6-2/

Note: You need to have Customer Experience Improvement Program(CEIP) and vSAN Performance Service turned on to get this feature enabled