What’s new with VMware vSAN 6.5?

Given that I’m a VMware vExpert for vSAN, I guess I’m kind of obliged to write about what’s new with the latest iteration of vSAN – 6.5….. =)

vSAN 6.5 is the 5th version of vSAN to be released and it’s had quite a rapid adoption in the industry as end-users start looking at Hyper-Converged Solutions. There are over 5000+ customers now utilising vSAN – everything from Production workloads through to Test & Dev, including VDI workloads and DR solutions! This is quite surprising considering we’re looking at a product that’s just under 3 years old… it’s become a mature product in such a short period of time!

The first thing to note is the acronym change…. it’s now little ‘v’ for vSAN in order to fall in line with most of the other VMware products! =)

So what are the key new features?


This is probably the most useful feature in 6.5 as it gives you the ability to create iSCSI targets and LUNs within your vSAN cluster and present these outside of the vSAN Cluster – which means you can now connect other VMs or physical servers to your vSAN storage (this could be advantageous if you’re trying to run a MSCS workload). The iSCSI support is native from within the VMkernel and doesn’t use any sort of storage appliance to create and mount the LUNs. At present only 128 targets are supported with 1024 LUNs and a max. LUN size of 62TB.


It seems quite simple to setup (famous last words – I’ve not deployed 6.5 with iSCSI targets yet). First thing is to enabled the vSAN iSCSI Target service on the vSAN cluster, after that you create an iSCSI target and assign a LUN to it… that’s pretty much it!

Great thing about this feature is because the LUNs are basically vSAN objects, you can assign a storage policy to it and use all the nice vSAN SPBM features (dedupe, compression, erasure-coding, etc).

2. 2-node direct connect for vSAN ROBO + vSAN Advanced ROBO

Customers find it quite difficult to try and justify purchasing a 10GbE network switch in order to connect together a few nodes at a ROBO site. VMware have taken customer feedback and added a new feature which allows you to direct connect the vSAN ROBO nodes together using a cross-over network cable.

In prior versions of vSAN both vSAN traffic and witness traffic used the same VMkernel port which prevented the ability to use a direct connection as there would be no way to communicate with the witness node (usually back in the primary DC where the vCenter resides). In vSAN 6.5 you now have the ability to separate out vSAN and witness traffic onto separate VMkernel ports which means you can direct connect your vSAN ports together. This is obviously great as you can then stick in a 10GbE NIC and get 10Gb performance for vSAN traffic (and vMotion) without the need of a switch!


The only minor issue is you need to use the CLI to run some commands to tag a VMkernel port as the designated witness interface. Also the recommended setup would be to use 2 VMkernel ports per traffic flow in order to give you an active/standby configuration.


It’s also worth noting that the new vSAN Advanced ROBO licenses now allow end-users to deploy all-flash configurations at their ROBO site with the added space efficiency features!

3. vSAN All-Flash now available on all license editions

Yup, the All-Flash Tax has gone! You can now deploy an All-Flash vSAN configuration without having to buy an advanced or enterprise license. However, if you want any of the space saving features such as dedupe, compression and erasure coding then you require at least the Advanced edition.

4. 512e drive support

With larger drives now coming onto the market, there has been a request from customers for 4k drive support. Unfortunately there is still no support for the 4k native devices, however there is now support for 512e devices (so physical sector is 4k, logical sector emulates 512bytes).

More information on 4Kn or 512e support can be found here: https://kb.vmware.com/kb/2091600

5. PowerCLI cmdlets for vSAN

New cmdlets are available for vSAN allowing you to script and automate various vSAN tasks (from enabling vSAN to the deployment and configuration of a vSAN stretched cluster). The most obvious use will be using cmdlets to automatically assign storage policies to multiple VMs.

More info on he cmdlet updates available here: http://blogs.vmware.com/PowerCLI/2016/11/new-release-powercli-6-5-r1.html

6. vSAN storage for Cloud Native Apps (CNA)

Integration with Photon means you can now use a vSAN cluster in a CNA enviroment managed by Photon Controller. In addition, now that vSphere Integrated Containers (VIC) is included with vSphere 6.5, you can now use vSAN as storage for the VIC engine. Finally Docker Volume Driver enables you to create and manage Docker container data volumes on vSAN.

For more information about vSAN 6.5, point your browsers to this great technical website: https://storagehub.vmware.com/#!/vmware-vsan/vmware-vsan-6-5-technical-overview

Deploying VSAN 6.1 ROBO

One of the things I’m fortunate to have access to at MTI Technology is the Solution Centre which has all sorts of kit that can be used for demos and for consultants to play around with.

After coming back from VMworld, one of the things I really wanted to test out was how easy it would be to deploy VSAN 6.1 in a ROBO solution. Fortunately I had a pair of old Dell R810s lying around and managed to cobble together enough disks and a pair of SSDs in order to create two VSAN nodes!

VSAN ROBO allows you to deploy a 2-node VSAN cluster (rather than the standard 3-nodes) with a Witness Server located on another site – usually this would be your primary data centre (as per diagram below). It also allows several ROBO deployments to be managed from a single vCenter Server. VSAN ROBO uses the same concepts as VSAN Stretched Cluster, using Fault Domains to determine how data is distributed across the VSAN nodes. The Witness Server is uniquely designed with the sole purpose of providing cluster quorum services during failure events and to store witness objects and cluster metadata information and in so doing eliminates the requirement of the 3rd physical VSAN node.


Note: Whenever you deploy any VMware product into a production environment, make sure that you check the Hardware Compatibility List!
In my case for VSAN, neither the server nor the storage controller in the R810 was supported – but as it was only a demo environment it wasn’t of top priority.

Before I go through how I configured VSAN ROBO, there are a few things I need to state upfront which I don’t recommend you doing in a production environment:

  1. Using the same subnet for the VSAN network – in my demo environment I only have 1 subnet, so I’ve had to stick everything on the same VLAN. Ideally you should separate out the VSAN traffic away from the Mgmt and VM traffic.
  2. Using a SSD from a desktop PC for the Cache drive – ideally this should be an enterprise grade SSD as VSAN uses the SSD for caching so you really need one that has a higher endurance rate.

Also there are a few features that are not supported in the ROBO solution (but available in standard VSAN):

  • SMP-FT support
  • Max value for NumberOfFailuresToTolerate is 1
  • Limit of 3 for the number of Fault Domains (2 physical nodes and the witness server).
  • All Flash VSAN.
  • VSAN ROBO licensing is purchased in packs of 25 VMs, with 1 license per site. This means a maximum of 25 VMs can be licensed per site! However, 1 pack can be used across multiple ROBO sites (so 25 VMs across 5 sites).

From a configuration perspective, configuring a VSAN Cluster for ROBO is extremely simple as it is performed through a wizard within the vSphere Web Client. From a network perspective, the two VSAN Cluster nodes are to be configured over a single layer 2 network with multicast enabled. There are a few requirements for the Witness Server:

  • 1.5 Mbps connectivity between nodes and witness
  • 500 milliseconds latency RTT
  • Layer 3 network connectivity without multicast to the nodes in the cluster


So for my demo environment, I have 2x R810s with 1x Intel Xeon X6550 and 32GB RAM. For my SSD I found an old 240GB Micron M500 SSD (MLC NAND flash) and stuck it into a Dell HD caddy, for my HDDs I have 5x 146GB SAS drives. The Witness server resides within my main VMware environment (which runs on UCS blades and a VNX5200).

I won’t go into how I installed vSphere ESXi 6.0 u1….. however, just remember that you’ll need to install ESXi onto an SD or USB drive as you want to use all local drives for VSAN (in my case I installed ESXi onto a 8GB USB drive).

I created a new VMware cluster within my vCenter and added the 2 VSAN nodes. I then deployed the Witness Server, which in my case was the nested ESXi host within a virtual appliance. There’s actually 3 sizes for the Witness Appliance – Tiny, Medium, Large. I deployed a Medium appliance. vsan1a vsan2vsan3

I won’t step through how to deploy the OVA as it’s pretty routine stuff. If you load up the console for the Witness server, you’ll be greeted with the familiar DCUI of vSphere ESXi.

Once it’s deployed and configured with the relevant IP address and hostname, you can add the Witness server into your vCenter Server as just another ESXi host.

vsan5 vsan6

One thing that’s slightly different is the Witness Server comes with its own vSphere license and so doesn’t consume one of your own licenses. Note that the license key is censored so you can’t use it elsewhere!

Once the Witness Server has been added to the vCenter Server you may find that there is a warning on the host which says “No datastores have been configured”

This occurs because the nestled ESXi host does not have any VMFS datastores configured, the warning can be ignored, but if you’re like me and hate the exclamation mark warnings in your environment you can easily get rid of the warning by adding a small 2GB disk to the witness appliance VM (Editing the Hardware settings) and then adding a datastore on top of the new disk.

You should be able to notice that the icon for the witness appliance within the vCenter Server inventory is slightly different from your physical hosts – it’s shaded light blue to differentiate it from standard ESXi hosts.

The next step is to configure the VSAN network on the witness server. There is already a portgroup pre-defined called witnessPgDo note remove this port group as it has special modifications to make the MAC addresses on the network adapters match the nested ESXi MAC addresses!
There should be a VMkernel port already configured in the portgroup, edit the port and tag it for VSAN traffic.
vsan11 vsan12

At this point, ensure that your witness server can talk to the VSAN nodes.

Note: Typically an ESXi host has a default TCP/IP stack and as a result only has a single default gateway – more often than not, this default route is associated with the management network TCP/IP stack. In a normal deployment scenario, the VLAN for the management network would be isolated from the VSAN network, as such there is no path between the two networks and no default gateway on the VSAN network. A way around this problem is to use static routes to define a routing entry which indicates which path should be used for traffic between the witness server and the VSAN nodes. I won’t go into configuring static routes, you can find more detailed information in the VSAN 6.1 Stretched Cluster Guide.

Once your witness server is talking to the VSAN nodes, it’s time to configure the VSAN ROBO solution. This is as simple as creating fault domains.

I won’t go into how to turn on the VSAN cluster and disk management as this is simple stuff and has been covered off in numerous other VSAN blogs/guides. One thing I will mention is that because I have 2 very old servers, I had to configure each individual disk as a RAID-0 set as the RAID controller in the server did not support pass-through. Once configured and detected by the ESXi host as storage devices, I then had to manual set the SSD device as a Flash Disk:

I also ended up manually claiming the disks for VSAN.


Once the 2 nodes have been configured for VSAN, next comes the creation of the Fault Domains. As previously mentioned, VSAN ROBO works by creating 2 Fault Domains and a witness server – just like you would for a VSAN stretched cluster. However, in this case only 1 server is assigned to each fault domain.

vsan15 vsan16 vsan17 vsan18 vsan19

Note: You probably have noticed that the wizard still states “VSAN Stretched Cluster” on all the screens, unfortunately VMware didn’t write separate code for VSAN ROBO, so it’s still classed as a stretched cluster.

Once VSAN ROBO has been deployed you can check the health of the VSAN by selecting the cluster and Monitor->Health.
vsan20 vsan20a
The first warning is regarding the VSAN HCL, and shows that my server and its RAID controllers are not listed in VMwares’ VSAN HCL. =)

Next license the VSAN ROBO cluster, note what features get switched off when licensing for VSAN ROBO.
vsan21 vsan22

There is already a default VSAN storage Policy, creating a VM and assigning this policy gives a Failure To Tolerate of 1. Viewing the Physical Disk Placement you can see that data is mirrored on the 2 VSAN nodes with metadata stored on the Witness Server.

Something I found very useful was the “Proactive Tests” option for VSAN which provides the ability to perform a real time test of cluster functionalities and dependencies – creating a small VM, checking network multicast between hosts plus storage IO.




Voila…. a basic VSAN ROBO deployment…..

Don’t forget to download the Storage Management Pack for vROps so you can get an in-depth view of your VSAN deployment from within vROps:

VMware Virtual SAN Ready Node – Certified server configs

Last month, VMware released a document which highlighted certain server configurations from its various OEM partners that are Virtual SAN ready. What this means is the server configurations have been tested and certified by VMware and the OEM partner for running Virtual SAN for several workloads.

All the big name server vendors are listed – Dell, HP, Cisco, Lenovo. Even the smaller vendors like Supermicro, Fujitsu, Hitachi and Huawei have got their boxes certified!
Notice that most of these guys are Qualified EVO:RAIL Partners (QEP)? =)

This is a great document if people are just looking for a server solution to run a virtual workload (say 100 VDI sessions or 50 VMs) using Virtual SAN as you can just use a single SKU code to order a certified box from the OEM Partner.

Click to access Virtual%20SAN%20Ready%20Nodes.pdf

This brings about another question – given that EVO:RAIL HCI Appliances seem to be so expensive, what’s stopping end users just deploying a 4-node Virtual SAN Ready Node and licensing accordingly? Pretty sure this will be a lot cheaper. Only issue is you don’t get the automated deployment provided by the really COOL EVO:RAIL Manager!

For a start, it will be a lot cheaper scaling out as you just purchase 1 server node, compared to a whole new EVO:RAIL Appliance!

VMware Online Technology Forum on Now!

VMware Online Technology Forum has started…. are you attending?

Live presentations on all the new goodies from VMware – vSphere 6, vRealize Automation/Operations, virtual SAN 6, App Volumes, vCloud Air, EVO:Rail…….

If you can’t attend today, then the content will be made available on demand from tomorrow (16th April):