VMworld 2016 US – Day 1 General Session Overview

So the replay of yesterdays Day 1 General Session is now online:

The biggest announcement is the tech preview of Cross-Cloud Architecture. This is obviously VMware’s next step in their “Any Cloud, Any Application, Any Device” vision.

According to VMware:
“This architecture extends VMware’s hybrid cloud strategy, enabling customers to run, manage, connect and secure their applications across clouds and devices in a common operating environment. VMware Cross-Cloud Architecture is delivered through VMware Cloud Foundation, a new set of Cross-Cloud Services VMware is developing, and VMware vRealize Cloud Management Platform.”

This new architecture gives customers a set of tools to manage their virtual estate both on-premise and off-premise across multiple clouds – a single pane of glass to manage VMs on the likes of AWS, Azure, Google, as well as vSphere clouds.

Most customers already utilise multiple clouds (unbeknownst to IT) and this new architecture will enable IT to resume control of what is out in the cloud – allowing network and security policies to be applied to workloads being deployed in the cloud. In addition to allowing migration between clouds!

Much like how vSphere ESXi was used to allow you to span multiple server hardware vendors (HP, Dell, IBM), and how NSX allows you to span multiple network hardware vendors (Cisco, Arista, Brocade), VMware Cross-Cloud Services will offer a common platform to overlay your cloud vendors to offer you the ability to deploy your applications across clouds without having to mess around with the underlying cloud services (which are inherently different depending on cloud vendor)!

VMware Cross-Cloud Services will centralize management, operations, networking, security and data management.


It looks like the common Network & Security piece will be handled by NSX – which will include a forthcoming feature called Distributed Network Encryption (DNE).

The Management and Visibility piece will be SaaS based (a cloud service) and allows you to connect your existing public cloud accounts to ingest those workloads into the management platform, it will then show you cost and utilisation across your clouds and allow you to deploy applications across clouds.

The other major announcement was the new VMware Cloud Foundation offering which basically bundles vSphere, VSAN and NSX into a single, fully integrated, SDDC stack that can be provisioned on premise or be run as a service in the cloud.

To quote VMware:
“VMware Cloud Foundation is a next-generation hyper-converged infrastructure for building private clouds that for the first time combines VMware’s highly scalable hyper-converged software (VMware vSphere and VMware Virtual SAN) with the world’s leading network virtualization platform, NSX. Cloud Foundation provides a consistent multi-cloud IaaS that is simple to deploy, operate, and maintain, and gives applications a consistent, scalable and highly available infrastructure services, regardless of where they run”

“The goal of Cloud Foundation is to be able to provision cloud infrastructure like you provision VMs.”


In addition to this announcement was the partnership with IBM Cloud to offer VMware Cloud Foundation as a service.

The key to the Cloud Foundation is the SDDC Manager which will be the tool for IT admins to build and maintain their cloud (making use of a lot automation policies to build the cloud and deploy workloads)

Other announcements include vCloud Availability for vCloud Director, which enables customers to leverage the vCloud Air Network ecosystem (ie VMware Partner cloud solutions) for simple, automated disaster recovery as a service (DRaaS) – much like the offering from vCloud Air DR.

VMware vCloud Air Hybrid Cloud Manager has added several major enhancements, including zero-downtime, bi-directional application migrations in and out of vCloud Air. This includes the migration of NSX security policies, providing simple migration of workloads to vCloud Air with no need for any network or security reconfiguration once the migration completes.


It seems that the main takeaway from yesterdays general session was that now it’s time to take back control of your cloud.


vCloud Director Convergence and Transition Plan

Very good blog post regarding how vCD will be integrated with vSphere and vCAC…..

So it seems vCD is being phased out of Enterprise customers with vSphere handling the cloud infrastructure and vCAC the automation and cloud management.

It sounds like vCD will become a VSPP product specifically aimed at service providers…..

Troubleshooting a vCloud Director Installation

The problem about working full time is it’s really hard to find time to blog, and also to find topics to blog about! =)

One of the great things about my job is we have a solution centre in the office which allows me to play around with kit! =)
Our solution centre is based around an EMC VSPEX architecture….. so EMC VNX storage, Cisco UCS blades and VMware virtualisation!!

I’ve been busy the last week or so putting together a vCloud solution for some of the engineers to play around with, as well as finally completing the detailed installation guide for deploying the vCloud Suite (one of these days I promise I will post it up).

Anyways, so I ended up installing two RHEL 6.2 VMs as my vCD cells on a MS SQL 2008 R2 DB, load-balanced using a vCNS edge….. but when I tried to start the vCD services on my linux VMs, they would say they’ve started (simple service vmware-vcd status command) but wouldn’t give me the vCD web console/UI….. all I got was a Blank Grey Webpage and after a while it would error out saying it couldn’t connect to the website!! Hmmmm……

Anyways, this gave me a good opportunity to test out my troubleshooting skills and offer a topic for my blog! =)

So here goes……

Troubleshooting vCD….

The Log files for vCloud Director are located at /opt/vmware/vcloud-director/logs. There are three main files to look at (well there’s more than 3 but these are the ones I usually use and 99% of the time I can work out what’s wrong):

1. cell.log

This log file provides information on the status of the vCloud Director cell services and the application as it starts up.
Use tail -f cell.log to view the live status when starting a vCloud Director Cell.
A successful start up will allow you to access the vCD web-console/UI and will display a started status for each service, plus 100% for Application Initialization.

Usually if there is an issue with accessing the web-front end UI then it is more than likely that the services are still waiting to complete, as below:


If you’re seeing lots of services showing a “WAITING” status, then check the other logs to determine what could be causing this issue.

2. vmware-vcd-watchdog.log

This log file shows any alerts, errors or information that the vCloud Director cell services maybe experiencing. A healthy vmware-vcd-watchdog.log looks similar to the below:


If there’s an issue, then you could get an ‘Alert’ entry, similar to the one below:


I believe vCloud Director will automatically try to re-start the services as I didn’t see a time stamp for an entry when I manually restarted the service. Also this log looks very similar to what you would get if you typed in ‘service vmware-vcd status‘ as that command reports on both the vmware-vcd-watchdog and vmware-vcd-cell services.

3. vcloud-container-info.log

This log file shows the status of the initial installation of vCloud Director and will log how the application is currently functioning. If you have any errors or failures during installation, this log file will provide you with the details required to troubleshoot the cause of the failure.
In addition, this log will also provide information on any errors that may cause the vCloud Director services to fail to start.
In my case, after doing a cat vcloud-container-info.log | more I discovered the following error:


Turns out that the error shows that the vCloud Director cell could not resolve its hostname in DNS.

When I went through the pre-reqs before installation, I realised that I had only put in DNS entries for the two IPs used for the HTTP and the Remote Console access….. I forgot to put an entry into DNS that resolved the hostname of the Linux VM to the HTTP IP address.
A quick edit to DNS and then a restart of the vCD services fixed the problem I experienced.

4. vcloud-container-debug.log

This log file shows the debugging information. The detail in this log file will be dependant upon the level of debugging set. I didn’t actually end up looking at this log as the error was discovered in the -info.log…. However, it’s another port of call if you can’t work out what’s causing your vCD services to fail.

Rights….. blog entry over…… I’m off to eat my dinner! =)

DO NOT disable DRS within a vCloud environment!!

I remember watching this video when going through my vCloud Director training, and stumbled across it again a few days ago… thought I’d share it with you all!


Basically by disabling DRS on your vCloud resource cluster you remove all resource pools in vCenter (unfortunate side effect of DRS).
Now vCloud Director relies heavily on these resource pools, in fact by disabling DRS you pretty much destroy your vCloud environment!! O_o”

When you create your vCloud, you usually create a Provider virtual DataCenter (PvDC) which is usually assigned to a HA cluster within vCenter. When you start to create Organisations and then the relevant resources you wish to assign to that Organisation, you create Organisation virtual DataCenters (Org vDCs – which basically is a pot of resources you’ve carved off the PvDC).
It’s these Org vDCs which are backed up by resource pools within vCenter, hence why if you disable DRS, you pretty much destroy all your Org vDCs within your cloud!!

There really isn’t an easy way around this (restoring a backup of your vCenter Server DB will go some way to repairing your vCloud)…..

As the video shows, whilst the VMs within a vApp will keep running if powered on (or warm booted), if you power them off then they die (because the resource pool it belonged to was destroyed)!
Plus you won’t be able to manage the vApp or doing anything within vCloud Director (like deploy a catalog template, power on another vApp, etc).

There’s a more indepth article by Chris Colotti that goes into what happens when you disable DRS:

Protecting your Cloud (vCloud & SRM)

So one of the BIG problems at the moment is that SRM does not fully support protecting your vCloud environment.

It supports protecting your management cluster (so the vCenter servers, vCD cells, vCNS manager, vCM, DBs, etc), but it doesn’t yet protect your resource cluster….. so all those VMs you’ve deployed in your organisations under vCD – well they’re not protected by SRM!

Definitely NOT COOL if your primary site goes tits up!!

From what I can gather, this is mainly due to the way SRM work….. When you setup SRM for DR, you have to ‘pre-create’ resources at the recovery site in order to map the resources from the protected site to them (stuff like resource pools, folders, network, placeholder VMs). Unfortunately vCD likes to have full control of a resource cluster and manages all the resource itself – this basically means that the vCD cells are not aware of the objects that have been created in the recovery site for SRM. It doesn’t matter if the names are the same, what matters is the Management object Reference IDs (MoRef ID) have changed and this is what vCD uses to construct its environment…..

MoRef IDs are used to correlate objects between vCD and the underlying vSphere/vCenter layer. Any changes to these identifiers will result in the loss of functionality because vCD will not be able to manage these objects as it will not be aware of them (ie the MoRef IDs will not exist inside the vCD DB).
The use of SRM would result in a change of the MoRef ID on the vCenter Server layer, resulting in an incorrect reference in the vCD database – and so leaving the object (eg. a VM) unmanageable from a vCD perspective. I believe SRM also re-signatures the storage volumes which will also confuse vCD.

About a year ago Chris Colotti and Duncan Epping wrote an article on how vCloud DR could be achieved, this involved the clever idea of putting the resource ESXi hosts at the recovery site into the same resource cluster as the resource ESXi hosts at the protected site (but in maintenance mode as obviously it won’t see the storage located at the protected site so can’t be used by vCD). Then using vSphere HA to take the ESXi hosts out of maintenance mode to handle the recovered workloads…. However, this solution did involved manual intervention to fail over the vCD resources correctly:

Earlier this year, another white paper was released which described how the majority of this manual process (ie the VMware bits) could be automated using PowerCLI:

However, what’s missing is the automation of the whole storage piece – breaking the replication and making the volumes read/write….. but then I guess this is really more storage-vendor dependent! =)
I guess if the storage vendor has exposed the array to VMware using VASA then it could be possible to script the storage steps as well….! =)

Anyways, it’s been an interesting read…… and definitely a problem I see VMware sorting out for the next release of SRM!

Given how powerful PowerCLI is, I really need to find some time to learn how to use it!!

Creating a Load Balancer in vCloud Director 5.1

So as promised, today I’m going to blog about how to manually create a load balancer service on an edge gateway within vCloud Director.

I’m assuming here that you know all about Edge Gateways and how to create them, so will by-pass that info – if you don’t know then VMware has a simple to follow video on creating a gateway: http://www.youtube.com/watch?v=v9XOOFhvDBk

(Note: with 5.1 you can now setup an edge gateway to run in HA mode – basically providing a secondary gateway device that can seamlessly take over if the primary gateway dies! Also worth noting is the multiple interfaces you can configure – now 10 are supported – and VXLAN support… for more info check out the release notes: http://www.vmware.com/support/vshield/doc/releasenotes_vshield_51.html. BTW, the latest version of vShield is 5.1.2).

So on an edge gateway within your Organisation vDC (virtual Data Centre), you can setup several gateway features (or services):

  • DHCP
  • NAT
  • Firewall
  • Static Routing
  • VPN
  • Load Balancing

I won’t go into each one otherwise this will end up being an extremely long post. For more info have a look at VMware’s video: http://www.youtube.com/watch?v=elG1zxGHheg

Creating a Load Balancer service on the edge gateway is a pretty simple process. The two main attributes that need to be configured are:

  • “Pool Servers” – which basically contains all the servers that you wish to load balance, as well as the protocol you wish to balance over.
  • “Virtual Servers” – this is basically where you assign a VIP (virtual IP) to the load balancer, determine which “Pool” of servers you wish to assign it to, and which protocols you wish to enable.

Step 1 – Configuring Load Balancer Service

When you navigate to the Edge Gateway tab within the Org vDC, right-click on the edge gateway you wish to configure and select “Edge Gateway Services”. This will pop up a window which allows you to configure all the services available on that gateway. In our case we’re configuring the Load Balancer, so click on that tab.


Step 2 – Configure Pool Servers

The first thing you need to do is configure the Pool Servers, it’s no use configuring the Virtual Servers as one of its requirements is that you assign the Virtual Server to a Pool….. Click on Add to bring up the Add Load Balancer Member Pool window.

Here you will just enter a Name for the Pool and a description. Try and use an unique and useful name (eg. <vApp Name>-LB-Pool) that helps to identify the Pool, this is because each load balancer service can have multiple pools and it could get confusing if you end up calling every pool “LBPoolxx”.


Next up is choosing what services/protocols are to be load balanced. One of the new things with vShield 5.1 (or vCloud Network and Security 5.1) is the ability to load balance over HTTPS and generic TCP connections (previous versions only allowed HTTP). Which is GREAT as you can now use an Edge gateway within vCenter Server to load balance vCloud Director cells!! (More on this another time).

So select the services you wish to balance and then decide what balancing methods you wish to use.

Here’s a quick rundown of balancing methods:

  • IP_HASH – This basically means the load balancer selects a server based on a hash of the source and destination IP address of each packet.
  • LEAST_CONN – This distributes the connection requests based on the number of connections already on the pool-member server. Basically new connections are sent to the server with the fewest connections! However, this does not take into consideration the amount of traffic being handled by that server. Usually great for load balancing long sessions (LDAP, SQL) but not that great for short sessions (like HTTP)
  • ROUND_ROBIN – Probably the most common algorithm to use (especially when the servers have equal processing capabilities), it allows equal distribution of traffic amongst the pool servers regardless of the number of connections (or response time). Basically each server in the pool is used in turn according to the weight assigned to it. Although be careful using this if you have servers in the pool with different capabilities as you may end up with servers receiving more requests than they can process. =)
  • URI – (Taken from vShield admin guide – tbh, I’ve never used URI as a method)The left part of the URI (before the question mark) is hashed and divided by the total weight of the running servers. The result designates which server will receive the request. This ensures that a URI is always directed to the same server as long as no server goes up or down.

Anyways, keeping it simple we can choose just to balance HTTP over Port 80 using a Round Robin algorithm. =)


Next we configure Health-checking. A health check checks that all servers in the pool are alive and answering queries. Usually the parameters I tend to use are the default ones configured.

  • Interval – Interval in secs at which a server is pinged.
  • Timeout – Time in secs within which a response from the server must be received.
  • Health Threshold – Number of consecutive successful health checks before a server is declared operational.
  • Unhealth Threshold – Number of consecutive unsuccessful health checks before a server is declared dead

It’s worth noting that with the default settings, I believe a server would be flagged as down after 60secs (3x timeout + 3x Interval – correct me if I’m wrong!). Obviously you can tune this to whatever you want. Just be aware that the worst thing to do is set the Timeout to 1 second as this can cause all sorts of issues because if a server did not respond to a ping within a second, it would be marked as a missed response!

Likewise setting the Unhealth Threshold to 1 would be inappropriate as that means if a server missed 1 response it would be flagged as down.

URI for HTTP service is basically where the load balancer queries to see if the server is up. Usually this is set to “/”, but if you wish to be smart then you can create a static web page to use on each server. Usually a “200 OK” response means a healthy status, a “4xx or 5xx” would usually mean you have a problem.


Next up is adding the servers you wish to load balance into the pool. Simply enter the IP address of the server, it’s weighting (indicates the ratio of how many requests are sent to this server), and the services and ports to be load balanced.


Once you’ve finished adding all the servers to the pool, click Next and Finish at the summary page.

Step 3 – Configure Virtual Servers

Once the Pool Servers has been defined, click on Virtual Servers tab and the Add button.

Again, like when you created the Pool, I suggest using an unique and useful name for the Virtual Server. =)

When creating the Virtual Server, you need to choose which network to apply it on, usually this would be the Org vDC network.

Specify the IP address to use as the load balancer VIP and then which Pool you wish to assign to be load balanced. Finally select the services you wish to load balance, ensure the ‘Enabled’ box is checked on both the services and the Virtual Server and click OK.


Give it about 30secs to reconfigure the edge gateway and there you go….. a working (hopefully) load balancer service on your Edge Gateway!

Simples…… =)

Next blog entry will discuss how we tie the vCenter Orchestrator Load Balancer Actions to our manual process!

Back to Blogging……

So I know I said I wasn’t going to blog much in the coming weeks, but giving the fact that my jury service has been cancelled next week (court case was cancelled so Jury was dismissed due to no other court cases running) and also the fact that my current work project has been cancelled (client cancelled the contract with my company), I pretty much have quite a bit of free time!

Not to mention that I had a sleepless night as all I could think about was that I NEED to blog some of the stuff that’s floating around my head regarding VMware – just so I can put my brain at rest!

So hopefully in the upcoming weeks, I intend to blog about the experiences I’ve had over the past couple of months touching upon:

  • Changing the SSL certificates of VMware products (away from the self-signed VMware ones to a CA certified one).
  • Transact-SQL scripts for creating databases for VMware products.
  • Loadbalancing workflow that I wrote recently to automate the deployment of a loadbalancer in vCD (and hope to generalise so that others can use it).

That should basically fill out my blog for a couple of weeks due to the vast amount of information to get down on paper (or in this case on screen).

First up tomorrow (yes, procrastination doesn’t disappear even when you have some free time!) will be a brief look at how you manually setup a loadbalancer within vCD, and then hopefully I can delve into how the vCO actions can be used for each manual step and what I’ve learnt.

Oh, and as for the job hunting part….. I’m quite thankful that at the moment it seems recruitment agents are calling me up rather than me desperately calling them up! I’m positive that I will be able to find another role that will allow me to continue my VMware journey! (and if you’re a potential employer, or recruitment agent reading this – please contact me if you have any opportunities of interest!)