A little while back I was approached to do some freelance writing for SearchVMware.com and I’m happy to say that I’ve completed my first article.. hopefully the first of many.. =)
So recently a number of customers have been experiencing a core dump target error after rebooting their ESXi hosts…. quite strangely I also recently experienced the same issue when my demo environment went down a few weeks ago due to a power failure.
There isn’t really a clear explanation why this happens, but it seems to be a common occurrence with end-users… it’s also quite simple to fix, but the KB isn’t exactly the clearest of instructions: http://kb.vmware.com/kb/2004299
Next, open a putty session to the host and login as root.
Check to see if there is currently an active diagnostic partition using the following esxcli command:
esxcli system coredump partition get
Check to see if there are any available diagnostic partitions by running the following command:
esxcli system coredump partition list
Usually the coredump partition is configured on the boot device. We now need to find the boot device and the diagnostic partition. Run the following command to list all the storage devices attached to the host.
ls /dev/disks/ -l
or ls /vmfs/devices/disks/ -l
Usually the boot device can be easily identified because it would be the only device with multiple partitions:
(If you want to understand more about partitions that are created by ESXi, have a look at this KB: http://kb.vmware.com/kb/1036609)
Usually the partitions will be labelled and you can easily identify the coredump partition – labelled “vmkDiagnostic” this is quite often the 7th partition. If you’re unfortunate and don’t have labelled partitions, then usually you can identify the diagnostic partiton from the GUID displayed – this is usually “9D27538040AD11DBBF97000C2911D1B8”
Once you’ve identified the partition, you will have to re-point the coredump target to this partition.
To configure and activate a specific partition, use the command:
esxcli system coredump partition set –partition=”Partition_Name”
esxcli system coredump partition set –enable true
To automatically select and activate an accessible diagnostic partition, use the command:
esxcli system coredump partition set –enable true –smart
Once done, double check the core dump partition has been configured by running the following command:
esxcli system coredump partition get
If all is successful, reboot the host to complete the configuration and to ensure the partition is stored after rebooting.
So it seems there’s a known bug when using vSphere Client to log into your vCenter Server Appliance….. it actually affects vCenter Server Appliance 5.1, 5.5, and 6.0.
(Note: filter the vpxd.log using the time you tried to log in)
You can also view the vpxd.log file by logging into the console of the vCSA, enabling shell and navigating to /var/log/vmware/vpxd/
In there, you will see entries similar to:
<YYYY-MM-DD>T<TIME>+02:00 [7F1C10CCC700 error ‘GSSAPI’ opID=CEAEA705-00000004-2d] Cannot get user info for domain\user. Possible NSS configuration problem.
<YYYY-MM-DD>T<TIME>+02:00 [7F1C10CCC700 info ‘commonvpxLro’ opID=CEAEA705-00000004-2d] [VpxLRO] — FINISH task-internal-9727699 — — vim.SessionManager.loginBySSPI
<YYYY-MM-DD>T<TIME>+02:00 [7F1C10CCC700 info ‘Default’ opID=CEAEA705-00000004-2d] [VpxLRO] — ERROR task-internal-9727699 vim.SessionManager.loginBySSPI: vmodl.fault.SystemError:
dynamicType = <unset>,
faultCause = (vmodl.MethodFault) null,
reason = “Cannot get user info”,
msg = “”,
(Note: I ran a grep against the vpxd.log file looking for GSSAPI)
To work around this issue, manually enter user credentials instead of using the User Windows session credentials option.
Alternatively, to resolve this issue:
- Log in to vCenter Server Appliance as the root user.
- For vCenter Server Appliance 6.0 you need to enable the Bash shell in order to access the linux OS, to enable the Bash shell, run the shell.set –enabled True command.
- Open the /etc/nsswitch.conf file using a text editor (i.e. VI)
- Locate the passwd: compat ato entry and replace it with passwd: compat ato lsass.
Note: Remove lsass from the line if it is currently displayed
- Restart the services using /etc/init.d/vmware-vpxd restart.
You can read more in the KB here: http://kb.vmware.com/kb/2050701
VMware Online Technology Forum has started…. are you attending?
Live presentations on all the new goodies from VMware – vSphere 6, vRealize Automation/Operations, virtual SAN 6, App Volumes, vCloud Air, EVO:Rail…….
If you can’t attend today, then the content will be made available on demand from tomorrow (16th April):
I’ve been itching to deploy vSphere 6.0 GA for weeks now (since it was launched last month – wanted to replace my vSphere 6.0 Beta environment) but due to work commitments I’ve had to put this pet-project on the back-burner….. really hate when vendors release new toys at the end of quarter as it means I can’t get to play with it for a month or so!! >_<”
Installing and upgrading the vCSA 6.0 is significantly different than previous releases, it no longer gets distributed as an OVA which means you don’t use the OVF import in vSphere Client that we’re all so used to doing! Instead, vCSA 6.0 gets distributed as an ISO image – which is a bit weird for an appliance!
Hmm…. “So how do I deploy it?” is the most obvious question that most end-users will ask…. Well, you pretty much have to mount the ISO image onto your workstation/laptop/desktop/VM and then run the installation from the mounted drive…..
You may think that it’s a bit of a pain, but the installation process is quite simple and the wizard is very intuitive!
But why would VMware do away with the OVA package?!?
Well if I was to make an educated guess then this could be because they want to phase out the vSphere C# Client, and if you aren’t able to client onto your newly created host then how do you deploy an OVA?
For example, in a freshly installed ESXi host there’s no easy way to manage it without either a vSphere Client or a vCenter Server – at present you can’t open a web-client to the host in order to manage it (see below screenshot of the ESXi hosts’ landing page), so it makes sense to do away with the OVA deployment method and design it so you can mount the installation package for deployment of the vCSA without having to import the OVA via the soon-to-be-retired (maybe) vSphere client!
Now there’s two ways you can install vCSA 6.0 – Guided or Scripted. For ease of deployment, I’m going to discuss the Guided approach using the installation wizard. The Scripted approach is aimed at people who wish to automate the deployment of (several) vCSAs.
So before we get started, there are certain pre-requisites which must be completed prior to deploying the vCSA (in addition to what is listed in the documentation)
- Ensure that the hostname being assigned to the vCSA is in DNS, ideally both forward and reverse lookup. This will help with the installation process (I won’t go into the reasoning or what happens as several people have already posted online to mention the installation could fail if no DNS entry can be found).
- Ensure you install the Client Integration Plug-in before running the installation – the installer will not run without it installed! (This is both for fresh installs and upgrades!)
- Do not input more than 1 DNS server (even though the installer prompts that you can). This will cause the installer to fail – as pointed out in the Release Notes.
- Ensure you enter the network settings correctly, as there is no pre-check function available and any errors will lead to firstboot errors – again, as pointed out in the Release Notes!
Especially watch out for VLAN configuration errors, ensure the vCSA is on the correct VLAN and it’s routable to the machine you’re deploying from (as well as the ESXi host itself).
Right, now you’re ready to mount the ISO on your deployment device (my case – my Win 7 laptop) and start the installation process! In my case I’m using MagicDisc to mount the ISO.
Next launch the setup via the vcsa-setup.html file:
Next choose the deployment type. In my case I want to deploy the embedded PSC. I won’t go into the technicalities of what the PSC is, and the different deployment scenarios – if you wish to learn more than head along to Derek Seaman’s site which explains the PSC in more detail!
Input the network configuration details, ensuring the FQDN is resolvable in DNS. Pay attention to the NTP server, especially if deploying/connecting to another PSC – if they’re out of sync, it could cause installation issues!
Once complete, the installation wizard will give you the details to connect to the web client, the URL will be https://fqdn/vsphere-client (no more port number required at the end of the url!!). Remember, if you’ve changed the SSO domain earlier, then the login user will be administrator@SSO-Domain
Now that the vCSA has been deployed, there is a new way of joining it to an Active Directory Domain, which will help you configure the Identity Sources for SSO. Log into the web client and then on the home page select System Configuration.
Now you can add in the domain as a SSO Identity Source as you would usually do. However, you can choose Active Directory (Integrated Windows Authentication) and it should populate the domain details and pick up the information from when you joined the vCSA to the domain.
For more information, point your browsers to the vCenter Server 6.0 Deployment Guide.