vSphere & IPv6: how VMRC breaks it all

It’s been a bit quiet around my vSphere pure IPv6 lab setup lately. At first it seemed to be surprisingly easy, but the devil’s in the details…

As already posted, the initial setup went without any problems. The setup required IPv4 just for the kickstart process, or to be more exact for the first boot and installation phase. During the final configuration script the DHCP IPv4 address was dropped and the ESXi hosts were running IPv6 only: Continue reading

Reducing the vCenter memory – vCSA 5.5

An update for this post on how to minimize the memory requirements of the vCenter Server Appliance for release 5.5 was long overdue. Sorry for the delay, I was rather struggling with a pure IPv6 setup (and found out that the VMRC plugin as well as the Web Client break the whole thing – more on that soon).

Anyway, lets see what changed with V5.5 with regards to the memory requirements and JVM parameters. Amazingly quite a lot, but in a good way. Some of my recommendations are obsolete now since VMware changed the settings to more or less the same values I proposed. Must be a coincidence, of course. At least a nice confirmation that my settings were not that bad. 🙂

I had to update this post for vCSA 5.5 update 2 since the settings for the initial 5.5 release caused services to fail in 5.5U2. Overall, the memory requirements were significantly increased.

Continue reading

2001:db8:1ab:: up & running

The lab is (slowly, I have to admit) turning into an IPv6 only configuration.The first question might be “why would somebody want to do that?” Well, because it’s interesting and the main reason to have a lab anyway. But besides that I think every company should do something like this right now, to test their services and hardware (!) for IPv6 readiness. Now.

The basic setup in my lab is working fine, which means the firewall configuration is done and the local DNS is set up. For a start I’ll be using static IPv6 addresses and keep Router Advertisements and DHCPv6 for later. The NAS box (Debian 7.2) is configured with four interfaces for storage, two in VLAN 24 for iSCSI and two in a LACP setup in VLAN 25 for NFS, everything in a dual stack setup. The Kickstart configuration and scripts are modified as well, although I suppose the PXE boot itself will have to stay with IPv4, there’s no support for IPv6 in the firmware. Continue reading

Choose the right quad port NIC

With vSphere 5.5 VMware has dropped support for some of the hardware enthusiasts like me used in their home lab (which basically means I consider people who spent quite a few bucks for a home lab to be enthusiasts – are you really sending the right message to us, VMware?). I cannot approve that – to me it’s just a bad move to remove drivers for hardware that would otherwise work fine, even if it was not supported. Which basically would just mean you’d be on your own if something would not work, and VMware Support would just tell you that. Nothing to scare a whitebox user away.

Anyway. I had to face the fact that the onboard and additional PCI Realtek NIC of my lab ESXi would be unusable sooner or later, even if an older driver could be injected into the ESXi 5.5 image, as I explained in a previous post. So I decided to find a setup that would provide at least five network interfaces with presumably long term support. Continue reading

Heads up: ESXi 5.5 crashes Linux iscsi_trgt

If you’re running a standard Linux on your homelab storage box with iSCSI (ietd, iscsi_trgt), you need to take some actions before deploying ESXi 5.5 in your environment. It seems the new ESXi release issues some SMART command on iSCSI targets, which hits a bug in iscsi_trgt. I’m using Debian 7.2 on my lab NAS which comes with iscsitarget 1.4.20.2. Once an ESXi 5.5 tried to access an iSCSI LUN, the iSCSI subsystem crashed completely. You’ll see messages like these: Continue reading

ESXi 5.5 and Realtek 8168/8169

VMware has dropped the support for Realtek R8168 and R8169 NICs in ESXi 5.5, as already posted in some blogs even before vSphere 5.5 became GA. Really bad news for all of us whitebox homelab owners who use the cheap cards and onboard chips to increase the number of NICs available for vSphere networking. Fortunately the drivers included in ESXi 5.1 still work with 5.5, and there are good instructions available by Vladan Seget and Erik Bussink on how to create a custom installation ISO. But if you’re using a Kickstart server (as I still recommend, even with Auto Deploy and also for a small home lab) it is yet easier. Continue reading

How to clear phantom alerts on VMs

Under certain circumstances some or all VMs in a vSphere environment may show the red alert icon and an alert status in a VM list, but no alarm in the VM properties that could be acknowledged or cleared. These “phantom alerts” are an issue for quite some time now and still not fixed.
They seem to be caused by an interruption of the connection to the datastores these VMs reside on, for example by an ESXi host booting while the storage is not up & running. The VMs are first displayed as “disconnected”, but after the datastore connection is restored, the alert status stays.

There is no straightforward way to resolve this, but a vMotion clears the status, most likely because the VM registration is updated. So basically just move the VMs around to clear the alerts, the quickest way is to set the hosts into maintenance mode and evacuate them. A host reboot is not required.

Heads up: vShield Manager killed by update

There seems to be a serious problem in the update process of the vShield Manager appliance. I had to update a vSphere 5.0 / vCloud Director 1.5 environment running vSM 5.0.1-638924 to the latest releases (vSphere 5.1 update 1a / vCD 5.1.2 / vSM 5.1.2a). Disk space on the vSM was sufficient, the update itself using the VMware-vShield-Manager-upgrade-bundle-5.1.2-943471.tar.gz went fine, the appliance rebooted and was busy with the upgrade process for some time, as expected. Afterwards the UI showed the new release, the VM was idling, update apparently successful.
But after a manual reboot (due to a scheduled maintenance of the whole environment) before migrating the configuration to a new vSM instance the appliance would not come up anymore. The disk driver encountered fatal I/O errors, maybe due to file system or partitioning damage (although the symptoms do not clearly indicate that), finally resulting in a kernel panic: Continue reading

IPv6 Hands-On Lab

Just to keep you updated (no, this blog is not a straw fire!):

A colleague & I have meanwhile set up an IPv6 test lab on the same hardware I use in my home lab. And this means a complete setup: DHCPv6, RA, static IPv6, tunnels, firewalling, broad range of client OS – the whole nine yards.

It’s going to take some time to write a series of blog posts describing the setup, and I’m still tempted to use IPv6 for the vSphere infrastructure as well. Maybe even for iSCSI, although it’s no longer officially supported…

So stay tuned, there’s a lot of stuff coming soon! Just have to finish the setup, writing – and my holidays. 🙂

Workaround for the vCenter Server appliance 5.1U1 update delay

The update process from 5.1.x to 5.1 Update 1 contains a serious flaw. The update may take more than 45 minutes, some report more than one hour. VMware even mentions this in their release notes:

Update of vCenter Server Appliance 5.1.x to vCenter Server Appliance 5.1 Update 1 halts at web UI while showing update status as installing updates*
When you attempt to upgrade vCenter Server Appliance 5.1.x to vCenter Server Appliance 5.1 Update 1, the update process halts for nearly an hour and the update status at Web UI shows as installing updates. However, eventually, the update completes successfully after an hour.

Workaround: None.
(http://www.vmware.com/support/vsphere5/doc/vsphere-vcenter-server-51u1-release-notes.html)

The generic update documentation KB article 2031331 “Updating vCenter Server Appliance 5.x” mentions even longer durations:

The update process can take approximately 90 to 120 minutes. Do not reboot until the update is complete.
(http://kb.vmware.com/kb/2031331)

Well, there is a workaround, even a very simple one: