Nostalgia in the Cloud – Prince of Persia in the sky (online)

vCloud Air is now proudly hosting my favourite Nostalgia VM having recently decided to extend my Homelab into the cloud by installing an on premise vCloud Connector Server and a vCloud Connector Node. The entire setup process took literally no more than 20minutes and the beauty of the Nostalgia VM was that as its virtual disk is so small, even on a consumer ISP, the copy speed to transfer from the vCenter UI (note this is only accessible via the C# client) took barely any time.

The quick and dirty steps to create the hybrid integration involved:

  • Download and deploy vCloud Connector Server
  • Download and deploy vCloud Connector Node
  • Configure the on-premise vCloud node to register with my on-premise vCenter
  • Register the vCloud Connector Server with the on-premise node and the publically available vCloud Air node via the IP address of the datacentre assigned to you as part of your subscription (found here), along with your credentials (note that the name you define for your VDC is the ID visible within vCloud Director from within vCloud Air and not the display name of your VDC)
  • I then enabled the vCloud connector plugin from within the vSphere client and could now see both my private and cloud based VMs within the plugin.

If you haven’t already created a catalog within your VDC, you need to do this before you can copy a VM but this is a three click process within vCloud Director.

Copying VMs to vCloud Air also requires the source VM to be switched off.

After the the copy completed, I powered up the VM and opened up the web embedded console. Even though I’d only recently reacquainted myself with the original game through the VMRC on my vSan setup, playing it whilst running from the public cloud gave it a whole new dimension in playing Prince of Persia online! Admittedly my timings and responsiveness had to be slightly adjusted for keyboard inputs due to the larger than 1ms latency I get from my homelab to the VM, but I was pleasantly surprised to find that it was still playable in its full glory.


Transparent Page Sharing – Reduce Your Memory Footprint (Homelab recommended)

In late 2014, VMware changed their default stance on the use of TPS to share memory between Virtual Machines as a consequence of a low risk security threat. Given that VMware needed to remain squeaky clean on security compliance, they duly changed the out of the box way TPS worked forever which for some customers meant that memory budgets and allocations that were architected based upon the old implementation needed to be reconsidered. Views on whether you should or shouldn’t change the default behaviour of VMware to revert back to what is an extremely efficient way of sharing memory are discussions that need to be had with corporate security officers but for homelabs, where YOU wear the security hat (amongst many others), why not benefit from reclaiming some much needed RAM.

Reasons to consider whether or not to do this can be found on numerous other blog posts, but as a quick and dirty guide, here are the steps you need to follow to restore the original mechanism:

On each host within your cluster, through the Web Client, click on Manage –> Settings –> Advanced System Settings. Locate Mem.ShareForceSalting and change the default value of 2 to 0 (zero)

vMotion the workloads off or power them off and on again for the change to take effect!

To demonstrate the benefits of reverting TPS to the legacy way, on my 3 node VSAN cluster with 48GB of RAM, here are the memory stats before and after:





As you can see from the PSHARE/MB common: saving column, in total across all three hosts, with a little bit of maths, this is a memory reclamation of over 10GB which is just short of a quarter of the RAM available to my entire Cluster.  Why not try it and see what effect it has, at least on your homelab!

Nested ESXi Problems running on vSAN – Stateless caching – Cannot apply the host configuration

I was trying to get AutoDeploy to work in a Nested ESXi configuration whereby I configured stateless caching, but each time I would end up with less and less hair as the Host Profile just would NOT apply and I’d see “Host does not appear to have a compliant stateless cache”.


It always came down to the application of the “System Image Cache Configuration” that would result in the Profile failing to commit. Frustrating as it seemed, I was almost at the point of giving up, until I tried exhaustively hunting through the log files:

/var/log/syslog.log which always returned: “filesystem was created but mount failed on device “mpx.vmhba1:C0:T0:L0:6″.: Not found”

I then started looking through

/var/log/vpxa.log for more detail and found:

“usr/sbin/vmkfstools -C vmfs5 -b 1m -S datastore1 /vmfs/devices/disks/mpx.vmhba1:C0:T0:L0:3′ exited with status 1048320”

I Googled the status 1048320 and as if by magic, the article below was returned from William Lam from back in 2013 which effectively states that whilst vSAN does not make use of SCSI-2 reservations the LVM driver still requires it in order to create a VMFS datastore. By entering the following command on the ESXi hosts presenting vSAN (yes – the physical hosts, not the Nested VMs), this fakes the required SCSI reservations needed to proceed:-

esxcli system settings advanced set -o /VSAN/FakeSCSIReservations -i 1

After running the command on the first of my three host cluster nodes, compliance turned green and I had a smile back on my face. compliant


Given that vSAN is becoming more and more prevalent and the existence of so many Nested configurations, I figured I’d try to share and bring this important tip back to life.

How to run Nested ESXi on top of a VSAN datastore?



Unable to establish an ssl connection with vcenter server – Mac OSX

I’ve been seeing some strange behaviour whilst performing some deployment operations such as pushing out OVF templates in that occasionally I would see “Unable to establish an SSL connection with vCenter Server”

Initially, I could get away with just ignoring it and if I tried it again, it would sometimes work, however for some reason, it would now always fail.

I did some digging and the logs suggested something to do with the Certificates not matching. As my MAC is not domain joined, I started digging around in the Keychain Access tool¬† –> Applications –> Utilities –> Keychain Access

In here I found all sorts of historical certificates from my various lab environments that I’ve added over the months and thought I’d better clear up some legacy certs as well as duplicate named certificates as a start point. It was here that I found that I had two certs for the same domain object (one which I’d previously rebuilt) so figured that was a good start to remove at least the now no longer used duplicate.

The problem still existed so I did some further investigation as whilst I had a trusted certificate for my VCSA, I couldn’t see the Trusted Root CA. This isn’t visible by default so I had to add the X509Anchors keychain by clicking “File –> Add Keychain” and locate this within /System/Library/Keychains/X509Anchors

Once this was visible, I again could see a previously trusted Root CA from a legacy domain I no longer use at home, but not the Root CA of my Microsoft Domain. I removed the invalid entry and went across to my internal Microsoft CA website to get the Root CA to reimport.

I downloaded the Root CA by clicking on install this CA certificate and it downloaded it to my MAC.

I then attempted to double click the file to import it and when prompted selected X509Anchors however I then received: Error 100013.

A quick search on Microsoft led me to the following article:

so I followed these steps:-

cd ~/Desktop
cp /System/Library/Keychains/X509Anchors ~/Library/Keychains
certtool i cert_filename k=X509Anchors
sudo cp ~/Library/Keychains/X509Anchors /System/Library/Keychains

I refreshed the Keychain Access tool and lo and behold, the cert appeared.

I double clicked on the newly listed certificate and expanded “Trust” and set the top option of “When using this certificate” to “Always Trust” and closed the certificate properties.

I then reloaded the vSphere Web Client, repeated the OVF deployment and everything worked!

Hope this helps somebody!