I thought I’d see if I could break my environment again by starting from vSphere 6.0U1 and attempting to perform a completely hassle free upgrade to VSAN 6.2. My previous two posts have come back with problems so 3rd time lucky right?…
As before, I upgraded the VCSA (which was running as a VM within the environment I’m upgrading) from 6.0U1 to U2 and the hosts and rebooted everything to give me a clean slate. I migrated across a couple of VMs from another environment and did some typical admin tasks by cloning, vmotioning, powering down etc.
I hit the magic “Upgrade” button under VSAN Management and thankfully didn’t get any Task related errors. However, out of nowhere, the following error appeared : Cannot Upgrade the Cluster: Object(s) xxx.xxx.xxx are inaccessible in Virtual SAN.
Wow – third time lucky and a different error! This VSAN upgrade process is starting to become frustrating (umm, how much testing took place before it became GA?).
Anyhow, having seen a similar error, albeit displayed in a different way in this post, I figured I’d see what objects this particular upgrade failure attempt related to. As before, I SSH’d to the VCSA, RVC across to the Ruby Shell (other post has more instructions) and then ran:-
vsan.object_info BRAINS/ fcfbcd56-5731-55b0-42bb-0c4de9cd75c8
This time around, the objects seemed to be complaining about the lack of a Policy Entry within the CMMDS and LSOM Object Not found:-
As the objects were a) not found and b) reported to have a usage of 0.0GB I figured I’d get rid of them! So, SSH’d across onto an ESXi HOST (yes an ESXi host, people have tried this on the VCSA and it just doesn’t work) and ran the following command, replacing the Object ID where necessary. (note this is a homelab, proceed with caution)
/usr/lib/vmware/osfs/bin/objtool delete -u fcfbcd56-5731-55b0-42bb-0c4de9cd75c8 -f -v 10
The output reported “Successfully deleted” so I re-ran the upgrade and everything completed successfully.
As the battle to perform a seamless upgrade has been taking place, I have been made aware that there is currently a BUG that is in the process of being created into a KB around the deletion of objects that cause orphaned objects that result in problems such as these. As soon as I have more information, I’ll link it to the posts.