Unable to connect to the MKS: Console access to the virtual machine cannot be granted since the connection limit has been reached

Today I was in the process of managing my VMs and as I use a Mac with the VMware Remote Console, it can sometimes be a little flakey in terms of stability. This isn’t normally a big deal for me as typically I’ll reopen the MKS session and pick up from where I left off. For some reason, today was a little different and after the usual “crash”, I attempted to connect back across and was presented with “Unable to connect to the MKS: Console access to the virtual machine cannot be granted since the connection limit of 1 has been reached”.

mka

I then tried the integrated console but received a similar message of “You have reached the maximum number of connected consoles: 1. Please contact your administrator.”

mk2

I knew that restarting the VM would clear the issue, or powering it down to increase the number connections permitted (KB2015407) would be a work around but the problem was I didn’t know what state the VM was in as I was mid installation so couldn’t really justify pulling the plug.

At that point I thought I’d try a quick vMotion between the hosts and as if by magic, my subsequent attempt to connect to the console in either way sprung back into life!

Citrix User Group XV

Yesterday, I went to my first Citrix User Group in London. It was indeed an eye opener and I did stumble upon others I’ve spent time with in the industry before. App Layering played a big part in today’s events and discussions as to whether this was another knee jerk/stop gap opportunity were discussed. Appsensebigot played service to the wonder of saving CPU and Memory consumption as part of using the natively available Easylists to block adverts (aka AdBlock Plus) within IE, which from experience is a great way of getting natural buy in for a native applications extension from somewhat reluctant management when considering what to introduce as a way to reduce compute overhead in VDI/SBC environments.

I also met Rubén Spruijt who provided two extremely useful presentations for which I share similar interests and beliefs. He has a very positive outlook on technology and is not afraid to speak the truth on what he believes will make a difference in the future.

Jim Moyle stepped in for another missing presenter and came up with some great slides on I/O optimisation/sizing for Windows which was a joy to watch! He really got to the nitty gritty of IO distribution and the impact it has on SSD shelf life and how to understand the true calculations required particularly in VDI environments.

Today, I’m off to VMUG to see what this has to offer too, so I’m certainly sticking to my guns around staying actively involved where I can!

All Flash vSan – Mac Mini Upgrade – Permanent Disk Failure Fix?

I’ve been experiencing the disappearing drive act, more commonly known as Permanent Disk Failure whereby under duress, the host will mark the SSD as failed simply because it just can’t keep up and goes walkabouts. This was almost reproducible on demand by either committing a large snapshot or just powering everything on at the same time (basically heavy IO).

 

After some research into whats causing it (apart from my environment not being on any sort of HCL), it seems that the SATA AHCI controller on the Macs really can’t cope too well and even though I thought I’d bought a decent SSD drive to compliment vSAN (a Samsung 850 Pro), this actually appeared to be more of an achilles heel than the controller. Rather than start replacing my lab with more power hungry, noise demanding hardware to work around the issue, I thought I’d give it one more roll of the dice and whilst again not technically on the HCL for an all Flash vSan, have purchased some Intel DC3700 SSDs to act as the vSAN cache tier to the pre-existing 850 Pro SSDs.

 

IMG_5921IMG_5923

Goodbye Hitachi magnetic disk, hello Intel SSD.

 

If the 850s continue to provide me with problems, I’ll revert to SATA Magnetic disk, although in theory, I shouldn’t be driving the 850s hard enough now for their bottle neck to rear its ugly head – although having said that, in an All Flash vSan, all reads are directed to the capacity tier (gulp). Another consideration I had thought of was to look at ROBO and whilst vSphere 6.1 supports it, it doesn’t when using All Flash. For the time being I’ll be sticking with three Mac Mini hosts.

 

VCP 6 Qualified

CXTx1tYWAAARVLW

Today I sat 2V0-621D and upon completion of the test, it advised me I had passed successfully (woohoo). I now plan to continue on my learning path, juggling as do many people a hectic personal life, educational and work balance. I see 2016 as a big year for vSphere 6 with many customers making the leap from traditional Windows vCenter to VCSA. For me, taking the exam was an important milestone to bolster the experience I have with a properly certified status to reaffirm my commitment to the technology as I have done since my first VMware exam back in 2008.

VMware Certification – 8 years a VCP

VCP-DCV

I’ve been VCP certified since 2008 on VI3 and continued to develop my interest in VMware by following up with vSphere 4, 5.5 and on Friday 15th Jan 2016, I’ll be attempting my fourth VCP accreditation in a bid to become a VCP in vSphere 6.

I’ll admit that this time round, it has been far more of a study challenge for me as I’ve not had as much exposure on vSphere 6 through the clients I’ve been working with, largely due to many companies inability to keep moving at the pace that VMware releases new versions and the compliance challenges with the requirements Matrix due to overlapping and dependent technologies. Take for example Trend Micro Deep Security, with vSphere 6 hitting the market in March 2015 and Trend not releasing v9.6 to be compatible with vSphere 6 until August 2015, as most companies go, they rightfully didn’t want to be the first to deploy a new product and in this case, Trend was the new product, requiring a typical 3 month wide birth until proven in the field by other more willing audiences (I won’t mention their legacy Horizon View 5 implementation).

In order to prepare correctly, I hit my home lab and bumped it up from 5.5 to 6.1 but not without challenge. I decided to jump straight into vSAN using the Mac Mini setup that closely matches Peter Bjork:-

A killer custom Apple Mac Mini setup running VSAN

but I had already made a previous purchase/investment in consumer grade SSDs (SAMSUNG 850 Pros), and almost immediately hit performance issues with drives simply disappearing, not to mention very high read/write latency on the capacity magnetic disks. Long story short, the entire vSAN fell over, I lost a few VMs in the process, but this ultimately helped me to learn how vSAN worked and how I could piece things back together again, realising the importance of the HCL and how a vSanDatastore needs to be treated with more respect than a typical VMFS datastore (i.e. don’t just place things on there using the datastore browser).

Anyhow, back to education, I found the following very useful study guide posts on VCP6-DCV (thanks Vladan) and have worked through them meticulously in my lab environment, alongside the reference/blue print material from VMware so fingers crossed for a successful outcome on Friday!

http://www.vladan.fr/vcp6-dcv/

VMUG Advantage

To compliment my “being more involved” hat, I’ve subscribed to the VMUG Advantage program to take “advantage” of the benefits it can bring as well as to get more opportunity to engage with others in the community. Some of the most obvious reasons for me to sign up included:-

  • Discounts on Examinations
  • Discounts on Training
  • $600 worth of service credit for vCloud Air
  • Recognition
  • Access to EvalExperience for access to licensing of the Core VMware stack to tick along in the HomeLab

I’m hoping it will be a worthwhile investment so that 2016 can push my technical capabilities that extra mile.

Its been a while..

I’ve not actively blogged for quite a few years now, coinciding with the arrival of my two children in fact. Now that they are settled into a routine, now is the time for me to get back into the habit of sharing those daily challenges we all face in the world of infrastructure architecture and systems management. I look forward to being more involved in the community and actively offering advice and information where I can.

Orphaned, disconnected or inaccessible?

I was asked the other day by one of my colleagues to explain what the difference was between each of these VM states so I figured I’d write a quick overview of each.

Orphaned VM

In a nutshell its a VM that vCenter still has a record of within the database, yet it either doesnt actually exist anymore, or isn’t on the host where vCenter expected to find it.

So how did it get into this mess? Well, quite simply really. Imagine you’re managing a two host cluster within vCenter and someone decides to administer one of the individual hosts directly through the vSphere client. They then proceed to remove one of the virtual machines from the inventory. As a result of this, the ESX host itself drops its record of the VM ever being there, but the vCenter DB still has record of its existence. As such vCenter marks this VM as “Orphaned”. To rectify this, either re-add the VM back to the standalone guest or remove the “Orphaned” entry from the inventory and re-add it. (note, if you do not remove the orpahned entry from vCenter server, you will not be able to re-add it as the same VM name).

Inaccessible

This is usually when a datastore or its associated folder/files on the datastore have gone walkabouts and the host can no longer see the VMX file it used to talk to in order to maintain visibility within the vCenter server. This can sometimes happen if someone decides to rename the folder the VM resides in, without removing it from the inventory, renaming the folder and then re-adding it.

To resolve the inaccessible message, either relocate where the underlying VM has gone or remove it from the inventory completely.

Disconnected

This is usually as a result of the host that last managed the VM losing communication with the vCenter server. Any VMs that were running at the time of the break in communication (or indeed a manual right click on the host and clicking disconnect), will render the VMs under its control as disconnected.

To resolve the disconnected state, Connect the ESX host back into the cluster.

Unresponsive guest – hung VM

I’ve encountered a few scenarios where virtual machines have just refused to power off and each time I find myself hunting down the best method to kill them indefinitely. Occassionally these “hung” virtual machines are as a result of losing sight of their storage – yet the memory thread still stays resident.

Firstly, it’s best to determine if the VM really is still running:-

vmware-cmd -l
(this lists the Virtual Machines on the host – on and off)

copy the full path to the VM that you wish to query i.e. /vmfs/volumes/4a69985-29b83f0c-5ee5-001b3432f0d0/vm.vmx

and insert it into

vmware-cmd (path) getstate

i.e. vmware-cmd /vmfs/volumes/4a69985-29b83f0c-5ee5-001b3432f0d0/vm.vmx getstate

if the host believes the virtual machine is still on, it will return
getstate() = on

if the machine is in fact off, it will return
getstate() = off

If it is still running and you are unable to shut it down using the vSphere/VI client, here are a couple of ways to kill off any unresponsive virtual machines:-

vmware-cmd (path) stop

validate whether this has been successful with another getstate command

vmware-cmd (path) getstate

if unsuccessful, try a stop hard request

vmware-cmd (path) stop hard

once again, checking to see if this has worked

vmware-cmd (path) getstate

—————————-

Alternatively, you could try:

vm-support -x
(this displays a list of running VMs and their associated World IDs)

vm-support -X <wid>
(this attempts to kill off the process with the World ID specified)

—————————-

Finally, and as a last resort:-

ps -g | grep <VMname>

This will show the following

649451      vmm0:VMname
649453      vmm1:VMname
649640 649448 mks:VMname       649448 649448  /bin/vmx
649641 649448 vcpu-0:VMname    649448 649448  /bin/vmx
649642 649448 vcpu-1:VMname    649448 649448  /bin/vmx

The first column is the World ID (WID), the second column is CID and the fourth column is the Process Group ID (PGID). The PGID is the relevant value required (649448).

kill -9 <PGID>
i.e. kill -9 649448

Using the kill command, the unique processes for this VM should now be terminated. I have found that whilst this works, it does sometimes reset the VM.

VM-flat.vmdk file – check.. but where’s the vmdk file?

So after spending many hours migrating all my VMs back to their uber performing datastores, I went to power on my secondary DC only to find it would not start up.

Something to do with a missing file.

“The system cannot find the file specified.
Cannot open the disk ‘DC02.vmdk’ or one of the snapshot disks it depends on.
VMware ESX cannot find the virtual disk “DC02.vmdk”. Verify the path is valid and try again. ”

I immediately checked the configuration settings in vCenter and all appeared correct. The datastore browser confirmed that it could see the 10GB vmdk file – so what could it be?

I never trust a GUI, so ssh’d over to the TSM and did a quick directory listing to find that whilst the -flat.vmdk file was there, the .vmdk file wasn’t! In the migration back, somehow the VM had lost the file that controls its understanding of disk geometry, controller type, provisioned format (thin/thick).

Knowing I had the -flat file was re-assuring, had the shoe been on the other foot and all I was left with was the vmdk file, I would have been a little lot more concerned.

The first step to resolution was to create a new identically sized virtual disk to the -flat file I had been left with. In turn, this would then create a new VMDK file that I could borrow.

1) Determine the existing -flat.vmdk file size

ls -l *-flat.vmdk
-rw——-    1 root     root         4841537536 Feb 24 23:38 DC02-flat.vmdk

2) Determine the controller type associated with this disk

less *.vmx | grep -i virtualdev

scsi0.virtualDev = “lsilogic”

In this instance the VM used the lsilogic controller

3) Armed with this information, I now create a new vmdk

vmkfstools -c 4841537536 -a lsilogic -d thin temp.vmdk
(the -d thin parameter provisions the disk as thin as we don’t really want the -flat file anyway)

4) The result is a temp-flat.vmdk and a temp.vmdk file.

5) Rename the temp.vmdk file to match the VM name – in this case DC02

mv temp.vmdk DC02.vmdk

6) Edit DC02.vmdk using VI and update the extent description contents to match the server name

# Extent description
RW 20971520 VMFS “tempDC02-flat.vmdk”

7) If the original -flat.vmdk was thinly provisioned you do not need to modify any additional parameters in the file, however if it was thick, you must remove the following line:-

ddb.thinProvisioned = “1”

8) Delete the temp-flat.vmdk created in step 3 and you should be good to go!