High CPU TrustedInstaller in a VDI Environment

A client of mine recently reported regular occurrences of high CPU usage in their virtual desktops as a consequence of TrustedInstaller appearing. They couldn’t fathom out why it would crop up and it seemed to be at random.

It didn’t take long to track down offending machines as their performance metrics in vCenter would put their CPU usage at the top of the list so as soon as one cropped up, I started remotely querying the machine and trawling through the Application Event Logs.

Each of the VMs were 2vCPU Windows 7 machines and as can be seen in the image above, the process would effectively take an entire vCPU (50% of the total available to the machine in this case)

After doing so for about 5 machines and correlating the CPU Time for the executable back to the Application Event Log time (in the example above, I worked back 49 minutes) they all pointed back to the execution of a Bomgar Support Customer Client that was being installed when the support team were actually helping customers for other issues. The irony was, that once the support engineer had resolved the users issue, they were effectively leaving the user with half of their CPU because the installation of the remote control agent, triggered the TrustedInstaller.exe

I validated my findings by asking to be sent a remote support request and lo and behold the same problem appeared. At least now we had a way to make the problem repeatable. After that, I ran another user permitted executable that triggers the Windows Module Installer service (TrustedInstaller.exe) and precisely the same problem appeared. I now knew it wasn’t limited to the support tool, but a generic problem at the OS layer.

Knowing that TrustedInstaller writes to the following log file C:\Windows\Logs\CBS\CBS.log, I took a look inside to find there were no specific issues reported other than a mention of “Scavenge”

2017-02-09 12:45:14, Info                  CBS    Scavenge: Begin CSI Store
2017-02-09 12:45:14, Info                  CSI    00000012 Performing 1 operations; 1 are not lock/unlock and follow:  Scavenge (8): flags: 00000017

This task however didn’t ever seem to stop and despite leaving a machine for 24 hours, it never got anywhere to gracefully terminate the start of the service in the first instance. At that point I realised something was possibly out of place with respect to the Windows Update process and in my frantic search stumbled across the Windows Update Troubleshooter.

https://support.microsoft.com/en-gb/instantanswers/512a5183-ffab-40c5-8a68-021e32467565/windows-update-troubleshooter

I downloaded the CAB file and ran it against the Master image which in fact triggered TrustedInstaller.exe. At the end of the process, it stated it had fixed the Service Registration.

I checked the CBS.log file again and found the following repair entry:-

2017-02-09 22:24:14, Info                  CSI    0000000a [SR] Beginning Verify and Repair transaction
2017-02-09 22:24:14, Info                  CSI    0000000b Repair results created:POQ 0 starts:     0: Move File: Source = [l:192{96}]”\SystemRoot\WinSxS\Temp\PendingRenames\a5c17e3e2383d20103000000a8043012._0000000000000000.cdf-ms”, Destination = [l:104{52}]”\SystemRoot\WinSxS\FileMaps\_0000000000000000.cdf-ms”

After hitting “Close” on the troubleshooter, I noticed that TrustedInstaller.exe was still running so left the computer for about 10 minutes at which point the process closed. I went back to the CBS.log file and found that a subsequent Scavenge job had run again but this time had completed.

2017-02-09 22:34:35, Info                  CBS    Scavenge: Begin CSI Store
2017-02-09 22:34:35, Info                  CSI    00000012 Performing 1 operations; 1 are not lock/unlock and follow:  Scavenge (8): flags: 00000017           CBS    Reboot mark refs: 0
2017-02-09 22:34:35, Info                  CBS    Idle processing thread terminated normally
2017-02-09 22:34:35, Info                  CBS    Ending the TrustedInstaller main loop.
2017-02-09 22:34:35, Info                  CBS    Starting TrustedInstaller finalization.
2017-02-09 22:34:35, Info                  CBS    Ending TrustedInstaller finalization

For good measure, I rebooted the Master Image and took a new snapshot, recomposed the pool and tested.

Problem solved! Phew…

Let me know if the post has been useful and/or fixed any similar problems you may have had with TrustedInstaller.