Server 2016 PVS 2203 losing machine account password

[Jeff Riechers]

I have found something that I am not sure if it is a MS bug or something else.  

 

About a week ago we noticed that some Windows 2016 machines started getting out of sync with the machine account password.  We have to shut the machines down, reset the machine account password from the PVS console, then boot them back up and they are fine.  Looking at the event log on the machines we see the following message.

 

Event ID 1017

Source BNDevice

Data Updating machine account password - Client Service

 

Then immediately getting

 

Event ID 1015

Source BNDevice

Data Unable to negotiate new machine password - Error code: 0x2

 

These repeat every hour until we fix the machines.

 

Has anyone else seen this?  I am not sure if it is an AD Hardening issue, a MS Certificate change issue, or something else.

 

We have the necessary Manage machine acount passwords set in PVS for 1 week, and have the Domain Member Disable machine account password item set in GPO.

 

This has been running in place for years, just started cropping up about a week ago.

 

Server 2019 and 2022, and Window 10 do not seem to be affected.  Just Server 2016.

[Jeff Riechers]

Update:  This fixed the issue for us so far.  Looks like there is some security changes in 2016 in the January - March updates that breaks the PVS account management.

 

Ok, this is starting to roll into more environments.  Another 2016 environment that did updates on 3-10-23 started having machine account passwords not resetting correctly.

 

I believe it is tied to KB5023697 and also the SPNDowngradeProtection from https://support.citrix.com/article/CTX472962/error-connecting-to-pvs-farm-with-credentials-from-trusted-domain

 

So I set the that registry entry to 0 on the image itself, and then manually installed that KB as 2016 was having trouble installing it directly from Windows Update.

[Carl Fallis]

I would also take a look the Domain Controllers, I think you have to enable error logging to get the logs.  The DC may have a better error reported.

[Simon Graham]

BTW: Error Code 2 indicates that we were unable to fetch information about the target machine from the DC. I do note that we assume the Netbios Name for the machine is the same as the hostname (with a trailing $). Would definitely be worth looking on the DC for better error logs as Carl said

[Jeff Riechers]

Back at this same message after last year.  And I found the answer.

It wasn't that hotfix at all.  It was daylight savings time.

With Server 2016 if your reboot a machine after daylight savings time has shifted forward,  without updating the underlying image with the knowledge of DST shift, then your machine account password reset process will crash out.  You have to open your image, allow it to adjust the DST flag, then re-seal it and send it out.  

So last year when this came up, we updated the image with that reg entry, but that was not it.  It was just the update process that fixed it.

Doesn't happen on Server 2022 or 2019.