Machines don't always start after scheduled Reboot

[Tarik Chougui]

Hello everyone,

 

I'm having an issue on my XenDesktop 7.6 (server OS  2008 R2) machines.

I scheduled a reboot directly trhough studio at 3 am for all my delivery groups.

 

The problem is only half of my machines startup on the morning. The Vsphere console tells me that the machine shutdown at 3 am then bootup then shutdown again 20 minutes later ( i've added a screenshot of the vsphere events)

 

I turn  AutomaticPowerOnForAssigned and AutomaticPowerOnForAssignedDuringPeak to $false for all my delivery group.

 

But the issue still occurs.

I'm gonna try to bypass it and use a vmware restart task and see how it goes.

 

Any idea why its not working through Studio ?

 

Thanks for the help !

 

[Martin Stratmann]

Sorry for late response. Deleting and recreating of schedule reboot task via Studio didn´t fix it. 

 

Creating schedule tasks via  New-BrokerRebootScheduleV2 (decribed here) has done the job! It´s circuitous, but it works!

 

 

 

 

[Francois Callens]

Hello All,

 

I am new on this forum, and I read carefully all your messages. We meet exactly the same problem with reboot not working.

 

We upgraded from 7.8 to 7.12 last december (DDC + VDA), and since that moment scheduled reboots are not working properly.

 

In fact it works if the VDA server has no opened session. In the other way the server is skipped.

 

I tried to delete all the existing reboot schedule with the powershell command, and then recreate them with the "V2" commands (New-BrokerRebootScheduleV2). No success. I tried to add tags also, no success -> reboots don't work, with abandonned status in event.

 

I didn't try to downgrade the VDA agent version, I hope there is another solution with the VDA 7.12.

 

Francois

[Adam Wingler]

There is a private hotfix available to address VDA's not rebooting at their scheduled time after upgrading to 7.12.

[Christian Hock]

Where we can download this fix?

[Francois Callens]

You may ask it to Citrix support, because it is a private fix

[Christian Hock]

Solution:

 

https://jhmeier.com/2017/01/23/scheduled-reboots-not-working-on-citrix-xendesktop-7-12/

[Chris James1709157992]

Those of you having this problem, do your provisioned servers have a SCSI Adapter? Not sure if this fixed mine, but I've added the SCSI adapter, and my server now restart when scheduled with no issue.

 

Previously, one server would restart and then shutdown after 20 minutes, and the other would not reboot at all. Upgraded to 7.13 recently, but after reboot both servers shut down. So I'm going to (at the moment) assume the scsi adapter is the fix. I couldn't even live migrate in SCVMM, but now I can manage that task too.

[Nils Andreas Myhre]

Can you share a little more about your enviroment Chris?

 

What kind of SCSI adapter? What hypervisor are you running?

I too have this problem, even after upgrading to 7.13 where LC6766 is included.

Those of you having this problem, do your provisioned servers have a SCSI Adapter? Not sure if this fixed mine, but I've added the SCSI adapter, and my server now restart when scheduled with no issue.

 

Previously, one server would restart and then shutdown after 20 minutes, and the other would not reboot at all. Upgraded to 7.13 recently, but after reboot both servers shut down. So I'm going to (at the moment) assume the scsi adapter is the fix. I couldn't even live migrate in SCVMM, but now I can manage that task too.

[IT Leitstelle Tirol]

yep... still the same problem on 7.13

[Adam Holzberger]

Still the same problem on 7.13 with different customers.

[Aaron Barker1709155737]

I am having a similar problem. I use a PowerShell script to restart our XenApp 7.13 servers. The vSphere task log shows a task called "VirtualMachine.shutdownGuest.label", which is the initial shutdown command sent by MCS. Another task called "VirtualMachine.powerOn.label" is sent, which powers the server back on, but the server does not boot. The console shows a black screen and the server is not responsive. After about 20 to 30 minutes, XenApp still shows the server as powered on but unregistered, and a new task is sent to vSphere to shut down the guest OS again. This task fails, and is followed by a power off task.

 

This doesn't happen for every server, but a lot of them. When I use Citrix Studio to start the servers that failed to boot, some of them start successfully and some don't. They power on but don't complete the boot process. Eventually I can get them all to start by trying several times.

[IT Leitstelle Tirol]

i could solve our issue with the reboots. We had a faulty printer driver which blocked shutdowns and for some reason sometimes he also blocked the booting process.

 

i preinstalled the driver on the master images instead of deploying it via point-and-print group policy. now everything works fine.

[Ryan Campbell]

Have been seeing the same inconsistent issue, with 2008 R2 VDA's.

7.14 on controllers and VDA.

 

Following the trail I see, the reboot schedule action fire to vCenter as a power off action.

When things work good, I see a power on action and everything is happy.

 

When things go bad I do not see the power on action, but following PVS logs I see the machine boot.  But likely get hung on boot, then I see another poweroff action from the controller.  Assuming there is a timeout from the controller looking for the VDA to register after a scheduled reboot action, and when it doesn't see it in a certain time-frame it just powers the VDA off again.

 

I think my issue here is related to intermittent boot hang on PVS VDA's.

Any idea on how to tweak the controller behaviour to power on again after a power off action?

[Ryan Campbell]

Have been seeing the same inconsistent issue, with 2008 R2 VDA's.

7.14 on controllers and VDA.

 

Following the trail I see, the reboot schedule action fire to vCenter as a power off action.

When things work good, I see a power on action and everything is happy.

 

When things go bad I do not see the power on action, but following PVS logs I see the machine boot.  But likely get hung on boot, then I see another poweroff action from the controller.  Assuming there is a timeout from the controller looking for the VDA to register after a scheduled reboot action, and when it doesn't see it in a certain time-frame it just powers the VDA off again.

 

I think my issue here is related to intermittent boot hang on PVS VDA's.

Any idea on how to tweak the controller behaviour to power on again after a power off action?

 

I believe I found the fix in my case, from Patrick in:

https://discussions.citrix.com/topic/382996-xa-711712-win2012r2-vdas-do-not-boot-after-scheduled-restart/

 

Add the following registry key on Deliverry Controller

 

To enable the fix, set the following registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\DesktopServer\RebootSchedule

Name: ShutdownTimeoutRecovery

Type: DWORD

Value: 1

 

To disable the fix, set the following registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\DesktopServer\RebootSchedule

Name: ShutdownTimeoutRecovery

Type: DWORD

Value: 0

 

https://docs.citrix.com/en-us/xenapp-and-xendesktop/7-9/whats-new/fixed-issues.html

 

Seems to have resolved the issue, fingers crossed.

[Dennis Reimer1709157751]

One of our customers was experiencing this issue after upgrading from 7.6 to 7.15 LTSR. He is using Server 2008R2 SP1 (latest MS Updates) on XenServer 7.2. After upgrading to 7.15 LTSR we created new Machinecatalogues and Deliverygroups with a reboot schedule. But every morning some servers haven´t been started. Within the Xenserver Logs I can see that the VM has been Shutdown by XenDesktop but never been booted again. After trying different workarounds mentions in discussions or blogs I found a working solution for 7.15 LTSR:

 

If you create a reboot schedule within Citrix Studio. The Studio seems to create two reboot schedules, you can see them within powershell:

 

Add-PSSnapin citrix*

Get-BrokerRebootSchedule

Get-BrokerRebootScheduleV2

 

From my point of view I think this is the main problem. There are two Powershell commands trying to do the same. But this is only speculation.

 

As mentioned in several discussions I deleted both via the command below:

 

Remove-BrokerRebootSchedule -DesktopGroupName YourDeliveryGroupName

 

After that I set up a new RebootSchedule but only the V2

 

New-BrokerRebootScheduleV2  -AdminAddress "ddc01.domain.local:80" -Name Anythingyouwant -DesktopGroupName YourDeliveryGroupName -Enabled $True -Frequency "Daily" -RebootDuration 120 -StartTime "03:30:00" -WarningDuration 60 -WarningMessage "WARNING: This Computer will be Restarted in %m% minutes. Safe your work!" -WarningRepeatInterval 5 -Warningtitle "CAUTION: You will be logged off in a few minutes!"

 

With Get-BrokerRebootScheduleV2 you can see if the command was successfully.

 

If you now look at the reboot Schedule configuration of the deliverygroup within studio you will see it looks like there is nothing configured, because the GUI cannot read the V2 Rebootschedule.

 

After those changes I have been monitoring the Serverreboots for a week and every morning all servers are up and running.

 

Hope this helps.

 

Regards

 

Dennis

[Ronny Zacks]

On 11/30/2017 at 8:59 AM, Dennis Reimer1709157751 said:

One of our customers was experiencing this issue after upgrading from 7.6 to 7.15 LTSR. He is using Server 2008R2 SP1 (latest MS Updates) on XenServer 7.2. After upgrading to 7.15 LTSR we created new Machinecatalogues and Deliverygroups with a reboot schedule. But every morning some servers haven´t been started. Within the Xenserver Logs I can see that the VM has been Shutdown by XenDesktop but never been booted again. After trying different workarounds mentions in discussions or blogs I found a working solution for 7.15 LTSR:

 

If you create a reboot schedule within Citrix Studio. The Studio seems to create two reboot schedules, you can see them within powershell:

 

Add-PSSnapin citrix*

Get-BrokerRebootSchedule

Get-BrokerRebootScheduleV2

 

From my point of view I think this is the main problem. There are two Powershell commands trying to do the same. But this is only speculation.

 

As mentioned in several discussions I deleted both via the command below:

 

Remove-BrokerRebootSchedule -DesktopGroupName YourDeliveryGroupName

 

After that I set up a new RebootSchedule but only the V2

 

New-BrokerRebootScheduleV2  -AdminAddress "ddc01.domain.local:80" -Name Anythingyouwant -DesktopGroupName YourDeliveryGroupName -Enabled $True -Frequency "Daily" -RebootDuration 120 -StartTime "03:30:00" -WarningDuration 60 -WarningMessage "WARNING: This Computer will be Restarted in %m% minutes. Safe your work!" -WarningRepeatInterval 5 -Warningtitle "CAUTION: You will be logged off in a few minutes!"

 

With Get-BrokerRebootScheduleV2 you can see if the command was successfully.

 

If you now look at the reboot Schedule configuration of the deliverygroup within studio you will see it looks like there is nothing configured, because the GUI cannot read the V2 Rebootschedule.

 

After those changes I have been monitoring the Serverreboots for a week and every morning all servers are up and running.

 

Hope this helps.

 

Regards

 

Dennis

 

Hi Dennis,

the V2 schedules were introduced with v7.12 (as far as i am aware - http://tcafkb.blogspot.co.uk/2017/03/reboot-schedule-internals-citrix-blogs.html), so this may resolve the issue for those versions, but does not resolve or explain why this is happening in earlier versions.

we are seeing identical issues on XA/XD 7.9, PVS 7.11.0.6, Vsphere v5.5.0 Build 2417603 :

some machines reboot some do not, different machines every day .

running get-brokerrebootcycle i am getting the following output:

 

CatalogName           :
CatalogUid            :
DesktopGroupName      : DESKTOP
DesktopGroupUid       : 2
EndTime               : 05/12/2017 03:56:56
MachinesCompleted     : 2
MachinesFailed        : 3
MachinesInProgress    : 0
MachinesPending       : 0
MachinesSkipped       : 0
MetadataMap           : {}
RebootDuration        : 60
StartTime             : 05/12/2017 03:00:15
State                 : Completed
Uid                   : 1045
WarningDuration       : 5
WarningMessage        : This server will reboot in 5 minutes. Please save and logoff.
WarningRepeatInterval : 0
WarningTitle          :

 

I just added the below registry key to the DDCs:

HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\DesktopServer\RebootSchedule
Name: ShutdownTimeoutRecovery
Type: DWORD
Value: 1

as outlined in here: https://docs.citrix.com/en-us/xenapp-and-xendesktop/7-9/whats-new/fixed-issues.html?_ga=2.203626067.2113567864.1512472784-593911805.1484926031

 

fingers crossed, this may resolve this.. but im not very optimistic

any one found a better/lasting solution to this ?

 

 

[Navpreet Sidhu]

We still have this issue, last night matter of fact, using XenApp 7.17 w/MCS.  We have 10 VDA's, 2 of them failed to power up.

 

Any fixes?

 

There is no entry for "HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\DesktopServer\RebootSchedule"

[Mirco Geibel]

Hi kemba1.

 

You can ask Citrix Support for the VDA fix, if you got a SA on your Xenapp

 

https://jhmeier.com/2017/01/23/scheduled-reboots-not-working-on-citrix-xendesktop-7-12/

 

We had this issue since the Version 7.12. And the issue was and is still existing in further versions we installed.. (7.15, 7.17, 7.18)

we fixed it with this simple tool: https://tpetersit.blogspot.com/2017/01/citrix-reboot-scheduler.html

We are using 7.18 and it works great :)

 

But before setting up a schedule with this tool you need to delete your existing schedule plans in Citrix Studio. 

 

Greets

Mirco

 

 

Edited September 27, 2018 by GeibelM
add content

[Navpreet Sidhu]

Thanks, we do, I have case open as well.  I'll look into your links.  Thanks.

[Balaji Muthukrishnan]

One other possible reason were VDAs failed to power from its daily reboot schedule...

 

https://discussions.citrix.com/topic/395579-missing-registration-events-in-studio-console_with-fault-state-failed-to-start-power-state-off/page/2/

 

Its worth checking the Certificate thumbprint of the Certificate imported in Delivery controller with the one written to Database (Site Database, Table name : HypervisorConnectionSSLThumbprint). In my case it was weird that out of 4 VMWare Hyp Hosting connection i was able to see only 2 entries in the Tables for the Hyp Cpnnection (Esxi 5.5) but missing other 2 entries for 2 other Hyp Hosting connection (VMware Esxi 6.0/6.5)

 

This was identified when we did a Test connection against Hosting Connection in Studio console. test came out successfull, but DC was not able to connect to VC due to cert error

~~~~

Cannot connect to the vCenter server due to a certificate error. Make sure that the appropriate certificates are installed on the VCenter server, and install appropriate certificates on every controller in the site

~~~~

 

But i have the right certificates in all the DCs and in the right stores and they are trusted. Got this info from Carl's blog post and checked the (https://www.carlstalhood.com/delivery-controller-cr-and-licensing/#vcenter  - Hosting Resources Section, Point no 9) Cert thumprint in DB, surprisingly i was missing the hosting connection entry, If i create a new host connection, it is written back to the database tables (Test Host connection also didn't through back any cert errors), but for the existing host connection entry was missing.

 

This was causing VDAs Failure to Power On from its daily reboot schedules resulting in FailedToStart errors. we were not able to power on from Studio/Director or through powershell cmd. The only way is to login to vSphere web console and power on. If we reboot the Delivery controller issue will be resolved temporarily but after a week or so issue is coming back

 

I guess the only option is to create a new Hosting connection for the missing Hyp entries and move the VDAs to the newly created Hosting connection

[Balaji Muthukrishnan]

pl refer...

 

https://discussions.citrix.com/topic/375659-cannont-connect-to-vcenter-server-due-to-a-certificate-error/page/2/#comment-2055820

[David Joye]

Informational:

I initially used a VMware monitoring tool (Veeam ONE Monitor) to work around this. I created an alarm that recognized the powered-off VM and then had Veeam ONE power on the VM after a specified amount of time; this is not clean. While the machine will be re-registered when it boots and users will be able to access it, any queued image updates will not apply when the VM is powered on by something other than the Citrix controller. After recognizing this, I restarted the machine via Studio and the image updated on that machine, same result on other machines that were powered on by the monitoring tool.

 

tl;dr: Don't use an external tool to power on the VMs if using MCS, use the provided registry fixes.