Jump to content
Updated Privacy Statement
  • 2

XenServer tools timesync disable


Patrik Nyberg

Question

I recently encountered a very annoying problem on my Windows 10/Server 2016 VMs running on XenServer 7.1 CU1.

On startup of the VMs the time was advanced 54 minutes when XenInterface/XenAgent started, when the w32time service started on the VM the time was then synchronized to the DCs of the domain and was ok again.

 

My issues I got was that some of my VMs lost connection to the domain - because the time was off with more than 5 minutes, and this was happening randomly - depending on when the Xen Agents was starting. The impact for me was that the startup scripts configured in the domain was not running on the VMs...

 

My troubleshooting was to check NTP settings on the hosts, on the DCs and the HW clock on the Hosts - all to no avail - meaning no clock issues on any of them. Also the timeoffset on the VM metadata I tried to set differently - but also did not change the behaviour.

 

So in the end, my only workaround left was to try to disable the timesync between the VMs XenAgent (XenInterface) and the host. But no official documentation stated that this was possible..

However, after a bit of digging I found in the Xen source code for the XenAgent - it seemed there was a registry setting inside the VM that was checked for a specific value - if that was found it would synchronize the VMs' time with the host. So in order to disable that you can just set tha t value to anything else - or remove the value all together would also do the trick I believe.

 

The registry key/value to disable time sync with host (works on Windows 10 VMs (tested on 1709),  Server 2016, XenServer tools 7.1.1305, XenServer 7.1 CU1):

 

HKLM\Software\Citrix\XenTools\HostTime="Local"

HKLM\Software\Wow6432Node\Citrix\XenTools\HostTime="Local"

 

Default value is HostTime="UTC" - so set anything else than that - should disable sync with the host.

 

For me the problem went away completely and I have implemented this registry setting on all my VMs now.

 

 

Link to comment
  • Answers 57
  • Created
  • Last Reply

Top Posters For This Question

Recommended Posts

  • 0
On 9/5/2019 at 10:20 AM, Michael Martis said:

I also experienced the time sync. issue after the management agent was updated from 7.1.1327 to 7.1.1396
Both keys below had value of UTC
HKLM\SOFTWARE\Citrix\XenTools\HostTime

HKLM\SOFTWARE\Wow6432Node\Citrix\XenTools\HostTime 

 

Just for peace of mind, cleared both values , and it fixed the issue. 

 

I found it odd, for Wow6432Node key, HostTime was the only value. No other values or sub keys.  ~shrug~

 

I have the same issue, with the same version numbers as well. Doing this resolved the issue for us as well. Thank you.

Link to comment
  • 0

Applied all 7.1 CU2 patches then I updated all my 2016 VMs to 7.1.1396 recently - out of 30 VMs 3 of them had this issue.  They would not register in Studio and this was why.  The 3 had UTC in those registry values.  The rest of the VMs (spot checked) have a blank value in those two registry locations.  Same windows patch level on each VM.

 

Cleared out these registry key on the 3 - fingers crossed.  Thanks for sharing.

Link to comment
  • 0

Hi Mike,

 

That sounds for me as if these servers would have a different time source to sync with or different settings such as frequency of sync's. Did you check / compare the time sync config on those systems:

w32tm /query /source
w32tm /query /configuration
w32tm /query /peers
w32tm /query /status

You might also try to observe few systems to see if they jump around or so:

w32tm /stripchart /computer:<your-time-source-fqdn> /period5

and compare it to some other time source such as:

w32tm /stripchart /computer:pool.ntp.org /period5

BfN, Konrad

Link to comment
  • 0
On 4/5/2020 at 9:14 AM, Mike Berry said:

I also had this issue and now running the most current XenTools. 7.2.1555.  I have some 2012R2 Servers (Not PVS'd, just standard vm)  that we removed the UTC from those reg keys and are spot on with time.  PVS'd 2012R2 Servers, even though have same fix and driver I am finding have time drift of anywhere from 2 minutes before "correct time" to two minutes after.  Below is just a sampling...  webxxx you can see are in sync as opposed to the hsxxx pvs'd servers.  Our XenServers are LSTR and current with all patches, VM's have up today tools (7.2.1555) and XenApp CU5, PVS is 7.15 LTSR with Current CU5 as well.  No matter what I try, w32tm /resync, etc.....  they are never in "sync" like other Servers in our environment.  

Mike I have this exact same problem since last Sunday 03.00 when our DST became active. (Netherlands)
This whole week I've been searching for the cause and I noticed several things:

 

On VM's with a recent XenTools version, I noticed in the System eventlog that every 30 minutes the system time is adjusted by Kernel-General by a few seconds.

This also happened on my AD's so the Time Sync was completely off track.

 

After a lot of testing and frustration I found the following:

I had to stop the XenServer Agent service on my AD's to get rid of the time adjustments.

Now my time is in sync again with an external NTP.

 

Currently I'm also testing this procedure with other VM's who are also drifting and not syncing their time with my AD's.

 

This problem is not occurring with version 7.1.1229 VM's that I also have running.

 

Hope this also helps you.

 

Ard

 

 

Link to comment
  • 0

Hello,

I´m back with the same problem... the time is running out of sync. Its only with xenserver, our vmware clients dont have this problem.

I can reproduce it: with "w32tm /stripchart /computer:dc01 /period:5" i see the time from the client against our timeserver. from minute to minute enlarged the difference.

now every 30 minutes triggers the xenserver agent (i use 7.2.0.1537) and syncs the time. but not to the exact time from the timeserver (a physical server!)... it was seconds away from the realtime. Have a look at the screenshot...

Actually i test a workaround with a manual "w32tm /resync" - every 15 minutes.

 

Is it possible to deactivate the trigger from the agent? The clients are in an AD....

xenservertime.thumb.PNG.60ce652367424213dd508504c0314f49.PNG

Best regards,

Ralf

Link to comment
  • 0

Hallo Ralf,

 

You could try to get some better understanding from what happens by analysing the output of

w32

w32tm /query /peers
w32tm /query /status

"peers" will tell you when each peer is being contacted, whereas "status" will tell you more insights about the drift and last sync.

Many of the time sync params can be configured through registry / GPO's.

Details about the current client config:

w32tm /query /configuration

BfN, Konrad

Link to comment
  • 0

Yesterday I worked with MS (and have Citrix Case open) and they had me hardcode the IP of one of the DC's in the Servers below, that member DC that was hard coded to
PDC Emulator.  This was the output from this morning.  (HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Parameters NTPServer)

DC1  04/08/2020 07:52:27
DC2  04/08/2020 07:52:27

HS001  04/08/2020 07:52:28
HS002  04/08/2020 07:52:27
HS003  04/08/2020 07:52:27
HS004  04/08/2020 07:52:23
HS005  04/08/2020 07:52:10
DC1  04/08/2020 07:52:27
HS006  04/08/2020 07:52:09
HS007  04/08/2020 07:52:12
HS008  04/08/2020 07:52:26
HS009  04/08/2020 07:52:28
HS010  04/08/2020 07:52:26
DC1  04/08/2020 07:52:28
HS011  04/08/2020 07:52:29
HS012  04/08/2020 07:52:17
HS013  04/08/2020 07:52:28
HS014  04/08/2020 07:52:28
HS015  04/08/2020 07:52:27
HS016  04/08/2020 07:52:28

Interesting enough, when I run w23tm /query /source .. status ... peers they show some other DC.  here is some of the output from two servers
in the list above that were off at least 20 seconds from one another.  They are two different DC's but when I check times between all DC's in
org, they are in perfect sync with their times

 

E:\_Wid_>w32tm /query /configuration
[Configuration]

EventLogFlags: 2 (Local)
AnnounceFlags: 10 (Local)
TimeJumpAuditOffset: 28800 (Local)
MinPollInterval: 10 (Local)
MaxPollInterval: 15 (Local)
MaxNegPhaseCorrection: 4294967295 (Local)
MaxPosPhaseCorrection: 4294967295 (Local)
MaxAllowedPhaseOffset: 300 (Local)

FrequencyCorrectRate: 4 (Local)
PollAdjustFactor: 5 (Local)
LargePhaseOffset: 50000000 (Local)
SpikeWatchPeriod: 900 (Local)
LocalClockDispersion: 10 (Local)
HoldPeriod: 5 (Local)
PhaseCorrectRate: 1 (Local)
UpdateInterval: 30000 (Local)


[TimeProviders]

NtpClient (Local)
DllName: C:\Windows\system32\w32time.dll (Local)
Enabled: 1 (Local)
InputProvider: 1 (Local)
CrossSiteSyncFlags: 2 (Local)
AllowNonstandardModeCombinations: 1 (Local)
ResolvePeerBackoffMinutes: 15 (Local)
ResolvePeerBackoffMaxTimes: 7 (Local)
CompatibilityFlags: 2147483648 (Local)
EventLogFlags: 1 (Local)
LargeSampleSkew: 3 (Local)
SpecialPollInterval: 3600 (Local)
Type: NT5DS (Local)

VMICTimeProvider (Local)
DllName: C:\Windows\System32\vmictimeprovider.dll (Local)
Enabled: 1 (Local)
InputProvider: 1 (Local)
NtpServer (Local)
DllName: C:\Windows\system32\w32time.dll (Local)
Enabled: 0 (Local)
InputProvider: 0 (Local)


E:\_Wid_>w32tm /query /peers
#Peers: 1

Peer: ahvadd.somedomain.org
State: Active
Time Remaining: 92.6170218s
Mode: 3 (Client)
Stratum: 4 (secondary reference - syncd by (S)NTP)
PeerPoll Interval: 10 (1024s)
HostPoll Interval: 10 (1024s)

E:\_Wid_>w32tm /query /status
Leap Indicator: 0(no warning)
Stratum: 5 (secondary reference - syncd by (S)NTP)
Precision: -6 (15.625ms per tick)
Root Delay: 0.1035004s
Root Dispersion: 5.2563088s
ReferenceId: 0xAC18AA41 (source IP:  172.x.x.x)
Last Successful Sync Time: 4/8/2020 8:05:39 AM
Source: ahvadd.somedomain.org
Poll Interval: 10 (1024s)

=====================================

C:\OtherApps>w32tm /query /source
AHDC.somedomain.org

C:\OtherApps>w32tm /query /configuration
[Configuration]

EventLogFlags: 2 (Local)
AnnounceFlags: 10 (Local)
TimeJumpAuditOffset: 28800 (Local)
MinPollInterval: 10 (Local)
MaxPollInterval: 15 (Local)
MaxNegPhaseCorrection: 4294967295 (Local)
MaxPosPhaseCorrection: 4294967295 (Local)
MaxAllowedPhaseOffset: 300 (Local)

FrequencyCorrectRate: 4 (Local)
PollAdjustFactor: 5 (Local)
LargePhaseOffset: 50000000 (Local)
SpikeWatchPeriod: 900 (Local)
LocalClockDispersion: 10 (Local)
HoldPeriod: 5 (Local)
PhaseCorrectRate: 1 (Local)
UpdateInterval: 30000 (Local)


[TimeProviders]

NtpClient (Local)
DllName: C:\Windows\system32\w32time.dll (Local)
Enabled: 1 (Local)
InputProvider: 1 (Local)
CrossSiteSyncFlags: 2 (Local)
AllowNonstandardModeCombinations: 1 (Local)
ResolvePeerBackoffMinutes: 15 (Local)
ResolvePeerBackoffMaxTimes: 7 (Local)
CompatibilityFlags: 2147483648 (Local)
EventLogFlags: 1 (Local)
LargeSampleSkew: 3 (Local)
SpecialPollInterval: 3600 (Local)
Type: NT5DS (Local)

VMICTimeProvider (Local)
DllName: C:\Windows\System32\vmictimeprovider.dll (Local)
Enabled: 1 (Local)
InputProvider: 1 (Local)
NtpServer (Local)
DllName: C:\Windows\system32\w32time.dll (Local)
Enabled: 0 (Local)
InputProvider: 0 (Local)


C:\OtherApps>w32tm /query /peers
#Peers: 1

Peer: AHDC.somedomain.org
State: Active
Time Remaining: 435.9544659s
Mode: 3 (Client)
Stratum: 4 (secondary reference - syncd by (S)NTP)
PeerPoll Interval: 10 (1024s)
HostPoll Interval: 10 (1024s)

C:\OtherApps>w32tm /query /status
Leap Indicator: 0(no warning)
Stratum: 5 (secondary reference - syncd by (S)NTP)
Precision: -6 (15.625ms per tick)
Root Delay: 0.1484070s
Root Dispersion: 2.1566554s
ReferenceId: 0xAC18AA64 (source IP:  172.x.x.x)
Last Successful Sync Time: 4/8/2020 8:15:49 AM
Source: AHDC.somedomain.org
Poll Interval: 10 (1024s)

 

 

If I do a net time /DOMAIN.somedomain.org /set /y they are all in sync at that point in time but within 30 min they all start to skew again.. The longer they are running they longer they skew.   So frustrating !!

Link to comment
  • 0

Ard-Jan Verhage,

I'll try disabling the xen agent service and see if that helps at all. 

 

Konrad Ruess                        

I'll also do some of the testing you recommended as well to see if anything gets flushed out..  To be honest MS and Citrix Support haven't been real helpful so far...  : (

 

Link to comment
  • 0
5 hours ago, Konrad Ruess said:

Hallo Ralf,

 

You could try to get some better understanding from what happens by analysing the output of

w32


w32tm /query /peers
w32tm /query /status

"peers" will tell you when each peer is being contacted, whereas "status" will tell you more insights about the drift and last sync.

Many of the time sync params can be configured through registry / GPO's.

Details about the current client config:


w32tm /query /configuration

BfN, Konrad

Posted some results in thread

Link to comment
  • 0
On ‎4‎/‎6‎/‎2020 at 5:29 AM, Ard-Jan Verhage said:

Mike I have this exact same problem since last Sunday 03.00 when our DST became active. (Netherlands)
This whole week I've been searching for the cause and I noticed several things:

 

On VM's with a recent XenTools version, I noticed in the System eventlog that every 30 minutes the system time is adjusted by Kernel-General by a few seconds.

This also happened on my AD's so the Time Sync was completely off track.

 

After a lot of testing and frustration I found the following:

I had to stop the XenServer Agent service on my AD's to get rid of the time adjustments.

Now my time is in sync again with an external NTP.

 

Currently I'm also testing this procedure with other VM's who are also drifting and not syncing their time with my AD's.

 

This problem is not occurring with version 7.1.1229 VM's that I also have running.

 

Hope this also helps you.

 

Ard

 

 

So far disabling agent seems to be promising.

Link to comment
  • 0
1 hour ago, Mike Berry said:

So far disabling agent seems to be promising.

 

Same here, time sync is running much better now.

I don't know what the service actually does and if it can be disabled without consequences...

...it was introduced in a XenTools 7.1.0.13xx version. Before that it didn't exist.

 

 

Link to comment
  • 0
39 minutes ago, Ard-Jan Verhage said:

 

Same here, time sync is running much better now.

I don't know what the service actually does and if it can be disabled without consequences...

...it was introduced in a XenTools 7.1.0.13xx version. Before that it didn't exist.

 

 

Still running some test but yes, it seems much better with the Xen Agent stopped.  I've verified all DC's were all in sync with each other with w32tm /monitor.  I have also used the w32tm /stripchart /computer:hostname /dataonly on some nodes out of sync for validation along with other commands listed in this thread.  On host's that still have the agent running and out of sync, we have verified they are pulling their time from our DC's (which are in sync with our PDC Emulator)

 

I have asked Citrix to move my case to their development team.  I'll let you know what they find.

Link to comment
  • 0

Sorry, I was going to post this yesterday but time got away....  Having the XenService stopped or disabled does effect communication between VM's and Hypervisor...  as a side effect that impacts reboots cycles, etc...  I did get a call back from Citrix and they say they are aware of this issues and a fix was coming out this month for 7.15 LTSR but also said that the issue was fixed on 8.1 Hypervisor (It's not, I have the same issue on that version as well) so not overly confident that the fix coming out is to address the time drift over time (several seconds or a minute or two) as opposed to time issue caused from UTC reg value that was causing time to be off by hours

Link to comment
  • 0
On ‎4‎/‎17‎/‎2020 at 5:27 AM, Markus Einberger1709159532 said:

We can reproduce exaltly the same issues with CVAD 7.15 LTSR CU5 and XenServer 7.1 CU2

Host is in sync, serveral guests are out of sync

we referenced in our Case Number 79676630 to the other case 79642045.

I received an email from our EMR vendor that was also able to reproduce issue in their lab.  They were given a private fix and they said it seems to fix their issue.  Hopefully a public fix will be out shortly

Link to comment
  • 0

I heard back from Citrix this morning and this is the most current update

Also wanted to let you know that the hotfix for XenServer 7.1 CU2 LTSR will be XS71ECU2035 when it gets released later this month.

 

They are also tracking potentially another issue in XenServer 8.1

 

I'll keep this post up to day as I get more information.

Link to comment
  • 0

Update for time sync issue on Xen Server 8.1.x for escalation

 

As you may already know, we opened this other case to track your time sync issue in your Hypervisor 8.1.

Just as a quick overview, there is a known issue on Citrix Hypervisor 8.1, on Windows VMs with latest XenTools, V9 drivers, where there is a time sync drift every 30 minutes or so.

Basically, there is a race between XenAgent time sync and AD time Sync.

There are some workarounds for this that we know:

1. Disable AD time sync and only allow XenAgent time sync.

2. Disable XenAgent and leave AD time sync, but by doing this you will loose graceful shutdown, reboot and some other features of XenAgent on VMs.

3. The last workaround is to install a third party time sync tool. We know of NetTime v3.14 available for free from www.timesynctool.com. This utility can be set to sync against domain controllers internally and resync after a period of time. We currently have it set to sync every 15 minutes and adjust time if the skew is more than 1 second then adjust the time....This has so far been working great on their test environment.

Just as a side note, we haven't completely tested these workarounds, some of them were tested by other users and we don't know if there are other implications so far, so I just provided them just in case you are in any urgency and would like to test.

On the other hand, we have an internal case with our developers, they are working on a possible fix that basically will allow you to disable XenAgent time sync only without disabling other features of XenAgent, and leaving the users to be able to use AD time sync only for the VMs.

For this, we don't have an ETA yet but I will provide details as soon as we get them.

 

Link to comment
  • 0

Support assures me that the fix for 7.x CU2 LSTR will be out by end of week.  It's a XenServer patch XS71ECU2035.  As far as the issue with 8.1.x, they are aware of the issue but don't have an eta for fix yet.  I tested disabling Xen Agent like others but that's disruptive when it comes to reboots, etc.  I ran a script to stop the time service on host in one DataCenter.  That seemed to work as a short gap but messed with me mentally having to disable / stop native windows time sync functionality.  I believe someone else posted this earlier and what I ended up doing for the short term, which was to  scheduled a task to run every min to for a sync.  Time drift is still their but from what I can tell no more that 5 seconds between any one host.   If you want to try that, I've attached the scheduled task xml.   The syntax to push out to a list of servers using psexec was...  

psexec @f:\scripts\listofservers.txt schtasks /Create /XML "d:\timesync.xml" /TN "TimeSync"  After it runs a few times, the servers will be in sync, "well as much as they can +/-a few seconds"   

 

I'll update thread as I know more.

timesync.xml

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×
×
  • Create New...