Jump to content
  • 0

After Upgrade from Xenserver 7.1 CU2 to Citrix hypervisor 8.2, got an session login error from the slave


Merlin Radebold

Question

Hi All,

I have got a question about the Upgrade from Xenserver 7.1 CU2 to Citrix hypervisor 8.2.

After the Upgrade I got the following error message in our Log. For me it looks like the slave could not establish a connection to the Master. The error is coming up every minute. 

xenserver2 xapi: [error||401 INET :::80||backtrace] session.login_with_password D:XXXXXXXXXXX failed with exception Server_error(HOST_IS_SLAVE, [ XXX.XXX.XXX.XXX ])
xenserver2 xapi: [error||401 INET :::80||backtrace] Raised Server_error(HOST_IS_SLAVE, [ XXX.XXX.XXX.XXX ])
xenserver2 xapi: [error||401 INET :::80||backtrace] 1/8 xapi Raised at file ocaml/xapi/xapi_session.ml, line 492
xenserver2 xapi: [error||401 INET :::80||backtrace] 2/8 xapi Called from file ocaml/xapi/xapi_session.ml, line 45
xenserver2 xapi: [error||401 INET :::80||backtrace] 3/8 xapi Called from file ocaml/xapi/xapi_session.ml, line 47
xenserver2 xapi: [error||401 INET :::80||backtrace] 4/8 xapi Called from file ocaml/xapi/server_helpers.ml, line 95
xenserver2 xapi: [error||401 INET :::80||backtrace] 5/8 xapi Called from file ocaml/xapi/server_helpers.ml, line 121
xenserver2 xapi: [error||401 INET :::80||backtrace] 6/8 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xenserver2 xapi: [error||401 INET :::80||backtrace] 7/8 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 35
xenserver2 xapi: [error||401 INET :::80||backtrace] 8/8 xapi Called from file lib/backtrace.ml, line 177
xenserver2 xapi: [error||401 INET :::80||backtrace]
xenserver2 xapi: [debug||402 INET :::80|Get RRD updates. D:6bc56def144f|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
xenserver2 xapi: [debug||202 |watching networks for NBD-related changes D:XXXXXXXXXXX |xmlrpc_client] stunnel pid: 8304 (cached = true) connected to XXX.XXX.XXX.XXX:443
xenserver2 xapi: [debug||202 |watching networks for NBD-related changes D:XXXXXXXXXXX |xmlrpc_client] with_recorded_stunnelpid task_opt=None s_pid=8304
xenserver2 xapi: [debug||202 |watching networks for NBD-related changes D:XXXXXXXXXXX |xmlrpc_client] stunnel pid: 8304 (cached = true) returned stunnel to cache
xenserver2 xapi: [debug||202 |watching networks for NBD-related changes D:XXXXXXXXXXX |xmlrpc_client] stunnel pid: 6940 (cached = true) connected to XXX.XXX.XXX.XXX:443
xserver2 xapi: [debug||202 |watching networks for NBD-related changes D:XXXXXXXXXXX |xmlrpc_client] with_recorded_stunnelpid task_opt=None s_pid=6940
xenserver2 xcp-rrdd: [ info||7 ||rrdd_main] GC live_words = 345851
xenserver2 xcp-rrdd: [ info||7 ||rrdd_main] GC heap_words = 1118208 xenserver2 xcp-rrdd: [ info||7 ||rrdd_main] GC free_words = 772346
xenserver2 xapi: [debug||44 heartbeat|Heartbeat D:66378090f9ae|stunnel] stunnel start
xenserver2 xapi: [debug||44 heartbeat|Heartbeat D:66378090f9ae|xmlrpc_client] stunnel pid: 24411 (cached = false) connected to XXX.XXX.XXX.XXX:443
xenserver2 xapi: [debug||44 heartbeat|Heartbeat D:66378090f9ae|xmlrpc_client] with_recorded_stunnelpid task_opt=None s_pid=24411
xenserver2 xapi: [debug||201 |timeboxed_rpc D:2c287dacb3d1|xmlrpc_client] stunnel pid: 8647 (cached = true) returned stunnel to cache
xenserver2 xapi: [debug||201 ||dummytaskhelper] task timeboxed_rpc D:b43aa0e65a7f created by task D:3910a28570a1
xenserver2 xapi: [debug||201 |timeboxed_rpc D:b43aa0e65a7f|xmlrpc_client] stunnel pid: 8652 (cached = true) connected to XXX.XXX.XXX.XXX:443
xenserver2 xapi: [debug||201 |timeboxed_rpc D:b43aa0e65a7f|xmlrpc_client] with_recorded_stunnelpid task_opt=DummyRef:|b43aa0e6-5a7f-40fb-8caf-522e1c0b486e|timeboxed_rpc s_pid=8652
xenserver2 xapi: [debug||44 heartbeat|Heartbeat D:66378090f9ae|stunnel] stunnel start
xenserver2 xapi: [debug||44 heartbeat|Heartbeat D:66378090f9ae|xmlrpc_client] stunnel pid: 24619 (cached = false) connected to XXX.XXX.XXX.XXX:443
xenserver2 xapi: [debug||44 heartbeat|Heartbeat D:66378090f9ae|xmlrpc_client] with_recorded_stunnelpid task_opt=None s_pid=24619

It is weird, because the VMs are running fine and there is no other problem with it.

What we tried yet:

·         Restart Toolstack

·         Restart Server

·         Renew the Pool token

·         Restart the NTP Service

·         Clean Install with Hypervisor 8.2

Now I do not know what else I can try. I would be happy for any suggestions.

Best Regards

Merlin

Link to comment

13 answers to this question

Recommended Posts

  • 0
On 11/6/2020 at 8:04 PM, Tobias Kreidl said:

Time is synched OK to NTP and within reasonable tolerances? The hotfixes are identical on all servers?

Yes all server got the identical hotfixes. The last applied hotfix is "XS82E009".

I have got the following tolerances:

XenServer01:

[root@Xenserver01 sysconfig]# chronyc tracking
Reference ID    : 0A1C0401 (ntp01.domain.de)
Stratum         : 3
Ref time (UTC)  : Mon Nov 09 10:13:25 2020
System time     : 0.002282776 seconds fast of NTP time
Last offset     : +0.000296919 seconds
RMS offset      : 0.002853009 seconds
Frequency       : 9.896 ppm slow
Residual freq   : +0.106 ppm
Skew            : 3.752 ppm
Root delay      : 0.046875019 seconds
Root dispersion : 0.060823191 seconds
Update interval : 256.2 seconds
Leap status     : Normal
[root@Xenserver01 sysconfig]# chronyc sources
210 Number of sources = 3
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* ntp01.domain.de		        2   9   377   469  -5196us[-4899us] +/-   84ms
^+ ntp02.domain.de 		        3   6   377    10   +723us[ +723us] +/-  184ms
^+ ntp03.domain.de		        3   8   377   142  +7304us[+7304us] +/-  183ms

 XenServer02:

[root@XenServer2 ~]# chronyc tracking
Reference ID    : 0A1C0401 (ntp01.domain.de)
Stratum         : 3
Ref time (UTC)  : Mon Nov 09 10:07:36 2020
System time     : 0.000823885 seconds fast of NTP time
Last offset     : -0.000067807 seconds
RMS offset      : 0.000319652 seconds
Frequency       : 49.278 ppm slow
Residual freq   : -0.000 ppm
Skew            : 0.091 ppm
Root delay      : 0.046875022 seconds
Root dispersion : 0.061535560 seconds
Update interval : 1043.9 seconds
Leap status     : Normal
[root@XenServer02 ~]# chronyc sources
210 Number of sources = 3
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* ntp01.domain.de        2  10   377   390   +309us[ +241us] +/-   98ms
^+ ntp02.domain.de        3  10   377   689   +793us[ +726us] +/-  172ms
^+ ntp02.domain.de        3  10   377   219    +10ms[  +10ms] +/-  181ms

 

Link to comment
  • 0

Merlin,

Looking more carefully at the error messages, I see "failed with exception Server_error(HOST_IS_SLAVE)".

Maybe you need to force the pool master to be one or the other host. Do you have 2 hosts or more than 2 hosts in your pool?

Running with just 2 hosts with HA can lead to issues, just as an FYI.

 

You can follow the procedure outlined here:  https://support.citrix.com/article/CTX139197

It certainly cannot hurt at this point.

 

-=Tobias

Link to comment
  • 0

Hmmm, do you have HA enabled? It's not officially supported on a 2-node pool. If so, consider using https://www.halizard.com/

With 2 nodes, if one host is down and comes back up, the pool master selection can get confused. Make sure also the hosts are properly NTP-synchronized with each other.

Check to be sure one is the amaster and one is not; on each host, run: cat /etc/xensource/pool.conf

-=Tobias

Link to comment
  • 0

For me everything looks good. I could not find any errors.

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether XX:XX:XX:XX:XX:XX  txqueuelen 1000  (Ethernet)
        RX packets 716914  bytes 121904347 (116.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 42  memory 0xdb800000-dbffffff

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether XX:XX:XX:XX:XX:XX  txqueuelen 1000  (Ethernet)
        RX packets 11305770  bytes 14821112126 (13.8 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 19841872  bytes 28902440295 (26.9 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 45  memory 0xdc800000-dcffffff

eth2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether XX:XX:XX:XX:XX:XX  txqueuelen 1000  (Ethernet)
        RX packets 27423230  bytes 21451275005 (19.9 GiB)
        RX errors 140  dropped 1  overruns 140  frame 0
        TX packets 2871255  bytes 1687601920 (1.5 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 46  memory 0xcbfff000000-cbfff7fffff

eth3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether XX:XX:XX:XX:XX:XX  txqueuelen 1000  (Ethernet)
        RX packets 27029188  bytes 21280832223 (19.8 GiB)
        RX errors 97  dropped 0  overruns 97  frame 0
        TX packets 2653454  bytes 1766091403 (1.6 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 44  memory 0xcbffe000000-cbffe7fffff

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 833795  bytes 1510404507 (1.4 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 833795  bytes 1510404507 (1.4 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

xapi0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet XXX.XXX.XXX.XXX  netmask 255.255.255.0  broadcast XXX.XXX.XXX.XXX
        ether XX:XX:XX:XX:XX:XX  txqueuelen 1000  (Ethernet)
        RX packets 2538878  bytes 14236636105 (13.2 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1461828  bytes 27618700295 (25.7 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

xapi1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether XX:XX:XX:XX:XX:XX  txqueuelen 1000  (Ethernet)
        RX packets 19601488  bytes 12077955358 (11.2 GiB)
        RX errors 0  dropped 7259  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

xapi4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether XX:XX:XX:XX:XX:XX  txqueuelen 1000  (Ethernet)
        RX packets 451412  bytes 22455417 (21.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

 

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...