xapi high CPU load - pool slow

[Jimmy Koerting]

Hi there,

 

we have a pool (7.2) with 25 host machines and about 900 VMs.

It runs since month without any problems. Since yesterday all API related actions (not the VMs) are very, very slow. I checked logs etc and found that xapi is running with more than 100% CPU load (multple CPUs, therefore more than 100%).

 

One obvious effect is that there are tons of

 

VM.update_allowed_operations

 

tasks on the pool master.

No warning/error messages in xensource.log except every few minutes:

 

May 15 23:52:53 xenserver72-y26 xapi: [error|xenserver72-y26|540 INET :::80|dispatch:pool_update.detach D:e194113687ad|backtrace] pool_update.detach R:2a7d62ff9184 failed with exception Server_error(CANNOT_FIND_UPDATE, [ 072886d9-4a3b-48a1-a306-0edf13aa7420 ])
May 15 23:52:53 xenserver72-y26 xapi: [error|xenserver72-y26|540 INET :::80|dispatch:pool_update.detach D:e194113687ad|backtrace] Raised Server_error(CANNOT_FIND_UPDATE, [ 072886d9-4a3b-48a1-a306-0edf13aa7420 ])
May 15 23:52:53 xenserver72-y26 xapi: [error|xenserver72-y26|540 INET :::80|dispatch:pool_update.detach D:e194113687ad|backtrace] 1/6 xapi @ xenserver72-y26 Raised at file rbac.ml, line 236
May 15 23:52:53 xenserver72-y26 xapi: [error|xenserver72-y26|540 INET :::80|dispatch:pool_update.detach D:e194113687ad|backtrace] 2/6 xapi @ xenserver72-y26 Called from file server_helpers.ml, line 76
May 15 23:52:53 xenserver72-y26 xapi: [error|xenserver72-y26|540 INET :::80|dispatch:pool_update.detach D:e194113687ad|backtrace] 3/6 xapi @ xenserver72-y26 Called from file server_helpers.ml, line 91
May 15 23:52:53 xenserver72-y26 xapi: [error|xenserver72-y26|540 INET :::80|dispatch:pool_update.detach D:e194113687ad|backtrace] 4/6 xapi @ xenserver72-y26 Called from file lib/pervasiveext.ml, line 22
May 15 23:52:53 xenserver72-y26 xapi: [error|xenserver72-y26|540 INET :::80|dispatch:pool_update.detach D:e194113687ad|backtrace] 5/6 xapi @ xenserver72-y26 Called from file lib/pervasiveext.ml, line 26
May 15 23:52:53 xenserver72-y26 xapi: [error|xenserver72-y26|540 INET :::80|dispatch:pool_update.detach D:e194113687ad|backtrace] 6/6 xapi @ xenserver72-y26 Called from file lib/backtrace.ml, line 176
May 15 23:52:53 xenserver72-y26 xapi: [error|xenserver72-y26|540 INET :::80|dispatch:pool_update.detach D:e194113687ad|backtrace]

and

 

May 15 23:54:10 xenserver72-y26 xenopsd-xc: [ warn|xenserver72-y26|64817 |org.xen.xapi.xenops.classic events D:fb915738e0e8|memory] Failed to parse ionice result: unknown: prio 4
May 15 23:54:11 xenserver72-y26 xenopsd-xc: [ warn|xenserver72-y26|64818 |org.xen.xapi.xenops.classic events D:fb915738e0e8|memory] Failed to parse ionice result: unknown: prio 4
May 15 23:54:11 xenserver72-y26 xenopsd-xc: [ warn|xenserver72-y26|64820 |org.xen.xapi.xenops.classic events D:fb915738e0e8|memory] Failed to parse ionice result: unknown: prio 4

 

That's all. When enabling debug/info messages, I get tons of (filtered with grep for 'VM.update_allowed_operations'):

 

May 16 00:05:21 xenserver72-y26 xapi: [debug|xenserver72-y26|2830 INET :::80||dummytaskhelper] task dispatch:VM.update_allowed_operations D:614e9f92f674 created by task D:c3b0ebd5232a
May 16 00:05:21 xenserver72-y26 xapi: [debug|xenserver72-y26|612 INET :::80||dummytaskhelper] task dispatch:VM.update_allowed_operations D:4e44c36a40d8 created by task D:a12c4bc9c14d
May 16 00:05:21 xenserver72-y26 xapi: [debug|xenserver72-y26|209 INET :::80||dummytaskhelper] task dispatch:VM.update_allowed_operations D:355028e409ce created by task D:cef2ab9a8258
May 16 00:05:21 xenserver72-y26 xapi: [ info|xenserver72-y26|2830 INET :::80|dispatch:VM.update_allowed_operations D:614e9f92f674|taskhelper] task VM.update_allowed_operations R:65373c8e3d29 (uuid:21055373-2fd1-ec3c-65b1-69494bfbacae) created (trackid=4bceee1a5555baaf325ae95776481242) by task D:c3b0ebd5232a
May 16 00:05:21 xenserver72-y26 xapi: [debug|xenserver72-y26|2830 INET :::80|VM.update_allowed_operations R:65373c8e3d29|audit] VM.update_allowed_operations: VM = '48a665fc-1351-06c6-2fc7-838664af8727 (denaxi)'
May 16 00:05:21 xenserver72-y26 xapi: [ info|xenserver72-y26|612 INET :::80|dispatch:VM.update_allowed_operations D:4e44c36a40d8|taskhelper] task VM.update_allowed_operations R:aa47bad7a513 (uuid:34a1baa2-c84f-04b8-b161-111b57b8949e) created (trackid=b31e7c57c92966cc9c7aee08ea5b21a4) by task D:a12c4bc9c14d
May 16 00:05:21 xenserver72-y26 xapi: [debug|xenserver72-y26|612 INET :::80|VM.update_allowed_operations R:aa47bad7a513|audit] VM.update_allowed_operations: VM = '5dc2b010-6a56-d7e3-ed57-3068c2a325f7 (cezita)'
May 16 00:05:21 xenserver72-y26 xapi: [ info|xenserver72-y26|209 INET :::80|dispatch:VM.update_allowed_operations D:355028e409ce|taskhelper] task VM.update_allowed_operations R:0e0cf39812b7 (uuid:3dc563fe-ef89-f5aa-c289-b1452becf538) created (trackid=602256a4a42b87b1c8bec8f08dfca3da) by task D:cef2ab9a8258
May 16 00:05:21 xenserver72-y26 xapi: [debug|xenserver72-y26|209 INET :::80|VM.update_allowed_operations R:0e0cf39812b7|audit] VM.update_allowed_operations: VM = '30d6c301-0de1-2a3a-c414-8e8149485ae6 (renaku)'
May 16 00:05:21 xenserver72-y26 xapi: [debug|xenserver72-y26|381 INET :::80||dummytaskhelper] task dispatch:VM.update_allowed_operations D:074db0affa62 created by task D:7db4b0abb031
May 16 00:05:21 xenserver72-y26 xapi: [ info|xenserver72-y26|381 INET :::80|dispatch:VM.update_allowed_operations D:074db0affa62|taskhelper] task VM.update_allowed_operations R:a218d7540c68 (uuid:e2c6a58a-25df-c1c3-5fe2-127057d82389) created (trackid=609bd475a5453a080869bbcb6ab89afb) by task D:7db4b0abb031
May 16 00:05:21 xenserver72-y26 xapi: [debug|xenserver72-y26|381 INET :::80|VM.update_allowed_operations R:a218d7540c68|audit] VM.update_allowed_operations: VM = '10742c8f-fbcd-ea53-a3ce-0b319fbf95a2 (goledu)'
May 16 00:05:22 xenserver72-y26 xapi: [debug|xenserver72-y26|9649 INET :::80||dummytaskhelper] task dispatch:VM.update_allowed_operations D:55ef89a13ced created by task D:10dfd8b7d5a2
May 16 00:05:22 xenserver72-y26 xapi: [debug|xenserver72-y26|396 INET :::80||dummytaskhelper] task dispatch:VM.update_allowed_operations D:802606ee0206 created by task D:acd32b25be4e
May 16 00:05:22 xenserver72-y26 xapi: [ info|xenserver72-y26|9649 INET :::80|dispatch:VM.update_allowed_operations D:55ef89a13ced|taskhelper] task VM.update_allowed_operations R:3cb8dbf275bd (uuid:c53debd3-39f4-6609-cc61-2c390b9b9d83) created (trackid=1c130c68275f501c5f3a19c3be946284) by task D:10dfd8b7d5a2
May 16 00:05:22 xenserver72-y26 xapi: [debug|xenserver72-y26|9649 INET :::80|VM.update_allowed_operations R:3cb8dbf275bd|audit] VM.update_allowed_operations: VM = 'f5b63cb0-3268-db6a-94a5-90a5f05c53b9 (qaroda)'
May 16 00:05:22 xenserver72-y26 xapi: [debug|xenserver72-y26|2544 INET :::80||dummytaskhelper] task dispatch:VM.update_allowed_operations D:7ff18d3207ee created by task D:2f7dbba98c1a
May 16 00:05:22 xenserver72-y26 xapi: [ info|xenserver72-y26|396 INET :::80|dispatch:VM.update_allowed_operations D:802606ee0206|taskhelper] task VM.update_allowed_operations R:4e2001a5c2a4 (uuid:d266f699-f56b-ef8f-2162-fc3e499a1ca6) created (trackid=a8965c1762f76f99d1d695160272735a) by task D:acd32b25be4e
May 16 00:05:22 xenserver72-y26 xapi: [debug|xenserver72-y26|396 INET :::80|VM.update_allowed_operations R:4e2001a5c2a4|audit] VM.update_allowed_operations: VM = '2e598abd-889d-9c52-99b3-906e5ab93194 (voqube)'
May 16 00:05:22 xenserver72-y26 xapi: [ info|xenserver72-y26|2544 INET :::80|dispatch:VM.update_allowed_operations D:7ff18d3207ee|taskhelper] task VM.update_allowed_operations R:071d98b3bbb0 (uuid:6b044ed6-c206-8dd8-85d9-75b8182d1530) created (trackid=f0dc5f31f8ab754af4468ed45d66fd4b) by task D:2f7dbba98c1a
May 16 00:05:22 xenserver72-y26 xapi: [debug|xenserver72-y26|2544 INET :::80|VM.update_allowed_operations R:071d98b3bbb0|audit] VM.update_allowed_operations: VM = 'e8406158-f0cb-ac3e-aa51-1e96f0a1122c (kuvugi)'
May 16 00:05:22 xenserver72-y26 xapi: [debug|xenserver72-y26|4690 INET :::80||dummytaskhelper] task dispatch:VM.update_allowed_operations D:367a7c198fc5 created by task D:c1bed27ae61b
May 16 00:05:22 xenserver72-y26 xapi: [debug|xenserver72-y26|467 INET :::80||dummytaskhelper] task dispatch:VM.update_allowed_operations D:280fb99a7996 created by task D:389e9f8c1d64
May 16 00:05:22 xenserver72-y26 xapi: [debug|xenserver72-y26|18123 INET :::80||dummytaskhelper] task dispatch:VM.update_allowed_operations D:c89eae219fa1 created by task D:a3cae53cbc15
May 16 00:05:22 xenserver72-y26 xapi: [ info|xenserver72-y26|4690 INET :::80|dispatch:VM.update_allowed_operations D:367a7c198fc5|taskhelper] task VM.update_allowed_operations R:2b4c21cf1352 (uuid:85616770-d674-f183-7772-df6494367c76) created (trackid=866f6094382b527b4acbc1ed4e89b675) by task D:c1bed27ae61b
May 16 00:05:22 xenserver72-y26 xapi: [debug|xenserver72-y26|4690 INET :::80|VM.update_allowed_operations R:2b4c21cf1352|audit] VM.update_allowed_operations: VM = '734fc5de-2001-9d90-febe-1ff420c7e399 (nefiza)'
May 16 00:05:23 xenserver72-y26 xapi: [debug|xenserver72-y26|18148 INET :::80||dummytaskhelper] task dispatch:VM.update_allowed_operations D:1d67f7e85a9b created by task D:c7278e38e9fb
May 16 00:05:23 xenserver72-y26 xapi: [ info|xenserver72-y26|467 INET :::80|dispatch:VM.update_allowed_operations D:280fb99a7996|taskhelper] task VM.update_allowed_operations R:b623a4e1add3 (uuid:296c4789-c664-756b-c1a9-55a39a2290ad) created (trackid=bbca458815b17a8c61473d52307f4d78) by task D:389e9f8c1d64
May 16 00:05:23 xenserver72-y26 xapi: [debug|xenserver72-y26|467 INET :::80|VM.update_allowed_operations R:b623a4e1add3|audit] VM.update_allowed_operations: VM = '7367f6a3-fd9e-7b47-ae64-811cb59a865e (pujara)'
May 16 00:05:23 xenserver72-y26 xapi: [ info|xenserver72-y26|18123 INET :::80|dispatch:VM.update_allowed_operations D:c89eae219fa1|taskhelper] task VM.update_allowed_operations R:a4ab60448bf2 (uuid:9b5740d8-3c8d-9f80-9fbb-6dc248415213) created (trackid=7cd8192bcb74e1ac5ceca77e276dfd6a) by task D:a3cae53cbc15
May 16 00:05:23 xenserver72-y26 xapi: [debug|xenserver72-y26|18123 INET :::80|VM.update_allowed_operations R:a4ab60448bf2|audit] VM.update_allowed_operations: VM = '16f6ea74-68bc-e668-065c-fdccd7d25086 (xuziba)'
May 16 00:05:23 xenserver72-y26 xapi: [debug|xenserver72-y26|1923 |org.xen.xapi.xenops.classic events D:fb915738e0e8|mscgen] xapi=>xapi [label="VM.update_allowed_operations"];
May 16 00:05:23 xenserver72-y26 xapi: [debug|xenserver72-y26|20070 UNIX /var/lib/xcp/xapi||dummytaskhelper] task dispatch:VM.update_allowed_operations D:1c25ebb67ac8 created by task D:fb915738e0e8
May 16 00:05:23 xenserver72-y26 xapi: [ info|xenserver72-y26|18148 INET :::80|dispatch:VM.update_allowed_operations D:1d67f7e85a9b|taskhelper] task VM.update_allowed_operations R:2c3d3b2d7eae (uuid:9c1ccdba-8b64-9864-bf10-384c9da26065) created (trackid=3dccde7ec1986f364e19c3f8f2985ddc) by task D:c7278e38e9fb
May 16 00:05:23 xenserver72-y26 xapi: [debug|xenserver72-y26|18148 INET :::80|VM.update_allowed_operations R:2c3d3b2d7eae|audit] VM.update_allowed_operations: VM = '35748c80-427a-22a9-389b-84e9b05db536 (bepoce)'
May 16 00:05:23 xenserver72-y26 xapi: [debug|xenserver72-y26|388 INET :::80||dummytaskhelper] task dispatch:VM.update_allowed_operations D:588f90ae9b05 created by task D:0be2824e3cdf
May 16 00:05:23 xenserver72-y26 xapi: [ info|xenserver72-y26|20070 UNIX /var/lib/xcp/xapi|dispatch:VM.update_allowed_operations D:1c25ebb67ac8|taskhelper] task VM.update_allowed_operations R:ba1f64898326 (uuid:3b70f16a-13ef-8230-f67f-905f1504f0b9) created (trackid=2b61ac5e642483918c0af5b371d91bb1) by task D:fb915738e0e8
May 16 00:05:23 xenserver72-y26 xapi: [debug|xenserver72-y26|20070 UNIX /var/lib/xcp/xapi|VM.update_allowed_operations R:ba1f64898326|audit] VM.update_allowed_operations: VM = 'f79326db-b1f9-9dc3-0f1f-cdde9fb7be75 (culexu)'
May 16 00:05:23 xenserver72-y26 xapi: [ info|xenserver72-y26|388 INET :::80|dispatch:VM.update_allowed_operations D:588f90ae9b05|taskhelper] task VM.update_allowed_operations R:92699c38518b (uuid:ed669f68-5306-3186-e94a-234e79ec6e39) created (trackid=8ca8bf618df2916b54f546cdf947a23f) by task D:0be2824e3cdf
May 16 00:05:23 xenserver72-y26 xapi: [debug|xenserver72-y26|388 INET :::80|VM.update_allowed_operations R:92699c38518b|audit] VM.update_allowed_operations: VM = '723bd70d-1212-3f5e-08c7-904673b0ba2f (teloco)'
May 16 00:05:23 xenserver72-y26 xapi: [debug|xenserver72-y26|2351 INET :::80||dummytaskhelper] task dispatch:VM.update_allowed_operations D:63f1b6ab70fe created by task D:a222093755f5
May 16 00:05:23 xenserver72-y26 xapi: [debug|xenserver72-y26|540 INET :::80||dummytaskhelper] task dispatch:VM.update_allowed_operations D:c095fa8c5d7d created by task D:0e8dcadbea95
May 16 00:05:24 xenserver72-y26 xapi: [ info|xenserver72-y26|2351 INET :::80|dispatch:VM.update_allowed_operations D:63f1b6ab70fe|taskhelper] task VM.update_allowed_operations R:6a97bb9d9464 (uuid:93389591-262e-fee9-1c05-1d081190db8e) created (trackid=ba169b0b2dc200fa958e4433e1f428d1) by task D:a222093755f5
May 16 00:05:24 xenserver72-y26 xapi: [debug|xenserver72-y26|2351 INET :::80|VM.update_allowed_operations R:6a97bb9d9464|audit] VM.update_allowed_operations: VM = '75fe3ff6-c2ca-8488-9016-ab31dace8595 (miqajo)'
May 16 00:05:24 xenserver72-y26 xapi: [ info|xenserver72-y26|540 INET :::80|dispatch:VM.update_allowed_operations D:c095fa8c5d7d|taskhelper] task VM.update_allowed_operations R:bd1c582cd8ad (uuid:da2e62c6-bbcc-c776-82b6-79fb0ffde8a0) created (trackid=61d9384ae3d08dd7b0476e7ede89fad5) by task D:0e8dcadbea95
May 16 00:05:24 xenserver72-y26 xapi: [debug|xenserver72-y26|540 INET :::80|VM.update_allowed_operations R:bd1c582cd8ad|audit] VM.update_allowed_operations: VM = 'b34de94d-041f-f0ee-98b3-d5142f064ee0 (gafuji)'
May 16 00:05:24 xenserver72-y26 xapi: [debug|xenserver72-y26|2866 INET :::80||dummytaskhelper] task dispatch:VM.update_allowed_operations D:e0ab5b8c5f92 created by task D:e46a633b6077
May 16 00:05:24 xenserver72-y26 xapi: [debug|xenserver72-y26|5787 INET :::80||dummytaskhelper] task dispatch:VM.update_allowed_operations D:01e7f29a7620 created by task D:1ec33d408c74
May 16 00:05:24 xenserver72-y26 xapi: [ info|xenserver72-y26|2866 INET :::80|dispatch:VM.update_allowed_operations D:e0ab5b8c5f92|taskhelper] task VM.update_allowed_operations R:f07b5d805474 (uuid:51af8037-acd0-1e2a-c6d2-eacadbbc0f0d) created (trackid=4b170bff998affe8e7741bdee4563a4e) by task D:e46a633b6077
May 16 00:05:24 xenserver72-y26 xapi: [debug|xenserver72-y26|2866 INET :::80|VM.update_allowed_operations R:f07b5d805474|audit] VM.update_allowed_operations: VM = '64415b4e-ba74-262b-53ee-9d210881e285 (wogufa)'
May 16 00:05:24 xenserver72-y26 xapi: [ info|xenserver72-y26|5787 INET :::80|dispatch:VM.update_allowed_operations D:01e7f29a7620|taskhelper] task VM.update_allowed_operations R:6e2c0479e91f (uuid:95d853be-8cea-3062-50f3-fd2a0ae40b38) created (trackid=7a460a85819b5351224b07bbfeb6c410) by task D:1ec33d408c74
May 16 00:05:24 xenserver72-y26 xapi: [debug|xenserver72-y26|5787 INET :::80|VM.update_allowed_operations R:6e2c0479e91f|audit] VM.update_allowed_operations: VM = '7e3908e2-2043-f82f-74ec-4b9365a19e6c (kiyoda)'
May 16 00:05:24 xenserver72-y26 xapi: [debug|xenserver72-y26|2577 INET :::80||dummytaskhelper] task dispatch:VM.update_allowed_operations D:6f6ee1f376a4 created by task D:0539083affc7
May 16 00:05:24 xenserver72-y26 xapi: [debug|xenserver72-y26|502 INET :::80||dummytaskhelper] task dispatch:VM.update_allowed_operations D:4f8a0d907504 created by task D:d94e2d534cdf
May 16 00:05:24 xenserver72-y26 xapi: [ info|xenserver72-y26|2577 INET :::80|dispatch:VM.update_allowed_operations D:6f6ee1f376a4|taskhelper] task VM.update_allowed_operations R:c804cccf405d (uuid:dc276ad6-63a0-54a4-e9ee-739ac33535f6) created (trackid=62cea79fc0fe96a4c9e47d3691b302bf) by task D:0539083affc7
May 16 00:05:24 xenserver72-y26 xapi: [debug|xenserver72-y26|2577 INET :::80|VM.update_allowed_operations R:c804cccf405d|audit] VM.update_allowed_operations: VM = 'c0db48dc-7b66-4fa7-5a2c-bb216c885bb4 (qodeyi)'
May 16 00:05:24 xenserver72-y26 xapi: [ info|xenserver72-y26|502 INET :::80|dispatch:VM.update_allowed_operations D:4f8a0d907504|taskhelper] task VM.update_allowed_operations R:2a20c87e2665 (uuid:4b6d8b06-3138-1c13-d169-b11618c47b3c) created (trackid=5a5fe81bc41b410dff87d398eb220589) by task D:d94e2d534cdf
May 16 00:05:24 xenserver72-y26 xapi: [debug|xenserver72-y26|502 INET :::80|VM.update_allowed_operations R:2a20c87e2665|audit] VM.update_allowed_operations: VM = 'c483fcfd-d320-0f9c-4a47-5da86ff1fb2f (yimiqa)'
May 16 00:05:25 xenserver72-y26 xapi: [debug|xenserver72-y26|416 INET :::80||dummytaskhelper] task dispatch:VM.update_allowed_operations D:250cd9214f89 created by task D:eae5a2988310 

 

I'm sure this is the reason, why xapi is overloaded, but why?

Any hint is welcome, as we currently have serious problems!

 

Many thanks in advance!

 

 

 

UPDATE: I forgot to mention, that I already designated a new poolmaster. Same problem.

 

Edited May 15, 2020 by hostnet1994
added update comment

[Tobias Kreidl]

There are several issues here. First off, until Citrix Hypervisor 8.X, the official maximum number of hosts in a pool was limited to 16 (see: https://docs.citrix.com/en-us/legacy-archive/downloads/xs-config-limits-7-2.pdf ), hence I would first of all strongly suggest upgrading to CH 8.1 where the maximum supported liit is 64. You may also need to increase the size of dom0 on your hosts since this is a huge pool with a lot of hosts as well as VMs. Running top and xentop will give you an indication as to whether your CPU and/or memory resources are not large enough. I'd guess you'd want easily 16 GB of memory allocated to dom0. You can also run the utility iostat from the CLI and look at the wait states and other I.O parameters to see if queue saturation is taking place or read and writes are taking too long.

 

Another consideration would be to split your pool into pieces, but that may be less esily done depending on your storage resources.

 

-=Tobias

[Jimmy Koerting]

Hi Tobias,

 

many thanks for your input!

So first of all: the tons (more than a few per second) of VM.update_allowed_operations task are not a sign of something else? Corrupted Metadatas, corrupted database, or what ever?

I have other pools here - all with 7,2 and even 6.5, which have nothing of these tasks - at least for days (and yes - they are much smaller ;) )

 

Upgrading to 8.1 is not an option atm, but splitting would be no problem. First of all, I will raise the Dom0.

But could you confirm that the very high number of the above tasks is not a hint for another problem in my pool?

 

Many thanks again!!!

 

Jimmy

[Tobias Kreidl]

Jimmy,

Have you installed all the released hotfixes? Is there adequate disks space on your host OS installs for updates, logs, etc.? The new partition layout is so much better in that regard. Your error message seems similar to this thread - take a look:

https://discussions.citrix.com/topic/391062-xenserver-71-update-upload-patch-hangs-in-pending-state/

 

-=Tobias

[Alan Lantz]

I agree with all the above. Hotfixes, 16 hosts per pool, more Dom0 memory, new partition layout. Going to the latest 8.x release of 

course if possible.  All standard, but great advice. 

 

--Alan--

 

[Jimmy Koerting]

Tobias,

 

yes, all hotfixes installed and plenty of disk space.

For now, I will try to move a bunch of VMs and hostserver to a new pool. This will take some days, as everything is quite slow on the pool atm.

 

But, I really wonder why there are so many tasks about updating the allowed operations for each VMs (at least that's what the log messages look like). My simple guess was, that there is something wrong with whatever database xen is using for holding such data. Ok, ~900 VMs are many of that informations, but when I see the numbers of tasks, all of the 900 VMs informations should be done within some minutes. So, what is updated there for hours and days? As there are not that much changes, someone must forget something somewhere :)

 

I'll be back with an update, after moving some hostservers. At least the load of xapi should go down a bit, if the reason is to much host in the pool, right?

 

Many thanks again! Your help is really appreciated as always!

[Austin Marsh]

Did you ever find a reason for this? I have the same issue, similar setup with large pools and VM counts.