Citrix XML Services Failures - ID 0, 4003, and 28

[TheMissouriSpartan]

We host an on-prem environment running CVAD 7 1906.2. 

The DDC/Storefront server (single server running both, and a single NetScaler, there is no HA, this is a small environment) had issues today. Multiple users reported being kicked from their sessions, and when looking at the DDC/Storefront server we noticed two things: 

1. Storefront web was throwing back a "503 service not available" error
2. In the Event Viewer logs for Citrix Delivery Services, we saw multiple instances of the following three entries: 

[ERROR] - The Citrix servers reported an unspecified error from the XML Service at address https://<DDCserver>.local/scripts/wpnbr.dll [NFuseProtocol.TRequestAddress].  ID 0

[ERROR] - All the Citrix XML Services configured for farm <farm name> failed to respond to this XML Service transaction. ID 4003

[WARNING] - Failed to launch the resource '<resourceName>' using the Citrix XML Service at address 'https://<DDCServer>.local/scripts/wpnbr.dll'. The XML service returned error: 'unspecified'. ID 28

Rebooting the DDC/SF server, resolved the issue. But now we're trying to determine WHY the Citrix XML services were failing and causing the broker, Storefront, and everything not to talk to each other. 

So far, NetScaler logs don't indicate any NetScaler-side issues, other than also being unable to talk to the DDC/SF server. Citrix XML Service doesn't appear in the Services list in the services console in Windows, so I may be confused as to what it means by "Citrix XML Services". 

The logs above are what we found on the DDC/SF server after we rebooted it and got services restored. 

Has anyone encountered this and what was your solution? 

[Robert Peck]

Citrix XML Service is now named the Citrix Broker Service. Hopefully that piece of info can get you somewhere.

From those log errors it feels like the Citrix Broker Service  on your controller stopped. Obviously, that could be for multiple reasons.

I am sure you realize that that is a very old version of CVAD.  Sounds like you are running a small shop on a tight budget. Best of luck. 

[Michael Adam]

Hello @TheMissouriSpartan

 

i think there are different problems in your environment! The storefront eventlog messages can cause a problem while starting a new application/desktop. But this is NOT the reason why existing sessions are kicked - this is a other reason. For last one, you will see eventlog messages on the affected VDA at the timestamp the user was kicked! Here you can see what was the reason for that. 

Maybe you have network problems it the storefront and the VDAs see problems at the same time. Do you check the connection quality, firewalls, etc? For the storefront there is a article which is valid for using a CloudConnector which having connection problems (https://support.citrix.com/s/article/CTX233424-users-unable-to-launch-the-published-desktops-and-applications-xml-errors-on-storefront-servers?language=en_US). Cause of that i think its a good idea to check this and take a look to the network.

And yes: CVAD 1906.2 is very old and out of support! Its important to update your environment to a supported version. 

Best regards,
Michael 

[Stefan Kroell]

On 10/2/2024 at 6:12 PM, TheMissouriSpartan said:

We host an on-prem environment running CVAD 7 1906.2. 

The DDC/Storefront server (single server running both, and a single NetScaler, there is no HA, this is a small environment) had issues today. Multiple users reported being kicked from their sessions, and when looking at the DDC/Storefront server we noticed two things: 

1. Storefront web was throwing back a "503 service not available" error
2. In the Event Viewer logs for Citrix Delivery Services, we saw multiple instances of the following three entries: 

[ERROR] - The Citrix servers reported an unspecified error from the XML Service at address https://<DDCserver>.local/scripts/wpnbr.dll [NFuseProtocol.TRequestAddress].  ID 0

[ERROR] - All the Citrix XML Services configured for farm <farm name> failed to respond to this XML Service transaction. ID 4003

[WARNING] - Failed to launch the resource '<resourceName>' using the Citrix XML Service at address 'https://<DDCServer>.local/scripts/wpnbr.dll'. The XML service returned error: 'unspecified'. ID 28

Rebooting the DDC/SF server, resolved the issue. But now we're trying to determine WHY the Citrix XML services were failing and causing the broker, Storefront, and everything not to talk to each other. 

So far, NetScaler logs don't indicate any NetScaler-side issues, other than also being unable to talk to the DDC/SF server. Citrix XML Service doesn't appear in the Services list in the services console in Windows, so I may be confused as to what it means by "Citrix XML Services". 

The logs above are what we found on the DDC/SF server after we rebooted it and got services restored. 

Has anyone encountered this and what was your solution? 

Hi,

did you check this: Users unable to launch the published desktops and applications - XML errors on StoreFront servers (citrix.com)

I changed the Bypass Duration to "0" and changed the value to "loadbalancing DDCs". (In Manage Delivery Conroller in Storefront)

Maybe this will help you

Best regards

Stefan

[TheMissouriSpartan]

On 10/3/2024 at 3:06 PM, Robert Peck said:

Citrix XML Service is now named the Citrix Broker Service. Hopefully that piece of info can get you somewhere.

From those log errors it feels like the Citrix Broker Service  on your controller stopped. Obviously, that could be for multiple reasons.

I am sure you realize that that is a very old version of CVAD.  Sounds like you are running a small shop on a tight budget. Best of luck. 

@Robert Peck, thanks for the reply. Sorry for the delay, it's been a crazy few weeks.

That's what we're now starting to suspect.

This happened on another environment running CVAD 1912 LTSR CU9. Same Event log errors on the DDCs/Storefront combo servers. We've checked everything from NetScaler (which shows "STA Authority Unreachable" errors all over that timeframe in its logs). It just so happens the STA's....are the DDC/Storefront servers. So, the NS couldn't talk to the DDC/SF servers, which couldn't find any VDAs to connect users to when they tried to connect again.  So, something definitely caused everything to grind to a halt and unable to communicate for roughly 6-7 minutes.

Which I suppose could be explained if the broker service failed. However....users getting kicked from their sessions is what really concerns me. We route external users through the NS, and internal just go directly through the edge gateway/load balancer into Storefront for sessions. It's possible a network event occurred, but so far event logs in our network monitoring appliance (CISCO Meraki) doesn't display any indication of failure during that time period anywhere on the site networks. ISP indicates no issues during that time frame either. 

[Stefan Kroell]

Hi,

i still get the same issues -> DDCs Events 28,0,4003.

We have 105 Users and get every morning ca. 10 connection errors from different users.

We use 2x DDCs, 2x NS, CVAD 2402 LTSR CU1.

We checked the connection via Wireshark -> On the bottom right the workspace app it stucks "get ica file" and in the wireshark log we can see that the request changes every second to another DDC1 and DDC2.

For me it seems that the DDCs cant reach a VDI at this time (they are non persistent).

any Ideas?

thank you

best regards

Stefan

Edited November 4 by Stefan Kroell