Cloud Connector updates during production hours

[Nick Casagrande1709152718]

Anyone else here completely miffed by the fact that Citrix pushes connector updates during the day which completely kicks your users to the curb?   WTF???   #epicfail

[CarlStalhood]

Do you have two Connectors per Resource Location? The Cloud Connector servers are dedicated with no other software?

[Nick Casagrande1709152718]

yes sir, virgin vm's.  both of mine were updated at 1045AM within 10 minutes of each and caused an outage and angry user base.  support says there is no way to delay or stop this as the team resides in EST and thats when they push updates.  if they were truly HA, this wouldn't happen.  

[John Cattaneo]

Nick, can you confirm if you and your users are leveraging the NetScaler Gateway service via the cloud connectors? 

[Nick Casagrande1709152718]

So here's the update.

 

I walked away from CWC.  Buggy, poor architecture, obviously lacking change control, connectors lack HA.  I was using the NGaaS, only VDA;s were in my control.  After 1000+ms of latency and the connector update pushed around 1045AM causing a complete outage for my users I was done.  Good in theory, but it's not very polished.  The support team in ATL is excellent, however much is out of their control.  Connector update controls need to be implemented immediately as it is the proxy for everything.  A user down the hall from the VDA has their ICA session bounced around the east coast (I counted about 11 hops at one point) resulting in horrific latency and with the connector reboot, I just couldn't continue down that road anymore.  Perhaps in another year it'll be ready for prime time.  

 

 

[Mark Hodges]

Connector update controls need to be implemented immediately as it is the proxy for everything

 

Connector updates should never happen unless I push a button (and have the ability to put a connector in maintenance mode to drain it)

 

[Nick Casagrande1709152718]

I don't mind them controlling it, I actually prefer it so it'll be hands off for me, however, it needs to be true HA without a hiccup like my Sonicwall NSA's are in an HA pair.  

[John Cattaneo]

Sorry to hear of your experience.

 

The reason why I asked specifically about NGaaS is because that is one of the only services that currently has 'state' on the connector (i.e. proxies HDX network traffic). 

 

The team is working on a solution to remove that state from the connector so the traffic goes straight to the GWaaS cloud POPs (eliminating the connector as a proxy for the HDX traffic). This should address any intermittent connection interruptions during connector upgrade.

 

No more VDA -> connector -> GWaaS . Instead VDA -> GWaaS. Read on it here: https://virtualfeller.com/2018/07/17/cloud-connector-vda-to-gateway-service/?es_p=7167124

 

[Nick Casagrande1709152718]

Thanks for the reference John and yes I was aware of this, however it doesn't address two immediate needs of mine

 

1. I am on LTSR CU2, specifically my vda's are still W7 of which 7.15 is the last VDA I'll ever see for them.  W10 VDA's are in process, but I still have 2 vendors that are stuck in the mud on their support statements.

2. Bypassing the connector is a good strategy, however that wasn't causing my latency.  I still have a ton of hops my users were going thru who were sitting down the hall (virtually speaking) from the VDA.  I realize that if I had my NSVPX still in the mix, this wouldn't have happened (the complete opposite of what sales told me btw).  I'm not looking for partial cloud solution honestly, I want to dump it all just like when I moved to O365 from Exchange 2013.  

3. For those that go "full CWC" a nice roadmap feature would be that the ICA ticket issued to the endpoint would have the endpoint traffic flow directly to the VDA (think users who are on the same LAN as the data center with onprem VDA's) and not bounce that traffic over the Internet.  If you're in an airport via public wifi and you're connecting, well then you get what you get, no issues there and typical Internet latency will apply there.

 

As previously mentioned a "maintenance mode on connector" would be kinda cool, but honestly I wouldn't want to bother with that, just do HA properly or have them bled down by their own intelligence to process an upgrade without user disruption.  

[Jeremy Stroebel]

@Nick Casagrande1709152718 You nailed my #1 complaint with your 3rd point.  Its my biggest gripe on Virtual Apps & Desktop Service.

 

We run Gateway on Prem for the sole reason that users on LAN shouldn't have to have all their traffic go through GWaaS jumping around the east coast or wherever they connect when the VDA is <1 ms away on LAN.  We need beacon functionality like we had on-prem that would then direct users straight to VDA if on LAN.

 

Right now the other way to do this is to run Storefront on Prem and do it that way (there are instructions in the Documentation for doing a split setup with LAN users pointing to local Storefront and remote to GWaaS & Cloud Workspace), but gateway seemed to be easier to manage, setup, and update along with allowing us to use Workspace.

[Nick Casagrande1709152718]

Thanks for backing me up.  Huge problem for Citrix if you're trying to move users to Azure for this with the obscene latency.  I see it this way.  I got 3 vm's onprem, NS, SF & DDC.  What's the point of only giving up the DDC to CWC?  I want it all gone.  I don't want to manage infrastructure anymore or endless upgrades.  I'd put my ESX cluster on AWS in a heartbeat if it wasn't $125K/year or so.  

[LANCE ALLEN]

I know this is an "older" post but the connector updates are still a huge problem.

We use a product that doesn't allow unauthorized applications to run, and we just had a 4 hour outage trying to decipher what happened.

 

The connectors tried to auto-update, but were unsigned by citrix so they were not allowed to execute/install properly.  This happened at you guessed it , 10:45am during production hours, and HA doesn't help as it hits and breaks all the connector servers at the same time.

 

Also this isn't just a xenapp affecting issue, as we are trying to leverage it for our Endpoint management. Which requires these connectors be live in order for anyone on mobile to function....fun stuff when 3000 users can't sign into their mobile devices.

[Nick Casagrande1709152718]

7 minutes ago, LANCE ALLEN said:

I know this is an "older" post but the connector updates are still a huge problem.

We use a product that doesn't allow unauthorized applications to run, and we just had a 4 hour outage trying to decipher what happened.

 

The connectors tried to auto-update, but were unsigned by citrix so they were not allowed to execute/install properly.  This happened at you guessed it , 10:45am during production hours, and HA doesn't help as it hits and breaks all the connector servers at the same time.

 

Also this isn't just a xenapp affecting issue, as we are trying to leverage it for our Endpoint management. Which requires these connectors be live in order for anyone on mobile to function....fun stuff when 3000 users can't sign into their mobile devices.

 

I live in FTL about 20 minutes from Citrix HQ.  Shall I pick you up at the airport and we go to there and express our grief in person?  I was on cloud for a week and it was the worst week of my life with Citrix.  Many people I spoke with in support know of this issue and why Citrix isn't throwing all hands on deck to fix these connectors is beyond me.  They need to be HA just like firewall devices so when one reboots no one is affected for even a second.  Until that happens and the latency issue is resolved, cloud is a no-go for me.  

[John Cattaneo]

Just wanted to let you know that your commentary does not fall on deaf ears.

 

The following solution is an alternative approach to having your Gateway Service connections tunneled through the connector. Instead gateway connections are established to/from the VDA itself. Any update to the connectors will not impact active sessions (or cause them to be re-established/reconnect across other connectors): https://www.citrix.com/blogs/2019/03/25/ica-and-the-gateway-service-have-a-new-rendezvous/?utm_source=47296&es_p=8921829

 

Also, don't be surprised if there's a solution for connector update scheduling in the very near future.

[John Cattaneo]

On 8/22/2018 at 4:27 PM, Nick Casagrande1709152718 said:

Thanks for the reference John and yes I was aware of this, however it doesn't address two immediate needs of mine

....

3. For those that go "full CWC" a nice roadmap feature would be that the ICA ticket issued to the endpoint would have the endpoint traffic flow directly to the VDA (think users who are on the same LAN as the data center with onprem VDA's) and not bounce that traffic over the Internet.  If you're in an airport via public wifi and you're connecting, well then you get what you get, no issues there and typical Internet latency will apply there.

....

 

 

For 3. Check out the new preview: https://docs.citrix.com/en-us/citrix-cloud/workspace-network-location.html 

[Nick Casagrande1709152718]

4 minutes ago, John Cattaneo said:

 

For 3. Check out the new preview: https://docs.citrix.com/en-us/citrix-cloud/workspace-network-location.html 

 

ah very cool, gotta check this out

[Jason Kornhaus]

Just to add to the discussion. We are also a Citrix Cloud hybrid customer and we also get brief connection interruptions during the Cloud Connector updates at 10:40am of all times. I don't see how this can be best practice from Citrix. Maybe if they get enough complaints they will offer a customer approved maintenance window.