Jump to content
Welcome to our new Citrix community!

Citrix gateway stuck with "loading your apps" screen


Recommended Posts

Worked fine in the local storefront mode

Created dns records for SF/DDC - pointed to new VIP of ADC respectively

Kept the same storefront base url instead change the dns to point to SF VIP of ADC 

Created cert for DDC FQDN and included LB address as SAN , installed in ADC as well

The virtual servers for SF and DDC both shows UP

Changed the DDC address to LB address in Storefront server and to https

Changed STA url to https://FQDN in both ADC and Storefront --> Citrix gateway 

Changed the portal theme to RfWebUI

 

Tested via gateway url and the webpage stuck with "loading your apps" screen.

If I rollback the change, i.e change the SF dns to it's own server IP and DDC address from LB to  DDC server IP, everything works fine 

 

Indicates there is something I might be missing here, any thoughts would be appreciated.

 

SF- Storefront, DDC - Delivery controller, LB - Load balancing 

Link to comment
Share on other sites

Areas of concern in your original summary (or I couldn't tell what you meant):

3 hours ago, Elavendhan Barathi1709162772 said:

Worked fine in the local storefront mode

Created dns records for SF/DDC - pointed to new VIP of ADC respectively

Kept the same storefront base url instead change the dns to point to SF VIP of ADC 

Created cert for DDC FQDN and included LB address as SAN , installed in ADC as well

The virtual servers for SF and DDC both shows UP

Changed the DDC address to LB address in Storefront server and to https

Changed STA url to https://FQDN in both ADC and Storefront --> Citrix gateway 

Changed the portal theme to RfWebU

 

 

Using some concrete names might have helped clarify this (even if just placeholders). Thoughts below of things to look out for and config requirements.  Config example later. Log notes at end.  I think either the issue is the dns cache and/or you load balanced STA's when you weren't supposed to; possibly gateways probe to storefront is failing due to dns cache issue.  All noted below.

 

Key Requirements noted as I read through your config notes:

1)  Gateway must point to storefront LB FQDN in its session policy.  

2)  STA's cannot be load balanced**.  Gateway must have explicit list of controller names as STA's list (so list 2 individually or more); StoreFront should continue to list the STA's individually as well (when load balancing the controllers, only the XML conversation is load balanced, not the STA lists or the VDA's list of controllers for VDA registrations). I couldn't tell if you updated the STA list or not in the description above. **Short answer. Gateway has to see the explicit controllers as individual sta's and cannot see a load balanced name for sta's.

3) The StoreFront base URL must match the FQDN used by users and gateway to access storefront or weird things happen. So if your users were going to https://<stf1 fqdn> the base URL does need to match https://<stf lb fqdn>. Propagate change to members of storefront server group. 

However, if the FQDN didn't change and you just changed the https://<original storefront fqdn> from OLD IP to NEW VIP, THEN remember the Citrix ADC/Gateway caches dns queries, did you flush the dns cache on the ADC to make it resolve the old name it already had to the NEW IP address.  (flush dns proxyrecords or Traffic Management > DNS > Records:: flush records (right pane).  This could result in gateway communication to storefront failing.  Confirm name to ip resolution via shell on ADC.  Also note, there is more to the gateway resolution of the storefront name and the STA addresses which I will note below.  (Couldn't tell which scenario you described, so I addressed either.)

 

Next, Basic example of config summary for following components: (thought this would be easier for you to compare as I wasn't always sure what you described)

- LB Vserver Storefront (storefront.demo.com) in place of stf1.demo.com and stf2.demo.com.  LB Method LeastConnections; PersistenceType:  SourceIP; PersistenceTimeout: 20 min (SSL:443)

-LB VServer Controllers (xml communication only) as (vdcxml.demo.com) in place of vdc1.demo.com and vdc2.demo.com. LB Method: LeastConnections; PersistenceType: none. (SSL:443)

- Gateway VPN as gateway.demo.com

 

Gateway VPN Config (on Gateway):

- Session Policy/Profile maps content to lb storefront as:  https://storefront.demo.com/Citrix/<StoreNameWeb> for web browsers.  Include settings in profile for domain (demo) to pass through, default authorization action (security tab), and enable passthrough to web applications (client experience), however, you MIGHT also need a TRaffic policy for sson (depending on  which version of firmware you are on and how authentication policies are configured.)

- Gateway vserver STA List:  must be individual controllers and not load balanced fqdn:  https://vdc1.demo.com, https://vdc2.demo.com

- See communication requirements below (at end of config summary)

 

StoreFront Store Config (for Gateway integration) on StoreFront

- Assumes you already have basic store created in internal mode to talk to controllers. 

- To use load balanced controllers, update Store's controller list with https://vdcxml.demo.com (using protocol port of load balancer and confirming storefront can resolve name to vip)

- Create Gateway definition for StoreFront:  Gateway FQDN:  https://gateway.demo.com.  Callback address can be blank for simple configs not getting into smartaccess filters.  Configure list of STA's with individual controllers. Make sure controllers on StoreFront are same controllers listed on Gateway:  https://vdc1.demo.com and https://vdc2.demo.com  (again adjust if ports are other than HTTPS:443)

- Enable Remote Authentication with Gateway on the Store

- Associate the defined Gateway with the storefront.

 

Reminder:  Controllers/ XDSite/CVADSite don't care about Gateway unless we get into gateway filters.  The VDA's must still use the individual controller FQDNs for vda registration and NOT the lb controller fqdn.  Only the XML conversation between StoreFront and Controllers is load balanced; not STA conversations and not VDA Registrations.

 

Additional Communication Requirements for Gateway to StoreFront:

-  Starting in 12.1, gateway actively verifies the Storefront address listed is valid through a background probe that will physically leave the appliance. Even if the Gateway vpn vserver and Storefront LB vserver are on the same appliance, this probe is NOT looking at the UP/DOWN state of the storefront lb vserver but relies on the gateway to resolve the stf fqdn and attempt to externally reach it. If there is no SNIP available (or firewall rules) that block the gateway to storefront probe, your LB vserver for StoreFront could be UP but the Gateway doesn't attempt the Storefront hand off.  Usually, if your load balancing a SNIP is already available, but this might be a factor.

- Must gateway/StoreFront troubleshooting can be done in the Gateway syslog (/var/log/ns.log) or in the SToreFront EventViewer (focus on the Citrix Delivery Service custom log under "Applications node" on the storefront servers in addition to the usual system/application event logs.).  (more log examples after this)

- HOWEVER, this one probe failure behavior logs in nslog (/var/log/newnslog) and you will see it as an internal probe to the <storefront fqdn> and will get details if this is succeeding or failing. 

 

Syslog on ADC:

- audit log and events related to features.  For gateway, useful to troubleshoot authentication issues, vpn authorization issues, sta failures.

shell

cd /var/log

tail -F /var/log/ns.log | grep -v CMD_EXEC

## displays all events for all features minus configuration changes and gateway related events span AAA/SSLVPN/TCP and other modules.

 

Nslog on ADC:

- debug and status/counters.

- must be viewed with nsconmsg commands which are case-sensitive big K is input file and little k is output file

shell

cd /var/nslog

# view current log

nsconmsg -K newnslog -d event

nsconmsg -K newnslog -d consmsg

 

I always forget which "output" this probe failure shows up in so check both events and console messages.

 

On StoreFront (If traffic is getting past gateway to the storefront),

- StoreFront event viewer the events under the Citrix Delivery Service event log is the best troubleshooter as it will show if the gateway credential handoff to storefront fails at storefront, if the storefront can't talk to controllers, if the storefront can't retrieve a ticket from an STA or which one it did use.  Then gateway events can be retrieved.

- If you can't tell at what phase of the communication flow you are in look at the PATH in the user's browser. 

- If you still see https://<gateway fqdn>/vpn/index.htm, /vpn/tmindex, /LogonPoint/,  or anything that isn't the storefront path, you are still on the GATEWAY and never got to storefront so start with gateway logs for troubleshooting.

- If you still see https://<gateway fqdn>/Citrix/<StoreNameWeb> then the gateway authentication/authorization complected and it attempted to get you to storefront. The issue might still be on the gateway side OR the storefront side, but you know the gateway tried to go to the next phase with Storefront.  I would check storefront servers for logged events first and then go review gateway as storefront's logs can be easier.

You may still need a network trace in addition to the logs.

 

Link to comment
Share on other sites

  • 3 weeks later...

Thanks for answering in detail.

 

I've addressed most of the concerns and or already have the configuration as suggested above. The outstanding issue is load balancing Storefront.

The Citrix gateway browser is still stuck at "loading your apps " screen.

 

Storefront base URL - changed to new URL pointing to load balancing virtual ip with respective DNS changes, flushed dns records

Created the certificate of CN as load balancing virtual ip and with SAN as server fqdn - Installed the same certificate in storefront and ADC.

Done IIS bindings to new certificate

ADC can ping and resolve storefront load balancing vip fqdn, 

ADC can ping and telnet storefront servers on 443 and vice versa

Gateway session policy is pointed to Storefront load balancing url and also the service account address.

 

The gateway url is stuck at https://<gateway fqdn>/logon/LogonPoint/index.html 

 

-No events in Storefront server

-Checked cd /var/log

tail -F /var/log/ns.log | grep -v CMD_EXEC  - No errors

-Checked cd /var/nslog - No errors 

 

This is driving me crazy, any feedback would be appreciated.

 

 

I believe my issue is around your comment below. Please advise how can i fix this with a bit more detail.

 

Even if the Gateway vpn vserver and Storefront LB vserver are on the same appliance, this probe is NOT looking at the UP/DOWN state of the storefront lb vserver but relies on the gateway to resolve the stf fqdn and attempt to externally reach it. If there is no SNIP available (or firewall rules) that block the gateway to storefront probe, your LB vserver for StoreFront could be UP but the Gateway doesn't attempt the Storefront hand off.  Usually, if your load balancing a SNIP is already available

 

 

Link to comment
Share on other sites

On 12/21/2021 at 5:25 PM, Elavendhan Barathi1709162772 said:

at https://<gateway fqdn>/logon/LogonPoint/index.html 

 

Gateway being stuck at this page means the issue is on the Gateway still and most likely related to the AAA authentication policy behavior. The gateway has not yet sent you to StoreFront, so no storefront events exist.

 

Check your ADC syslog and aaad.debug events (notes below). You are either having an issue in the authentication policies, aaa nfactor flow, gateway portal them, or authorization policies. Gateway is not getting to storefront.  If it was, you would see the path as /Citrix/<StoreNameWeb>.

 

Review your portal them, your aaa integration, and your nfactor policy configuration if applicable. Then confirm authorization decisions.

 

Syslog on ADC:

shell

cd /var/log

tail -f ns.log | grep -v CMD_EXEC

# shows all events related to AAA, TCP, and SSLVPN (everything but the audited commands)

 

AAA debug for authentication troubleshooting, view output during an authentication event to see if external authentication is or isn't failing.

shell

cd /tmp

cat aaad.debug

## view output while attempting a gateway sign in to review ldap/radius and any external authentication policy attempts. Not a log file; just a named pipe only outputs data during event.

 

 

 

 

 

 

Link to comment
Share on other sites

On 12/21/2021 at 5:25 PM, Elavendhan Barathi1709162772 said:

Even if the Gateway vpn vserver and Storefront LB vserver are on the same appliance, this probe is NOT looking at the UP/DOWN state of the storefront lb vserver but relies on the gateway to resolve the stf fqdn and attempt to externally reach it. If there is no SNIP available (or firewall rules) that block the gateway to storefront probe, your LB vserver for StoreFront could be UP but the Gateway doesn't attempt the Storefront hand off.  Usually, if your load balancing a SNIP is already available

For this particular issue, you must view your nslog output using the commands noted above. BUT, given your Logon page page path, you likely aren't getting this far.

If the NSLOG output confirms no snip/mip for probes available, then you will also need to solve this which may also be causing your authentication calls to fail if there is no ip available to egress the system to authentication or other destinations.

 

You will need to add a SNIP in the network that can reach authentication and storefront servers.  

Link to comment
Share on other sites

  • 3 weeks later...

Hi,

 

Appreciate your comments.

 

I'm at a stage where most of the issues are resolved but one final piece which I've explained below.

 

I do have the issue where when ADC fails over from A to B ( using private ip address method) the AWS route table is not getting changed automatically to ADC B SNIP interface. If I change it manually, it works fine.

 

Confirmed IAM permissions on both ADC instances and source/dest check on the interface.

 

But, upon inspecting cloud trail logs, ADC is making api call "Describe routes" but nothing like "delete routes" or "change routes" which I guess is expected from ADC after the "describe routes" call .

 

The failover of the ADC is no issue but the route table is not getting changed after it's failed over, no events in ha-daemon.log or ns.log.

 

Any response would be highly appreciated.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...