Kari Ruissalo

We ended up in creating a separate store for the thin clients and using the Chromium browser to handle the authentication & StoreFront view (our default SF Store forces HTML5 due to customer requirements, thus the separate store). As the majority of endpoints are Windows-based, I figured to identify the Igel OS / Chromium based on the User-Agent header it's sending (User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36) and have a higher prio Session Policy that picks the string X11; Linux x86_64 from the User-Agent. This will then redirect the endpoints to a Store enforcing the full WSApp client rather than HTML5. Going to keep my eye on the modern authentication support for CWAL though.

I really need to study more... found this https://docs.citrix.com/en-us/citrix-workspace-app-for-linux/authentication#support-for-authentication-using-fido2-when-connecting-to-on-premises-stores Support stated that FIDO2 should work but "it will not have Modern authentication available", whatever does that mean 🤯

@Julian Jakob, I actually walked through the same though pattern on the CA policies, but it wasn't about that. Hmmh... Changing this knob in Windows CWA enables the FIDO2 capability for on-prem environments -> https://docs.citrix.com/en-us/citrix-workspace-app-for-windows/authentication.html#using-gpo (tested and verified). As far as it goes for the thin clients (Igel OS running CWA for Linux), here's the statement from support:

One of our customers is about to deploy FIDO2/Passwordless in to their environment (with Igel OS thinclients), but we're facing a peculiar issue with the full client. If we configure our Workspace App to a cloud URL and pick the "Sign-in options" We can see the "Face, fingerprint, PIN or security key" -option: But if we do the same with an on-prem NetScaler Gateway configured as an SP for Entra ID (we've tried both OIDC and SAML),we only see the GitHub -option We can see that when we configure Citrix Cloud to use Entra ID authentication, an Enterprise Application is added to our Entra ID tenant, but the Multi-tenant App Registration is held by Citrix (?) and therefore we're unable to see inside that. This why we actually tried OIDC the first place, because we suspected it might have something to do with this. And I know, this is Windows client, but I'd have to assume we should get it working first on this before moving in to Igel OS (we're seeing the same issue there also). Interestingly this issue doesn't appear if we access the on-prem Gateway using web browser so it's on some level CWA related. --- Related version info: Workspace App: 24.2.0.172(2402), Win 11 23H2 Citrix Cloud: Cloud NetScaler Gateway: 13.1-51.15 We also have a support case opened, but so far we haven't gotten anywhere.

I can confirm that this was never the issue. It was the timeout. Seems that in the Global session settings the GUI shows the timeout value as "30" and the CLI also reflects this: > sh vpn param | grep -i "session timeout" Session timeout: 30 minutes Unrelated, but also had an interesting discussion regarding the mechanism how SAC detects if it's in the internal network. Using the "DNS Suffix" for this is quite clumsy if you ask me and I couldn't find it in the docs, but SAC only performs a lookup for the domain and if the result is RFC private IP, then SAC "knows" it's in the internal network. Also, what happens if you have listed several DNS Suffixes? I would really like to see this as a separate configuration item, like "Always-on VPN Internal beacon address" and some detailed instructions how it works. I can deal with this easily now, but I think it might be a bit confusing for many others...

Starts to look like we need to define the session timeout, because once we added that the environment looks like it started working set vpn sessionaction sess_prof_{gw-fqdn}-aon-split-vpn -sessTimeout 1440 Is this mentioned somewhere in the docs, because I wasn't able to find a mention that this is required. The timeout article is pretty thorough, but it advices to other direction (https://docs.netscaler.com/en-us/netscaler-gateway/current-release/vpn-user-config/configure-plugin-connections/configure-time-out-settings.html)

We have tried to rule out the DNS resolution issues by changing the DNS LB vServer from non-addressable to an addressable one and even added the Gateway FQDN to the client hosts-file. Getting pretty confident this is not a DNS/name resolution problem but something else. How does the split tunnel behave if we have an intranet application for 10.0.0.0/8 subnet and then hand out IP Pool IPs from a subnet inside that range (for example 10.0.123.0/24)? Shouldn't be an issue? In the SAC log we're seeing that the client is unable to add the route, which could well be the reason (you can see the 172.20.10.1 in this specific log, but this is not the issue for most of the clients, we're aware of the spoofed IP setting described CTX491477): 2024-04-19 12:08:30.288 | 04092 | VERBOSE | D | VPN POST-CONFIG | ---- pRoutes --- (13,4) {Gateway-IP}/255.255.255.255 -> 172.20.10.1 [32, ffffffff, ffffffff, ffffffff, ffffffff ] (c) proto 3 dwForwardPolicy 0, ----end--- | 2024-04-19 12:08:30.288 | 04092 | DEBUG | D | VPN POST-CONFIG | Trying route addition again for {Gateway-IP}/255.255.255.255 -> 172.20.10.1 metric 50 | 2024-04-19 12:08:30.288 | 04092 | EVENT | S | VPN POST-CONFIG | Adding IPV4 route entries | 2024-04-19 12:08:30.290 | 04092 | VERBOSE | D | VPN POST-CONFIG | Added (13) {Gateway-IP}/255.255.255.255 -> 172.20.10.1 metric 50 | 2024-04-19 12:08:30.291 | 04092 | EVENT | S | VPN POST-CONFIG | Adding IPV4 route entries | 2024-04-19 12:08:30.293 | 04092 | EVENT | S | VPN POST-CONFIG | IPV4 helper API : No error | 2024-04-19 12:08:30.293 | 04092 | ERROR | S | VPN POST-CONFIG | Interface (0) failed to add route [0.0.0.0/0.0.0.0 -> 0.0.0.0 metric 0], while trying to set. Error (2) Unknown | 2024-04-19 12:08:30.293 | 04092 | VERBOSE | D | VPN POST-CONFIG | ---- pRoutes --- (0,0) 0.0.0.0/0.0.0.0 -> 0.0.0.0 [0, 0, 0, 0, 0 ] (0) proto 3 dwForwardPolicy 0, ----end--- | 2024-04-19 12:08:30.293 | 04092 | DEBUG | D | VPN POST-CONFIG | Trying route addition again for 0.0.0.0/0.0.0.0 -> 0.0.0.0 metric 0 | 2024-04-19 12:08:30.293 | 04092 | EVENT | S | VPN POST-CONFIG | Adding IPV4 route entries | 2024-04-19 12:08:30.296 | 04092 | EVENT | S | VPN POST-CONFIG | IPV4 helper API : No error | 2024-04-19 12:08:30.296 | 04092 | ERROR | S | VPN POST-CONFIG | Interface (0) failed to add route [0.0.0.0/0.0.0.0 -> 0.0.0.0 metric 0], while trying to create. Error (2) Unknown | 2024-04-19 12:08:30.296 | 04092 | VERBOSE | D | VPN POST-CONFIG | ---- pRoutes --- (0,0) 0.0.0.0/0.0.0.0 -> 0.0.0.0 [0, 0, 0, 0, 0 ] (0) proto 3 dwForwardPolicy 0, ----end--- | 2024-04-19 12:08:30.296 | 04092 | EVENT | S | VPN POST-CONFIG | Setting IPv4 routes for new configuration | 2024-04-19 12:08:30.297 | 04092 | EVENT | S | VPN POST-CONFIG | Adding IPV4 route entries | 2024-04-19 12:08:30.299 | 04092 | EVENT | S | VPN POST-CONFIG | IPV4 helper API : No error | 2024-04-19 12:08:30.300 | 04092 | ERROR | S | VPN POST-CONFIG | Interface (0) failed to add route [0.0.0.0/0.0.0.0 -> 0.0.0.0 metric 307], while trying to set. Error (2) Unknown | 2024-04-19 12:08:30.300 | 04092 | VERBOSE | D | VPN POST-CONFIG | ---- pRoutes --- (0,0) 0.0.0.0/0.0.0.0 -> 0.0.0.0 [133, 0, 0, 0, 0 ] (0) proto 3 dwForwardPolicy 0, ----end--- | 2024-04-19 12:08:30.300 | 04092 | DEBUG | D | VPN POST-CONFIG | Trying route addition again for 0.0.0.0/0.0.0.0 -> 0.0.0.0 metric 307 | 2024-04-19 12:08:30.300 | 04092 | EVENT | S | VPN POST-CONFIG | Adding IPV4 route entries | 2024-04-19 12:08:30.302 | 04092 | EVENT | S | VPN POST-CONFIG | IPV4 helper API : No error | 2024-04-19 12:08:30.302 | 04092 | ERROR | S | VPN POST-CONFIG | Interface (0) failed to add route [0.0.0.0/0.0.0.0 -> 0.0.0.0 metric 307], while trying to create. Error (2) Unknown |

I observed that if we try to use loginSchema XMLs from an older build (13.0) and just copy them in to new release (13.1) they don't show up properly. So, I just ended up creating new customizations based on the original files in 13.1 and then everything worked. The screenshots look familiar.

We had successfully completed a test and pilot for the Always-On VPN Before Windows logon, but now that the deployment has extended, we started facing major issues with end users. The biggest issue seems to be the error "Your machine is connected to the company network. Citrix Secure Access will logout" when the tunnel should switch from machine to user. We have configured only one DNS Suffix for the Gateway and it's not resolvable from public DNS and didn't see this issue during test/pilot. Weirdly enough we have suspicions this might have something to do with the environment load (there's not much, under 50 sessions), because we're also seeing successful connections from time to time. One noteworthy thing we've observed from the logs collected from SAC: Working 2024-04-18 07:00:54.983 | 15940 | DEBUG | D | LOGIN | Discovering store for domain Name - {Gateway-FQDN} | 2024-04-18 07:00:54.983 | 15940 | EVENT | S | LOGIN | Making GET request to https://{Gateway-FQDN}:443/Citrix/Store/discovery | 2024-04-18 07:00:54.983 | 05020 | EVENT | S | LOGIN | AlwaysOn is enabled. We will not use previous cookie and try re-login | 2024-04-18 07:00:54.983 | 05020 | EVENT | S | LOGIN | Ready to accept new API/Browser connections... | 2024-04-18 07:00:54.983 | 15940 | VERBOSE | D | LOGIN | [<GET /Citrix/Store/discovery HTTP/1.1>] | 2024-04-18 07:00:55.077 | 03392 | DEBUG | D | LOGIN | Hide main window | 2024-04-18 07:00:55.077 | 03392 | DEBUG | D | LOGIN | Calling Shell_NotifyIcon | 2024-04-18 07:00:55.077 | 03392 | DEBUG | D | LOGIN | Shell_NotifyIcon succeeded | 2024-04-18 07:00:56.179 | 15940 | DEBUG | S | LOGIN | GET /Citrix/Store/discovery returned: 302 | 2024-04-18 07:00:56.179 | 15940 | DEBUG | S | LOGIN | HTTP request return value is: -7 | 2024-04-18 07:00:56.179 | 15940 | EVENT | S | LOGIN | Plugin didn't get proper response for store discovery, assuming on-prem gateway. | 2024-04-18 07:00:56.179 | 15940 | EVENT | S | LOGIN | Continuing with on-prem gateway authentication | Non-working 2024-04-18 13:23:30.310 | 13864 | DEBUG | D | LOGIN | Discovering store for domain Name - {Gateway-FQDN} | 2024-04-18 13:23:30.310 | 13864 | EVENT | S | LOGIN | Making GET request to https://{Gateway-FQDN}:443/Citrix/Store/discovery | 2024-04-18 13:23:30.310 | 13864 | VERBOSE | D | LOGIN | [<GET /Citrix/Store/discovery HTTP/1.1>] | 2024-04-18 13:23:42.337 | 13864 | ERROR | S | LOGIN | Send HTTP Request -- Error (12007) Unknown | 2024-04-18 13:23:42.337 | 13864 | DEBUG | S | LOGIN | Resolving hostname {Gateway-FQDN} | 2024-04-18 13:23:50.366 | 13864 | DEBUG | S | LOGIN | Get address info successful | 2024-04-18 13:23:50.366 | 13864 | VERBOSE | D | LOGIN | Adapter 1 | 2024-04-18 13:23:50.366 | 13864 | VERBOSE | D | LOGIN | Flags: 0x0 | 2024-04-18 13:23:50.366 | 13864 | VERBOSE | D | LOGIN | Family: AF_INET (IPv4) | 2024-04-18 13:23:50.366 | 13864 | VERBOSE | D | LOGIN | IPv4 address {Gateway-IP} | 2024-04-18 13:23:50.366 | 13864 | DEBUG | S | LOGIN | Resolved address family(ipv4) = 2 ipaddr = {Gateway-IP} using hostname | 2024-04-18 13:23:50.366 | 13864 | DEBUG | S | LOGIN | HTTP request return value is: -4 | 2024-04-18 13:23:50.366 | 13864 | DEBUG | D | LOGIN | performGenLogin ret 3 | 2024-04-18 13:23:50.366 | 13864 | EVENT | S | LOGIN | Ignoring captive portal check as we are in service mode. Service would have already done this check | 2024-04-18 13:23:50.366 | 13864 | EVENT | S | LOGIN | Can't connect to given gateway https://{Gateway-FQDN} | 2024-04-18 13:23:50.366 | 13864 | EVENT | S | LOGIN | login Vpn Messge Couldn't connect to server 'https://{Gateway-FQDN}' | 2024-04-18 13:24:07.707 | 13652 | DEBUG | D | LOGIN | AppReceiverInit | 2024-04-18 13:24:07.911 | 13004 | DEBUG | D | LOGIN | OnAPIMessage: wParam 1002 lParam 1002 | 2024-04-18 13:24:08.525 | 13652 | EVENT | S | LOGIN | Successfully registered AG with Receiver | 2024-04-18 13:24:08.572 | 13652 | DEBUG | S | LOGIN | AG (Access Gateway) plugin taskbar icon hiding is disabled | 2024-04-18 13:24:08.572 | 13652 | DEBUG | D | LOGIN | AppReceiverInit done | 2024-04-18 13:24:17.933 | 13004 | DEBUG | D | LOGIN | OnAPIMessage: wParam 1002 lParam 1002 | ... and what we're seeing with the endpoint is that the network connection seems to be going up and down. To my understanding the error 12007 has something to do with name lookup, although in the log we're seeing the proper IP for the Gateway. Version information: Platform VPX1000Advanced (in HA) Deployment Topology 2-arm (no routing issues we're aware of) Intranet IP pools configured (machine pool on vServer level and user pools for AAA groups) Machine tunnel using device certificates User tunnel with LDAP (user/pass) + EPA (domain check) NetScaler 13.1 build 51.15 Client Windows 11 22H2, SAC 24.2.1.15 Windows 10, SAC 23.8.1.5 (And yes, we have a support case opened already, just pulling all strings)

Thank you for your answer, but I think the nFactor path would apply for a case where the NetScaler would act as OAuth SP, but in this case it's acting as an IdP. Is it possible to bind OAuth IdP policies in nFactor Policy Labels? I didn't even think of that 🤔

I have configured NetScaler as an OIDC IdP and am using the grant_type=password. To be able to get a token from /oauth/idp/token endpoint I need to include the following in my POST request body: grant_type, username, password, client_id, client_secret, resource I can see that I'm able to set the expression for my OAuth IDP Policy to limit from which IP the user can get the token (expr. CLIENT.IP.SRC.EQ(123.123.123.123)), but I'm not able to use expressions to allow only users in specific group (expr. AAA.USER.IS_MEMBEROF("mygroup")) or specific user name (expr. AAA.USER.NAME.EQ("myuser")) to explicitly allow tokens to be issued for only selected users. Is there a way to limit the IDP Policies on user-basis? Eventually I will have several OAuth IDP Policies bound to this AAA vServer and I can limit the users for the AAA vServer -wide using nFactor and LDAP policies, but not per OAuth IDP.

Got this working. NetScaler OAuth IdP requires to have the "resource" attribute in the request body. Everything else was there already. If anyone else is trying to find what grant types are supported on NetScaler OIDC, here goes: authorization_code password refresh_token client_credentials

Added "resource=something" in to the request body and now I'm getting an access token. Unfortunately I would need to be able to support grant_type=password and extract user info in to the token also. 🤔

Does anyone happen to have a sample of a HTTP POST how to get a Token from NetScaler OAuth IdP? With just basic authentication for starters... and some tips on how to configure the NetScaler AAA for it to work? Also, does NetScaler support grant_type=password? Should we use grant_type=client_credentials instead? I got a few steps forward by adding the Authorization header (basic), but if I'm trying HTTP POST to /oauth/idp/token with the following: I encoded the Basic auth header using UTF-8 (tried Unicode too didn't change anything). Getting a http 400 bad request response. ... I'm getting the following error in the ns.log: OAUTHIDP: Client credentials ERROR: clientid or secreit or resource is absent I'm looking in to replacing F5 Big-IP with NetScaler. With Big-IP they only needed to POST the following to get a Bearer token: I'm testing this with Postman. Any tips are appreciated!

If I run the Nitro Client on a recent NetScaler build (13.1 build 27.59) the feature in the topic still reports "operation under construction !!!!! Please try different operation". We're trying to find a way to automate the NetScaler certificate update process with Ansible, but we would first need to be able to create an RSA key on the box, then a CSR and post it to the Public CAs services. The only way we seem to be able to do this currently would be to run the openssl commands via SSH which is not optimal. Also, getting the RSA key based authentication seems to work quite randomly (we followed this instruction -> https://support.citrix.com/article/CTX109011/how-to-secure-ssh-access-to-the-netscaler-appliance-with-public-key-authentication). We also have ADM service so we could run some of the bits via that one. There was some buzz about the Venafi integration (https://www.citrix.com/blogs/2021/03/22/automate-ssl-certificate-lifecycle-with-citrix-adm-and-venafi-integration/) back in the days, but I haven't heard of that for a while, any experiences on this one?