Kari Ruissalo Posted April 19 Share Posted April 19 We had successfully completed a test and pilot for the Always-On VPN Before Windows logon, but now that the deployment has extended, we started facing major issues with end users. The biggest issue seems to be the error "Your machine is connected to the company network. Citrix Secure Access will logout" when the tunnel should switch from machine to user. We have configured only one DNS Suffix for the Gateway and it's not resolvable from public DNS and didn't see this issue during test/pilot. Weirdly enough we have suspicions this might have something to do with the environment load (there's not much, under 50 sessions), because we're also seeing successful connections from time to time. One noteworthy thing we've observed from the logs collected from SAC: Working 2024-04-18 07:00:54.983 | 15940 | DEBUG | D | LOGIN | Discovering store for domain Name - {Gateway-FQDN} | 2024-04-18 07:00:54.983 | 15940 | EVENT | S | LOGIN | Making GET request to https://{Gateway-FQDN}:443/Citrix/Store/discovery | 2024-04-18 07:00:54.983 | 05020 | EVENT | S | LOGIN | AlwaysOn is enabled. We will not use previous cookie and try re-login | 2024-04-18 07:00:54.983 | 05020 | EVENT | S | LOGIN | Ready to accept new API/Browser connections... | 2024-04-18 07:00:54.983 | 15940 | VERBOSE | D | LOGIN | [<GET /Citrix/Store/discovery HTTP/1.1>] | 2024-04-18 07:00:55.077 | 03392 | DEBUG | D | LOGIN | Hide main window | 2024-04-18 07:00:55.077 | 03392 | DEBUG | D | LOGIN | Calling Shell_NotifyIcon | 2024-04-18 07:00:55.077 | 03392 | DEBUG | D | LOGIN | Shell_NotifyIcon succeeded | 2024-04-18 07:00:56.179 | 15940 | DEBUG | S | LOGIN | GET /Citrix/Store/discovery returned: 302 | 2024-04-18 07:00:56.179 | 15940 | DEBUG | S | LOGIN | HTTP request return value is: -7 | 2024-04-18 07:00:56.179 | 15940 | EVENT | S | LOGIN | Plugin didn't get proper response for store discovery, assuming on-prem gateway. | 2024-04-18 07:00:56.179 | 15940 | EVENT | S | LOGIN | Continuing with on-prem gateway authentication | Non-working 2024-04-18 13:23:30.310 | 13864 | DEBUG | D | LOGIN | Discovering store for domain Name - {Gateway-FQDN} | 2024-04-18 13:23:30.310 | 13864 | EVENT | S | LOGIN | Making GET request to https://{Gateway-FQDN}:443/Citrix/Store/discovery | 2024-04-18 13:23:30.310 | 13864 | VERBOSE | D | LOGIN | [<GET /Citrix/Store/discovery HTTP/1.1>] | 2024-04-18 13:23:42.337 | 13864 | ERROR | S | LOGIN | Send HTTP Request -- Error (12007) Unknown | 2024-04-18 13:23:42.337 | 13864 | DEBUG | S | LOGIN | Resolving hostname {Gateway-FQDN} | 2024-04-18 13:23:50.366 | 13864 | DEBUG | S | LOGIN | Get address info successful | 2024-04-18 13:23:50.366 | 13864 | VERBOSE | D | LOGIN | Adapter 1 | 2024-04-18 13:23:50.366 | 13864 | VERBOSE | D | LOGIN | Flags: 0x0 | 2024-04-18 13:23:50.366 | 13864 | VERBOSE | D | LOGIN | Family: AF_INET (IPv4) | 2024-04-18 13:23:50.366 | 13864 | VERBOSE | D | LOGIN | IPv4 address {Gateway-IP} | 2024-04-18 13:23:50.366 | 13864 | DEBUG | S | LOGIN | Resolved address family(ipv4) = 2 ipaddr = {Gateway-IP} using hostname | 2024-04-18 13:23:50.366 | 13864 | DEBUG | S | LOGIN | HTTP request return value is: -4 | 2024-04-18 13:23:50.366 | 13864 | DEBUG | D | LOGIN | performGenLogin ret 3 | 2024-04-18 13:23:50.366 | 13864 | EVENT | S | LOGIN | Ignoring captive portal check as we are in service mode. Service would have already done this check | 2024-04-18 13:23:50.366 | 13864 | EVENT | S | LOGIN | Can't connect to given gateway https://{Gateway-FQDN} | 2024-04-18 13:23:50.366 | 13864 | EVENT | S | LOGIN | login Vpn Messge Couldn't connect to server 'https://{Gateway-FQDN}' | 2024-04-18 13:24:07.707 | 13652 | DEBUG | D | LOGIN | AppReceiverInit | 2024-04-18 13:24:07.911 | 13004 | DEBUG | D | LOGIN | OnAPIMessage: wParam 1002 lParam 1002 | 2024-04-18 13:24:08.525 | 13652 | EVENT | S | LOGIN | Successfully registered AG with Receiver | 2024-04-18 13:24:08.572 | 13652 | DEBUG | S | LOGIN | AG (Access Gateway) plugin taskbar icon hiding is disabled | 2024-04-18 13:24:08.572 | 13652 | DEBUG | D | LOGIN | AppReceiverInit done | 2024-04-18 13:24:17.933 | 13004 | DEBUG | D | LOGIN | OnAPIMessage: wParam 1002 lParam 1002 | ... and what we're seeing with the endpoint is that the network connection seems to be going up and down. To my understanding the error 12007 has something to do with name lookup, although in the log we're seeing the proper IP for the Gateway. Version information: Platform VPX1000Advanced (in HA) Deployment Topology 2-arm (no routing issues we're aware of) Intranet IP pools configured (machine pool on vServer level and user pools for AAA groups) Machine tunnel using device certificates User tunnel with LDAP (user/pass) + EPA (domain check) NetScaler 13.1 build 51.15 Client Windows 11 22H2, SAC 24.2.1.15 Windows 10, SAC 23.8.1.5 (And yes, we have a support case opened already, just pulling all strings) Link to comment Share on other sites More sharing options...
Kari Ruissalo Posted April 19 Author Share Posted April 19 We have tried to rule out the DNS resolution issues by changing the DNS LB vServer from non-addressable to an addressable one and even added the Gateway FQDN to the client hosts-file. Getting pretty confident this is not a DNS/name resolution problem but something else. How does the split tunnel behave if we have an intranet application for 10.0.0.0/8 subnet and then hand out IP Pool IPs from a subnet inside that range (for example 10.0.123.0/24)? Shouldn't be an issue? In the SAC log we're seeing that the client is unable to add the route, which could well be the reason (you can see the 172.20.10.1 in this specific log, but this is not the issue for most of the clients, we're aware of the spoofed IP setting described CTX491477): 2024-04-19 12:08:30.288 | 04092 | VERBOSE | D | VPN POST-CONFIG | ---- pRoutes --- (13,4) {Gateway-IP}/255.255.255.255 -> 172.20.10.1 [32, ffffffff, ffffffff, ffffffff, ffffffff ] (c) proto 3 dwForwardPolicy 0, ----end--- | 2024-04-19 12:08:30.288 | 04092 | DEBUG | D | VPN POST-CONFIG | Trying route addition again for {Gateway-IP}/255.255.255.255 -> 172.20.10.1 metric 50 | 2024-04-19 12:08:30.288 | 04092 | EVENT | S | VPN POST-CONFIG | Adding IPV4 route entries | 2024-04-19 12:08:30.290 | 04092 | VERBOSE | D | VPN POST-CONFIG | Added (13) {Gateway-IP}/255.255.255.255 -> 172.20.10.1 metric 50 | 2024-04-19 12:08:30.291 | 04092 | EVENT | S | VPN POST-CONFIG | Adding IPV4 route entries | 2024-04-19 12:08:30.293 | 04092 | EVENT | S | VPN POST-CONFIG | IPV4 helper API : No error | 2024-04-19 12:08:30.293 | 04092 | ERROR | S | VPN POST-CONFIG | Interface (0) failed to add route [0.0.0.0/0.0.0.0 -> 0.0.0.0 metric 0], while trying to set. Error (2) Unknown | 2024-04-19 12:08:30.293 | 04092 | VERBOSE | D | VPN POST-CONFIG | ---- pRoutes --- (0,0) 0.0.0.0/0.0.0.0 -> 0.0.0.0 [0, 0, 0, 0, 0 ] (0) proto 3 dwForwardPolicy 0, ----end--- | 2024-04-19 12:08:30.293 | 04092 | DEBUG | D | VPN POST-CONFIG | Trying route addition again for 0.0.0.0/0.0.0.0 -> 0.0.0.0 metric 0 | 2024-04-19 12:08:30.293 | 04092 | EVENT | S | VPN POST-CONFIG | Adding IPV4 route entries | 2024-04-19 12:08:30.296 | 04092 | EVENT | S | VPN POST-CONFIG | IPV4 helper API : No error | 2024-04-19 12:08:30.296 | 04092 | ERROR | S | VPN POST-CONFIG | Interface (0) failed to add route [0.0.0.0/0.0.0.0 -> 0.0.0.0 metric 0], while trying to create. Error (2) Unknown | 2024-04-19 12:08:30.296 | 04092 | VERBOSE | D | VPN POST-CONFIG | ---- pRoutes --- (0,0) 0.0.0.0/0.0.0.0 -> 0.0.0.0 [0, 0, 0, 0, 0 ] (0) proto 3 dwForwardPolicy 0, ----end--- | 2024-04-19 12:08:30.296 | 04092 | EVENT | S | VPN POST-CONFIG | Setting IPv4 routes for new configuration | 2024-04-19 12:08:30.297 | 04092 | EVENT | S | VPN POST-CONFIG | Adding IPV4 route entries | 2024-04-19 12:08:30.299 | 04092 | EVENT | S | VPN POST-CONFIG | IPV4 helper API : No error | 2024-04-19 12:08:30.300 | 04092 | ERROR | S | VPN POST-CONFIG | Interface (0) failed to add route [0.0.0.0/0.0.0.0 -> 0.0.0.0 metric 307], while trying to set. Error (2) Unknown | 2024-04-19 12:08:30.300 | 04092 | VERBOSE | D | VPN POST-CONFIG | ---- pRoutes --- (0,0) 0.0.0.0/0.0.0.0 -> 0.0.0.0 [133, 0, 0, 0, 0 ] (0) proto 3 dwForwardPolicy 0, ----end--- | 2024-04-19 12:08:30.300 | 04092 | DEBUG | D | VPN POST-CONFIG | Trying route addition again for 0.0.0.0/0.0.0.0 -> 0.0.0.0 metric 307 | 2024-04-19 12:08:30.300 | 04092 | EVENT | S | VPN POST-CONFIG | Adding IPV4 route entries | 2024-04-19 12:08:30.302 | 04092 | EVENT | S | VPN POST-CONFIG | IPV4 helper API : No error | 2024-04-19 12:08:30.302 | 04092 | ERROR | S | VPN POST-CONFIG | Interface (0) failed to add route [0.0.0.0/0.0.0.0 -> 0.0.0.0 metric 307], while trying to create. Error (2) Unknown | Link to comment Share on other sites More sharing options...
Kari Ruissalo Posted April 22 Author Share Posted April 22 Starts to look like we need to define the session timeout, because once we added that the environment looks like it started working set vpn sessionaction sess_prof_{gw-fqdn}-aon-split-vpn -sessTimeout 1440 Is this mentioned somewhere in the docs, because I wasn't able to find a mention that this is required. The timeout article is pretty thorough, but it advices to other direction (https://docs.netscaler.com/en-us/netscaler-gateway/current-release/vpn-user-config/configure-plugin-connections/configure-time-out-settings.html) Quote In Always On (service mode or user mode), the VPN client ignores all the timeouts. Forced timeout and session timeout decisions occur on the NetScaler appliance and therefore those timeouts work as intended. If such timeout occurs, the VPN plug-in tries to perform automatic authentication. In Always On, as the user device must be connected via the VPN tunnel all the time, do not configure forced timeout or client idle timeout. However, session timeout can be configured to get rid of stale sessions. Some applications, such as Microsoft Outlook, automatically send network traffic probes to email servers without any user intervention. Citrix recommends that you configure Idle session time-out with session time-out to ensure that a session left unattended on a user device times out in a reasonable time. Link to comment Share on other sites More sharing options...
Julian Jakob Posted April 23 Share Posted April 23 On 4/19/2024 at 3:17 PM, Kari Ruissalo said: How does the split tunnel behave if we have an intranet application for 10.0.0.0/8 subnet and then hand out IP Pool IPs from a subnet inside that range (for example 10.0.123.0/24)? Shouldn't be an issue? This shouldn't be a problem. What is the AOService-Log displaying on a CSA client? I know in a customer setup I did also configured the session timeout parameter, but didn't know this could prevent me from your issue. Link to comment Share on other sites More sharing options...
Kari Ruissalo Posted April 23 Author Share Posted April 23 5 hours ago, Julian Jakob said: This shouldn't be a problem. What is the AOService-Log displaying on a CSA client? I know in a customer setup I did also configured the session timeout parameter, but didn't know this could prevent me from your issue. I can confirm that this was never the issue. It was the timeout. Seems that in the Global session settings the GUI shows the timeout value as "30" and the CLI also reflects this: > sh vpn param | grep -i "session timeout" Session timeout: 30 minutes Unrelated, but also had an interesting discussion regarding the mechanism how SAC detects if it's in the internal network. Using the "DNS Suffix" for this is quite clumsy if you ask me and I couldn't find it in the docs, but SAC only performs a lookup for the domain and if the result is RFC private IP, then SAC "knows" it's in the internal network. Also, what happens if you have listed several DNS Suffixes? I would really like to see this as a separate configuration item, like "Always-on VPN Internal beacon address" and some detailed instructions how it works. I can deal with this easily now, but I think it might be a bit confusing for many others... Link to comment Share on other sites More sharing options...
Julian Jakob Posted April 23 Share Posted April 23 10 minutes ago, Kari Ruissalo said: Unrelated, but also had an interesting discussion regarding the mechanism how SAC detects if it's in the internal network. Using the "DNS Suffix" for this is quite clumsy if you ask me and I couldn't find it in the docs, but SAC only performs a lookup for the domain and if the result is RFC private IP, then SAC "knows" it's in the internal network. Also, what happens if you have listed several DNS Suffixes? I would really like to see this as a separate configuration item, like "Always-on VPN Internal beacon address" and some detailed instructions how it works. I can deal with this easily now, but I think it might be a bit confusing for many others... Agree 100%! The way the config is done in GLOBAL DNS-Suffix settings for doing the location detection of the CSA client is very very crappy. I had the same question and asked citrix support what's happening when there are multiple suffixes configured. This was the answer (but never verified in a setup of mine, so no guarantee!) „If multiple DSN Suffixes are configured, for example ".abc.com" ; ".xyz.com", the OS will attempt to resolve one by one. If it succeeds with one suffix, the remaining suffixes are skipped.“ Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now