Jens Ostkamp

January 23

Usually Citrix releases firmware before they publish security bulletin. So you could write some script or use RSS to regularly check for new NetScaler firmware and once you see a new firmware, just download it immediately, just in case it is for a new CVE. Sadly this has become best practice for me to be able to patch new CVEs right away, as the download site is heavily overloaded once Citrix publishes new CVE and I usually have to wait until the next day

January 23

On 11/30/2023 at 11:11 PM, Ajin Lathikumar1709162671 said:

Hey @Jens .. did you fix this issue?

Hey, the issue gone after the customer resolved some AD issues. I cannot recall what exactly was the issue, but it wasn't NetScaler-sided

January 23

On 12/28/2023 at 1:34 PM, Johannes Norz said:

192.0.0.0/24 is a network reserved by IETF. It doesn't exist on the internet but is not widely used in enterprise networks. So it can be used similarly to 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/26, or 169.254.0.0/16. Citrix is using this network to avoid IP address conflicts with resources existing either in the client-side or in the datacentre-side network.

The client device sends DNS requests to the 192.0.0.1 DNS server, and the gateway intercepts and resolves the name on the behave of the client device, proxying the request to whichever DNS server you configured. This might sound stupid, but that's how it works.

The problem with updated DNS records seems to be a problem with caching on the NetScaler device. I never came across this issue (as DNS entries usually are rather static). Failing over from active to passive node would solve the issue, if I am right about this.

Hey, thanks for your response, just saw the answer now ? What you described is/was exactly our issue. As we don't have HA, we cannot do failover to update DNS as a workaround, but we disabled DNS caching on NetScaler and that seemed to do the trick. After disabling DNS caching, DNS updates did work (conbfigured DNS param -cachehitBypass).

January 23

Are both backend services reachable with http or does it need to be TCP protocol?

Usually the NetScaler does it automatically, if you bind one server to service group port 80 and the other one to port 81, the NetScaler transmits the connection towards the desired port of the bound backend services. Is it not working currently? I have tested it briefly in my lab and it seemed to work just fine

January 23

On 12/29/2023 at 7:14 PM, Keith Giles1709159890 said:

The timeout is set to 60 seconds.

We have a frontend vserver for two radius nodes and the timeout, secret, etc is configured at the vserver level correct?

Are you loadbalancing the RADIUS backend services? If yes, did you configure persistence? From what you describe, it looks like the auth request is sent to ServerA and when prompted with the Challenge, the challenge-response is sent to ServerB (round robin)

November 22, 2023

Dear community,

I am having some issues/questions about DNS configuration regarding SSL VPN.

Right now we are using a SplitTunnel configuration with intranet IPs (intranet IPs 192.168.251.0/24). In general the normal VPN functionality is fine and working, but especially regarding DNS updates, there were some issues lately:

First question is - if I do a ipconfig /all on a VPN connected client, I see a "Citrix Virtual Adapter" with DNS server: 192.0.0.1 (this is not the DNS server configured neither in session profile nor NetScaler configuration). DNS in general works (see more further down the post), but I don't understand why it is showing me that weird of an IP address. The "real" DNS server is in another network (172.20.91.x) and VPN (and firewall) configuration is routing traffic for that network correctly.

Second question/the issue:

DNS updates are not working properly. If we update an existing DNS record, the vpn connected client will not recognize it. NEW A-Records are working fine, but as soon as I update an existing A-Record with a new IP, the client never gets it. I have tested everything with setting low TTL, clearing NetScaler DNS cache, clearing client DNS cache, enforcing ALL DNS requests to be sent over tunnel, nothing seems to work. I am not sure if the two questions/issues are related.

From my understanding, VPN clients should get the same DNS server as configured in NetScaler system settings, if not specified in session profile? Where does that 192.0.0.1 come from?

Full Configuration of that Adapter in attachment screenshot. Please note, Default Gateway is NOT blurred out, it is just empty when outputting it. DNS suffix and and MAC is blurred.

I have also already tried to activate two knobs given in some Citrix Support post regarding secure DNS updates (along with registry keys) - no success ?

The issue surfaced, because we are currently migrating a couple of servers and therefore updating the corresponding DNS records - these will not get recognized by the VPN clients, which is kind of an issue regarding productivity, and workaronding with a local host file publishing is obviously not that great of a workaround.

Thanks a lot in advance! ?

Best regards
Jens

November 22, 2023

On 11/9/2023 at 3:55 PM, Britt Adams1709156619 said:

Yup, Citrix support has been incredibly disappointing recently, unfortunately.

November 9, 2023

If I understood correctly you want to bypass NetScaler MFA for mobile devices using Citrix Files?

You can go for User-Agent Header of the mobile devices and build your authentication flow.

User-Agent expression could look like this:

http.req.header(User-Agent).contains("iOS") || http.req.header(User-Agent).contains("Android") || http.req.header(User-Agent).contains("Windows Phone")

I don't know these mobile headers 100% for sure, so you may need to adjust the header value according to what your mobile devices are and what they send within their header. You can check that via wireshark trace for example (or you just know it :D). You can use "EQ" instead of "CONTAINS" aswell, but this requires the complete and exact user agent header string or it won't work. Also consider using SET_TEXT_MODE(IGNORECASE) for case insensitivity

Now depending on your authentication flow, you have this rule as first priority connecting to an auth action without MFA and then second prio would be "true" rule which redirects to MFA.

E: Also please note, that User-Agent Headers can be manipulated within the request (just as any other Header), so an attacker who gets a grip of that configuration, could bypass MFA by pretending to send request from mobile device

November 9, 2023

6 hours ago, Stefan Wendrich1709160263 said:

Hi,

we use an internal vserver to do kerberos constrained delegation for active sync clients against exchange 2016. This works well, the most of the time.

But without an explanation, the service stops working. The only way to fix it, to disable the vserver, wait 15 minutes and enables the vserver again. Then all is good.

Has someone any idea what the problem could be?

At the moment we have NS13.1 49.13.nc running, but the error has existed since some older versions.

Could be a lot. Kerberos in general is DNS and time dependant. Check if your Appliance can resolve all necessary dns-records and if system time fits "real" time

November 9, 2023

On 11/7/2023 at 10:28 PM, Britt Adams1709156619 said:

Seems kind of odd being the VPX was setup from scratch a year ago with the most recent 13.0 appliance version at that time.

So was it updated since then? There have been four major security vulnerabilities within the last months and if you recently backed up your NetScaler without having these CVEs mitigated, there is a high possibility that you have been compromised on that appliance.

The mentioned .php Files from Defender Screenshot are looking odd aswell. Manipulated php-files have been used to steal credentials after compromising an appliance. You maybe want to look into these mentioned php-files and search for some odd code.

July 25, 2023

Hi,

i dont think there is an exact documentation of how the cookie security works. But from my understanding if you configure cookie setting for the rdp-file, the ADC will inject a validity cookie into the downloaded RDP File and when you open it, you will obviously still connect to ADC first, who will check validity of the cookie and if this passes, proxies the connection to desired rdp destination.

Fun fact: as far as I know there is no security feature like this from RDS in general, so basically the described MITM would be possible for RDS infrastructure with no ADC as RDP proxy / gateway in before. That's why ADC has to do its own validation setting

July 25, 2023

On 7/20/2023 at 9:23 AM, Frank Hedegaard Jensen said:

Hi

After upgrading to build 13.1.49, none of my message actions are logged in ns.log.
Why ?
Have even tried to create brand new message action and responer policy.
btw. Why does a new responder policy require to have an appflow action added ??

I think there are some bugs right now with responder policy settings. See my post here:

https://discussions.citrix.com/topic/419102-adding-multiple-expressions-to-responder-policy/

Maybe this affects whole evaluation of responder policies

July 25, 2023

Hey Kurt,

i think this is a bug right now in latest firmware release. You cannot add or edit a Responder Policy when there is now AppFlow Action bound to that policy. The error (in GUI) basically says, that you need to define an AppFlow Action (which is obviously not true). I have worked around like this:

a) configure Responder Policy over CLI (may be a bit more complicated regarding escaping special characters etc.)

b) create "dummy" AppFlow Action (just deselect everything, create collector with IP "1.2.3.4", do NOT activate AppFlow Feature when prompted) -> you can edit responder policy. After, you go to CLI/SSH and type "unset responder policy *name_of_responder_policy* -appflowaction"

This makes the AppFlow Action go away and keeps the changes to responder policy you initially did. Even though the "dummy" AppFlow Action shouldn't do anything if not activated.

Hope Citrix fixes this soon

Edit: I just tested this with logAction and it is the same bug. GUI expects you to select a logAction (even thought if you dont want one), so you need to do the same as described above but for logAction instead of AppflowAction

May 16, 2023

Hi,

yes I did manage to make it work - the issue was that in Azure the Application for OAuth was not configured correctly, but I currently don't remember the correct type as I do not have access to Azure GUI. I think the Application was configured as "Web SSO something" and it needed to be "Single Web SSO something". After changing that and putting the correct OAuth URLs into the Redirect URIs, everything worked fine.

But I am not 100% convinced of that implementation as there are A LOT of struggles to configure inline token validation. For example, if you have different subdomains for one application with OAuth (e.g "frontend.domain.com" and "backend.domain.com") and authenticated for "frontend.domain.com" the ADC would redirect you again to authenticate if you need resources of "backend.domain.com". I understand why ADC is doing this, as the authentication realm/domain is a different one, but I'd like an option where you can configure that kind of token validation so you don't need to authenticate again. We made it work by reprogramming the backend application to not use different subdomains and then everything worked as expected, but it took some time to figure it out. In the end OAuth works similar as SAML

February 27, 2023

Dear community,

I am currently having issues setting up CItrix ADC as OAUTH SP for an Azure Enterprise Application.

Currently, there is a Web Server (IIS) with an Application configured to do OAUTH towards Azure AD where an enterprise application is configured accordingly - this setup works fine.

For security reasons we want to put Citrix ADC as reverse proxy in front and do the OAUTH flow on ADC (Client -> Content Switch -> Load Balancing, where AAA Auth Srv with OAUTH Action configured -> Azure Login -> Redirect to ADC -> Backend IIS).

I have set up the OAUTH profile looking at following guides:

https://support.citrix.com/article/CTX234873/how-to-deploy-netscaler-as-both-oauth-sp-and-idp (only the SP part)

and

https://xenit.se/blog/2018/02/14/using-netscaler-as-openid-connect-sp-with-adfs-as-idp/

and

https://www.stuartc.net/blog/citrixadc/quick-post-configuring-citrix-adc-gateway-aaa-vserver-to-authenticate-against-azure-ad-using-oauth/

(I have tried different guides and try to fill out everything based on my understanding of OAUTH flow)

Currently I have setup following values in my OAUTH Profile:

OAuth Implementation Type - GENERIC

Client ID - ID of enterprise application

Client Secret - VALUE of the created Client Secret for ADC

Authentication - enabled

Authorization Endpoint - https://login.microsoftonline.com/Tenant_ID/oauth2/v2.0/authorize

Token Endpoint: https://login.microsoftonline.com/Tenant_ID/oauth2/v2.0/token

Cert Endpoint: https://login.microsoftonline.com/8bfee061-3780-4d59-8218-9126796e57e0/discovery/v2.0/keys

User Name Field: upn

Issuer: https://sts.windows.net/Tenant_ID

Grant Type - CODE

All hashing algorithms enabled (HS256, RS256, RS512)

Redirect to Azure works, Authentication in Azure works and as soon as the client is redirect to ADC again there is always the error message "Error validating Access Token, please contact your Administrator".

I have put all log level on debug and looked into ns.log, but I cannot find the exact reason why the validation is failing:

Feb 27 15:32:10 <local0.info> 10.240.3.100 02/27/2023:14:32:10 GMT 0-PPE-0 : default AAATM Message 123 0 : "Sending authn request Oauth"
Feb 27 15:32:10 <local0.info> ns syslogd: last message repeated 1 times
Feb 27 15:32:37 <local0.info> 10.240.3.100 02/27/2023:14:32:37 GMT 0-PPE-0 : default AAA Message 124 0 : "nFactor: deserialize aaa_info, action name copied to samlaction is [act_auth_oauth_rezeptposten]"
Feb 27 15:32:37 <local0.info> ns syslogd: last message repeated 1 times
Feb 27 15:32:37 <local0.info> 10.240.3.100 02/27/2023:14:32:37 GMT 0-PPE-0 : default AAATM Message 125 0 : "OAUTH RESP: ns_aaa_oauth_resp_handler, response code 401 is not 200 OK, bailing out "
Feb 27 15:32:37 <local0.info> ns syslogd: last message repeated 1 times
Feb 27 15:32:37 <local0.info> 10.240.3.100 02/27/2023:14:32:37 GMT 0-PPE-0 : default AAATM Message 126 0 : "AAATM Error Handler: Found extended error code 1310727, ReqType 16386 request /oauth/login?code=oauth_code_b64

I have searched for different troubleshooting possibilites (https://support.citrix.com/article/CTX234873/how-to-deploy-netscaler-as-both-oauth-sp-and-idp especially here), but nothing worked so far.

Used ADC Version is latest 13.1.

Any help or ideas are greatly appreciated.

Thanks a lot in advance!

Best regards

Jens

July 27, 2022

On 7/26/2022 at 11:08 AM, Lukas Rusch1709162191 said:

You all are best! - THANK YOU

I configured MFA with NPS first and it works fine but without nfactor implemented. We probably buy an upgrade from Citrix ADC Gateway Advanced VPX to Citrix ADC VPX 200 Standard Edition. With that, migration should be easy as you mentioned without affecting production or existing users without MFA.

Next I will configure and test SAML. With Standard Edition (nfactor) it should be possible to use MFA with Citrix Workspace, too. May that's also possible with NPS but I didn't test that yet. With nfactor I can test all authentication methods at the same time without reconfigure Citrix ADC and that is nice.

Again: Thanks, thanks, thanks to all of you! You really helped me.

Good to hear everything works so far.

But take care regarding licensing!

Citrix ADC STANDARD Edition does NOT contain licensing for nFactor (since the module is AAA is not licensed in Standard) - so you would need to upgrade towards Citrix ADC Advanced Edition (not Gateway Advanced) in order to use nFactor with AAA module.

But as mentioned before you could also try to upgrade your current Gateway Advanced to the newest 13.1 build - you should be able to use nFactor with Gateway then without any license upgrade.

Posted July 22, 2022 · July 22, 2022

On 7/21/2022 at 4:46 AM, Rhonda Rowland1709152125 said:

Only one vpn (and one AAA vserver) behind content switching and other web sites. Content switching doesn't switch between multiple vpn vservers.

Okay, thanks for clarifying. If Citrix could make this work, it would be such a major improvement for so many things regarding Gateway Deployments (and multiple AAA deployments)

Posted July 19, 2022 · July 19, 2022

Im not sure if your scheme is correct, it says that only the backend services are difference, while authentication and rdp profile are the same for both environments?

Regarding authentication you probably will need to implement nfactor to get a good result, binding more than one / different RDP profile for the same vServer is not possible as far as I know.

I haven't checked for a longer time, does Citrix ADC allow more than one VPN vServer behind Content Switch or is it still limited to one? If not this would be your way to go, create two different non-adressable Gateway vServers, do your configuration as needed and let Content Switch decide which one to contact based on hostname

July 19, 2022

This looks like there is some misconfiguration in your SAML Profile. Can you share the IDP/SP Configuration?

July 14, 2022

Excactly what Gunther says - NetScaler will present you the corresponding Logon Page depending on the RADIUS challenge it got from the local NPS (which constructs the challenge based on what the User has configured as his primary MFA method). You MAY want still to consider nFactor because you can implement your PoC in your production environment without affecting production user (which should not get AzureMFA) - in case you need this kind of scenario

If you use Phone Call or Push you will notice that after you entered credentials, the logon page "stalls" (basically loads endlessly) until you approve the challenge. For non-familiar user this could look like a timeout (there won't be a message telling you to approve the Push notification like it is in Azure).

One more note regardind SAML: don't underestimate the MUCH more configuration complexity and effort you need to do, as you will have a lot more point of failures and different products interacting with everything.

best of luck ?

July 13, 2022

On 7/11/2022 at 3:13 PM, Lukas Rusch1709162191 said:

Thanks Jens!

So if I understand you correctly, it should be possible for me to use nfactor, even if I only have Citrix Gateway Advanced VPX license? Or what do you mean with "...you need to upgrade to a recent build" - a license upgrade? We are on recent NS13.0 build-86.17.

Under "Configuration -> Traffic Management -> "AAA - Application Traffic" has a red exclamation mark but sub-configurations like "nFactor Visualizer", "Groups", "Users", etc. don't have.

But it seems that I could create and add virtual servers and nfactor flows within. Am I allowed to do so?

Nice to read that with UPN/SAM. UPN are matching on-premise with AAD

Yes, that should be possible, as you mentioned it would not be possible for customers with Gateway Only License to switch to advanced policies since nFactor/AAA is not licensed with Gateay. So Citrix made nFactor/AAA availbable for Gateway module when you only have a Gateway License. Im just not sure when they implemented it, thats why I said you may need to upgrade to latest firmware (not license, i meant something like most recent 13.1 - maybe it is available in 13.0 as well, im not that experienced with 13.x versions as they still have so many bugs and luckily Im only working with Advanced licensed appliances).

Oh and I was wrong regarding the expalantion with UPN/SAM, it doesn't change anything on what you need to do, but just to clarify and correct myself: If you authenticate with SAM at NetScaler, NetScaler will pass the SAM towards the local NPS (NetScaler does NOT do AD extraction, hence cannot replace the Sign On Attribute Name), NPS will check with local AD if user is present and at the same time will use the configured UserPrincipalName from the authenticating SAM. This UPN will then be passed towards Azure. You don't need to manually configure this on NetScaler, as you will just need a RADIUS policy. You will need to create appropriate NPS policies though, but there are plenty guides on what you need to do and why. (for example https://www.deyda.net/index.php/en/2019/03/20/microsoft-azure-mfa-cloud-service-in-citrix-adc-version-12-2/)

On 7/11/2022 at 6:14 AM, Alban Hoop said:

Thank you all for your help and answers to my thread.

May I'll test both possibilities. But first I'll start with NPS configuration, because I think it's the easier one. As I understand, with SAML it's possible to use all authentications (approving through push on smartphone, entering one time password (Authenticator App), also approving through phone calls)? ~~But with NPS only OTP is possible?~~ (Edit: It's possible to use calls, too as mentioned on website from Jens - Thank you!)

All the documentations I found only show screenshots with authenticator app. As I was told, we have to use phone calls, too - I did not know that yet so I may have to change POC to SAML. SAML may is better for user experience, too.

Do I really need AAD P2, is AAD P1 not enough for MFA?

I just began with 2FA configuration with Citrix ADC. I'm already struggling with enabling the nfactor flow visualizer. As I thought it would be available with Citrix Advanced VPX but I didn't find a way to enable it. We're on NS13.0 build-86.17.

Wish me luck ?

Using NPS you can still use all the authentication methods proposed by AzureMFA, but you cannot change them while authenticating. It will always use your preferred option, so if it is configured to use Push notification but you somehow lost your authenticator phone, you cannot change the MFA method to SMS/call - this only works when authenticating to Azure directly (which would be the SAML option). Using local NPS should support all methods, but you need to be aware of your RADIUS timeout, since calls/sms can take a bit longer than OTP/Push.

Im not a licensing guy ? But as far as I remember you need P2 user licenses, but maybe check it with someone who actually is into this whole big bubble of Azure licensing. There is something called "Azure MFA external/third party usage", which would be the option with local NPS and needs to be licensed with P2 (or maybe P1).

July 11, 2022

Hi Gunther,

thanks for your response.

Yes the success rule is just as yours, i mean it is basically Citrix recommended configuration for OWA SSO and it worked throughout all the builds. Citrix Support still quiet / checking logs, so no progress here. Downgrading is an option, but we want to stick with 13.1, just hope it gets fixed. We don't use OWA that frequently so it is fine to leave it like that for a couple of weeks, but it is still a pain in the a.

July 11, 2022

Yes you need to use nFactor with it. If you have only Gateway licensed you need to upgrade to a recent build, Citrix made nFactor for Gateway available in order to shift from basic policies to advanced.

You can use both, UPN or SAM should work, but in order to make AzureMFA work in general, your user need to have same UPN as mail. User authenticate with SAM at NetScaler -> NetScaler passes UPN to NPS, NPS passes UPN to Azure (where your user are identified via mailaddress, thats why both attributes must match)

July 6, 2022

You shouldn't delete the "current" log-files.

/var/log/ns.log <- current logfile

/var/log/ns.log.0-99 <- historic logfiles which can be deleted

one hint: 99% of the time, "newnslog" are the files which are consuming the most space, so connect via SCP to NetScaler appliance and find /var/nslog/newnslog.0-99.tar.gz (you will also see if comparing file sizes to ns.log files for example, that these are much much bigger) - delete as much as needed and maybe keep the last 5 days of logging

July 6, 2022

Depending on how good your knowhow with ADC is, you can do both.

Option 1)

Create a independent access (new virtual IP, new external IP or use same IP as productive one with a different port) => let test user use this access and configure your session profile to use the same store as the productive one

Option 2)

Create an nFactor Flow (can be extremely complex to understand how nFactor works, but in your case it should be okay as you "just" use two different methods based on AD group membership) which differentiates between different AD Groups, have one AD Group with productive user, using access as before (no MFA) and one Group with test user, using Azure MFA for authentication

This Option could also be achieved without nFactor by just doing some intelligent Basic Authentication Policy chain-binding on the Gateway vServer itself, but since Basic Authenticaiton Policy got deprecated with 13.1 (still working tho, no one knows for how long) you shouldn't use these.

In General:

You need to know what kind of Azure MFA implementation you want. There are two possibilities:

Use RADIUS:

Have a local NPS-Server with AzureMFA Extension (requires AAD P2 Licenses for each user), have NetScaler make a RADIUS request towards that local NPS-Server which then will forward the request towards Azure (see: https://www.deyda.net/index.php/en/2019/03/20/microsoft-azure-mfa-cloud-service-in-citrix-adc-version-12-2/)

2) Use SAML:

Set up FAS-infrastructure (requires an Enterprise CA) for Desktop SSO, make NetScaler request SAML towards Azure (in fact you won't use NetScaler for authentication anymore, NetScaler redirects the user towards Azure directly and uses SAML Answer to pass user towards Citrix infrastructure). Downside: more complex since you need to set up FAS correctly and need good know how about certificate and how they work

Upside: User experience is a lot better as you can use all the cool Azure MFA functions (change method with each sign on, same look a like etc, password won't get transmitted to NetScaler, see: https://www.deyda.net/index.php/en/2021/12/21/saml-authentication-between-citrix-microsoft-with-azure-mfa/)

I've configurd both, both is working great, in the end it's a design decision what to use, depending on your infrastructure in general.

If you will use SAML I'd strongly recommend you to create a completely new Virtual Server for an independent test-environment

P.S: Im really sorry, somehow my browser didn't load the other answers so I thought this was unanswered yet. Most of my points are already pointed out, sorry for answering it again.

Forums

Articles

Labs

Videos

TechZone

Citrix Community Articles

Events

Profiles

Posts posted by Jens Ostkamp