Jump to content
Welcome to our new Citrix community!

Load balance gPRC service


Ross Bender

Recommended Posts

Is it possible to load balance gRPC services on the Netscaler? gRPC is a type of HTTP/2 and I'm not quite sure if the Netscaler can load balance this and be able to add content policies for a request (https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md).

 

We have "load balanced" gRPC as a TCP service, but we'd now like be able to apply policies based on the request to load balance to different backends.

Link to comment
Share on other sites

Hi Ross,

 

I did not know this protocol existed before, so I'm going to give you what I think the NetScaler would do.

 

This looks kinda different from the HTTP/2 implementation, so possibly using HTTP LB will not work.

However, the NetScaler should disregard what protocol on Layer 4 and above is used if you use TCP LB instead. The problem with this is that if you use TCP Load Balancing you will not be able to use policies.

 

If the response that your service gives is compliant with the HTTP/2 standard, it should be possible to use policies. Did you try to do it ?

Link to comment
Share on other sites

I haven't had any luck with this so far with trying to do layer 7 balancing on gRPC. Configuring an HTTP virtual server has resulted in communication failures between the gRPC client and the Netscaler. The client/application logs show:

rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR

I've tried to make my HTTP/2 profile as lenient as possible, disabling all "drop invalid xxx" features and enabling as much as possible:

add ns httpProfile http2_grpc -conMultiplex DISABLED -dropExtraCRLF DISABLED -webLog DISABLED -http2 ENABLED -http2Direct ENABLED -altsvc ENABLED -http2HeaderTableSize 16384

If I run a capture I can see in the trace the Netscaler is closing the connection (RST_STREAM, GOAWAY).

 

Any other hints on the HTTP/2 side to get this to work?

Link to comment
Share on other sites

  • 3 months later...
On 1/21/2020 at 12:58 PM, Ross Bender said:

I haven't had any luck with this so far with trying to do layer 7 balancing on gRPC. Configuring an HTTP virtual server has resulted in communication failures between the gRPC client and the Netscaler. The client/application logs show:


rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR

I've tried to make my HTTP/2 profile as lenient as possible, disabling all "drop invalid xxx" features and enabling as much as possible:


add ns httpProfile http2_grpc -conMultiplex DISABLED -dropExtraCRLF DISABLED -webLog DISABLED -http2 ENABLED -http2Direct ENABLED -altsvc ENABLED -http2HeaderTableSize 16384

If I run a capture I can see in the trace the Netscaler is closing the connection (RST_STREAM, GOAWAY).

 

Any other hints on the HTTP/2 side to get this to work?

 

You should set this HTTP Profile to both the virtual server and service and also enable the HTTP/2 on the server side by using "set httpparam -http2ServerSide ON". With these two config, you should be able to layer 7 load balancing/content switching or applying any other NetScaler layer 7 features into gRPC request or responses.

Link to comment
Share on other sites

On 5/4/2020 at 2:09 PM, Krishna Khanal said:

 

You should set this HTTP Profile to both the virtual server and service and also enable the HTTP/2 on the server side by using "set httpparam -http2ServerSide ON". With these two config, you should be able to layer 7 load balancing/content switching or applying any other NetScaler layer 7 features into gRPC request or responses.

 

Thanks for the response, @Krishna Khanal. I didn't know about needing to set the global HTTP parameter. Unfortunately I am still seeing the same issue after enabling the setting and making sure the HTTP profile is bound to both the LBVS and the service.

 

We are using Netscaler version 12.1 build 56.22. Config as follows:

set ns httpParam -dropInvalReqs ON -http2ServerSide ON

add ns httpProfile nshttp_default_profile_grpc -dropExtraCRLF DISABLED -webSocket ENABLED -http2 ENABLED -http2Direct ENABLED

add service test-grpc-http2 it-lt79 HTTP 5557 -gslb NONE -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -sp OFF -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP YES -httpProfileName nshttp_default_profile_grpc -appflowLog DISABLED

add lb vserver Grpc-Http2 HTTP 192.168.202.65 5557 -persistenceType NONE -cltTimeout 180 -httpProfileName nshttp_default_profile_grpc -appflowLog DISABLED
bind lb vserver Grpc-Http2 test-grpc-http2

From the trace I captured, I can see the Netscaler is closing the request and not trying to send the request to the backend service:

image.thumb.png.ac40d8fdbe31d96768295d6ce6c38226.png

 

The HTTP2 part of the Netcalser RST_STREAM packet (3rd packet in above diagram) shows RST_STREAM (3) and PROTOCOL_ERROR (1). The HTTP2 part of the GOAWAY packet (4th in above diagram) shows GOAWAY (7).

 

Any other suggestions I can take a look at?

Edited by Ross Bender
added NS version
Link to comment
Share on other sites

On 5/8/2020 at 1:33 PM, Ross Bender said:

 

Thanks for the response, @Krishna Khanal. I didn't know about needing to set the global HTTP parameter. Unfortunately I am still seeing the same issue after enabling the setting and making sure the HTTP profile is bound to both the LBVS and the service.

 

We are using Netscaler version 12.1 build 56.22. Config as follows:


set ns httpParam -dropInvalReqs ON -http2ServerSide ON

add ns httpProfile nshttp_default_profile_grpc -dropExtraCRLF DISABLED -webSocket ENABLED -http2 ENABLED -http2Direct ENABLED

add service test-grpc-http2 it-lt79 HTTP 5557 -gslb NONE -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -sp OFF -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP YES -httpProfileName nshttp_default_profile_grpc -appflowLog DISABLED

add lb vserver Grpc-Http2 HTTP 192.168.202.65 5557 -persistenceType NONE -cltTimeout 180 -httpProfileName nshttp_default_profile_grpc -appflowLog DISABLED
bind lb vserver Grpc-Http2 test-grpc-http2

From the trace I captured, I can see the Netscaler is closing the request and not trying to send the request to the backend service:

image.thumb.png.ac40d8fdbe31d96768295d6ce6c38226.png

 

The HTTP2 part of the Netcalser RST_STREAM packet (3rd packet in above diagram) shows RST_STREAM (3) and PROTOCOL_ERROR (1). The HTTP2 part of the GOAWAY packet (4th in above diagram) shows GOAWAY (7).

 

Any other suggestions I can take a look at?

 

We need to look at the trace to identify the reason for the RST_STREAM. Please send across the trace so that we can suggest the recommendation.

Link to comment
Share on other sites

2 hours ago, Ross Bender said:

 

@Krishna Khanal Please find attached capture between client and Netscaler. If you get some more detail on it, please share.

 

Thanks!

2020-05-08-capture.cap 1.99 kB · 1 download

 

Your gRPC client isn't sending value of ":authority: " header due to which NetScaler resets the stream with RST_STREAM. Once you correct that, the gRPC traffic should be accepted and processed.
 

  • Like 1
Link to comment
Share on other sites

On 5/11/2020 at 11:06 AM, Krishna Khanal said:

 

Your gRPC client isn't sending value of ":authority: " header due to which NetScaler resets the stream with RST_STREAM. Once you correct that, the gRPC traffic should be accepted and processed.
 

 

Thanks for the detail, @Krishna Khanal. Can you please share how you came to find this answer?

 

Do you know why the Netscaler is doing gRPC validation, or if there's a way to disable it?

Edited by Ross Bender
tag user and clarify question
Link to comment
Share on other sites

  • 3 months later...
  • 4 weeks later...

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...