Jump to content
Welcome to our new Citrix community!

Load Balancing Hadoop DB cluster = really bad performance


Michael B

Recommended Posts

Hi all.  We have recently deployed a Hadoop cluster for some big data needs.  Our app team is using SAS/ACCESS to run queries to this data which the DB servers are behind a pretty basic TCP LB Vserver/ VIP.  However the performance has been VERY poor when trying to use this.   For example a query that takes 16 minutes when ran direct to one of the Hadoop DB servers takes almost 4 hours when connected through the ADC.  Not sure if anyone out there has had any experience with something like this or can offer any suggestions?? 

I have done some basic testing like disabling a server so all queries hit one DB server, and then disabling the other.  Adding persistence, removing persistence.  Using Services instead of service group.  So far nothing has improved.  Of course the NetScaler is the easy thing for them to blame, I suspect it may be an application setting and have asked for our App team to verify through the vendor about load balancing the cluster, but figured I would cover my bases here as well.

Link to comment
Share on other sites

I have seen issues like that before. In most of these cases, using TCP profiles helped. an other wheel to tune would be server side TCP time-out. I suspect network to be guilty. Do network traces. Sometimes, caching may speed up data base queries very much.

Just to mention the simplest thing as well: You don't run into bandwidth issues? Your ADC has a big enough license?

Your attempts (persistence, services instead of groups) don't sound helpful to me.

  • Like 1
Link to comment
Share on other sites

You also might need to look into CS + LB and separate the SELECT/read requests from the WRITE requests and only LB the SELECT statements. But I would start with Johannes recommendations of 1) confirm your NS license and bandwidth and 2) review your TCP profiles.

 

Services vs Service Groups will not affect performance, but settings on the services/service groups might. TCP profiles will handled most of those settings. But you may need to look to see if there are any other unexpected features/settings in use that might be a factor.

 

 

Link to comment
Share on other sites

Thanks for the responses.  I have monitored performance while a job was running and didn't see any capacity issues, and we are licensed for platinum so that shouldn't be a problem.

I haven't used TCP profiles for anything so not sure what setting to look at to help with this.   I don't see using a CS helping as so far all queries are SELECT statements.  Our app team is working with the vendor so hopefully they may find something on that end of this.

Link to comment
Share on other sites

  • 4 weeks later...

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...