Jump to content
Welcome to our new Citrix community!
  • 0

Ethernet download speeds extremely limited with Guest Tools installed?


George Plymale

Question

I'm fighting an issue with an installation of a Windows guest computer. I've been attempting to use the Windows guest as a gateway server via some software called Connectify, but it seems that every time Connectify is started the bandwidth goes through the floor (normally Internet speeds are around 110-119 Mbps, but they go down to roughly 15-30 Mbps when Connectify is fired up).

 

I tested  and according to support we found it was the actual driver somehow.  I received the following back from support and wanted other Xen Server users' thoughts on this issue as I think we're going to need some help from Citrix on this one.  This was a reply from the support person having installed Xen Server themselves to see if they could reproduce the problem:
 

Quote

 

Prior to installing the Guest Tools on the VM, the ethernet NICs show up as 100 Mbps ethernet. With this setup, I did not see any difference in performance between having Connectify running and not running. Both were able to fully utilize the 100 Mbps.

 

After installing the Guest Tools, it uses a Xen driver for the ethernet which supports 1 Gbps speeds. On a 1 Gbps Internet connection without Connectify running, I could get around 900 Mbps down and 900 Mbps up. With Connectify running, I was getting 10 Mbps down and 900 Mbps up. So I think I was able to reproduce the problem you are seeing. It only seems to affect the download speeds. It seems to be some sort of conflict between the Xen ethernet drivers and the Connectify filter driver, but I'm not sure which is at fault.

 

I was able to confirm that after uninstalling the Guest Tools, the ethernet went back to 100 Mbps ethernet, and I was able to get 100 Mbps speeds with Connectify running. I haven't had a chance to play around with the advanced options on the Xen ethernet drivers to see if changing any of the offloading settings makes a difference in performance with Connectify running.

...


Does anyone have any ideas of settings or advanced ethernet driver options that could make a difference in this scenario?

Link to comment

8 answers to this question

Recommended Posts

  • 0

You might want to look into taskoffload for the VM and also disabling offloading at the XenServer Dom0 level. Thats the only thing I could think of 

that might make a difference. How does it work without the XenTools installed ? If you don't need to migrate the VM that may be an option.

 

--Alan--

 

Link to comment
  • 0
8 hours ago, Alan Lantz said:

You might want to look into taskoffload for the VM and also disabling offloading at the XenServer Dom0 level. Thats the only thing I could think of 

that might make a difference. How does it work without the XenTools installed ? If you don't need to migrate the VM that may be an option.

 

--Alan--

 

 

He says that if you disable Xen Guest Tools the problem goes away, BUT the bandwidth is throttled as it doesn't go over 100 Mbps. I guess the Tools are required to get it on up to 1 Gbps speeds...

 

As far as "taskoffload" goes, is this to speed up performance of the guest in some fashion? I'm not personally familiar with this terminology yet in context of Xen Server.  When I measured the processor utilization and so forth I didn't see any problems in the Windows guest during transfer speed tests, so I don't know if that would help unless it somehow manages to improve low level I/O issues that I'm not seeing for devices such as the ethernet cards, etc.

Link to comment
  • 0
On 2/8/2018 at 0:53 PM, Alan Lantz said:

Yes taskoffload is a registry change in the Windows OS. Command line on XenServer you can make additional offloading type settings with ethtool 

or by modifiying VM's with xe commands.

 

https://support.citrix.com/article/CTX212540

 

--Alan--

 

 

 

So I found that `tcpdump -i eth0 -v -nn| grep -i incorrect` showed me a huge number of checksum errors when a speed test was executed.  I then proceeded to issue `ethtool -K eth0 rx off tx off sg off tso off ufo off gso off gro off lro off` and all of these went away upon running a speed test again!  I also found that my speed was back up to where it should be!  So I was very pleased with that.

 

But I have some questions: 

  1. when I executed the above to turn those off, it replied with "Cannot change the udp-fragmentation-offload" and "Cannot change large-receive-offload".  What should I do about these?
  2. I assume that using `ethtool` is temporary and so I need to proceed with steps 5 and 6?  What is PIF vs VIF and how do I choose or do I do both?
  3. In the link we're referencing that you sent here, step #2 says: "If you find any incorrect checksum, then there is need for disable the offloading parameters."  This is confusing since it seems that we are proceeding to disable offloading in the next steps.  I found checksums and so we need to disable offloading it appears...  Is this document step #2 incorrect or simply not being explained clearly?
  4. I do not see any documentation about changing things in the Windows OS registry that you reference... Am I missing something?  Even without disabling anything in the Windows registry I'm still seeing these speed improvements on the rest of the network......
Link to comment
  • 0

1) probably because the nic you haven't doesn't support those features. Just ignore the messages.

2) yes its temporarly, adding to pif/vif will make it permanent.

3) I don't think the wording is clear. If you see incorrect checksums then you need to add entries to disable offloading

4) https://docs.microsoft.com/en-us/windows-hardware/drivers/network/using-registry-values-to-enable-and-disable-task-offloading

   You can disable individual items or all by using DisableTaskOffload at the bottom. This is generally what I do in my

   VM's which I find easier that all of the pif/vif changes. It would probably be best to do it on XenServer, but I just

   haven't saw enough performance issues to make it worth the effort, disabletaskoffload in my Windows VM's seem

   to be enough.

 

--Alan--

 

Link to comment
  • 0
1 minute ago, Alan Lantz said:

1) probably because the nic you haven't doesn't support those features. Just ignore the messages.

2) yes its temporarly, adding to pif/vif will make it permanent.

3) I don't think the wording is clear. If you see incorrect checksums then you need to add entries to disable offloading

4) https://docs.microsoft.com/en-us/windows-hardware/drivers/network/using-registry-values-to-enable-and-disable-task-offloading

   You can disable individual items or all by using DisableTaskOffload at the bottom. This is generally what I do in my

   VM's which I find easier that all of the pif/vif changes. It would probably be best to do it on XenServer, but I just

   haven't saw enough performance issues to make it worth the effort, disabletaskoffload in my Windows VM's seem

   to be enough.

 

--Alan--

 

 

Thank you so much.  This is fantastic. 

 

So I can bypass the pif/vif step if I do #4?  If that's true then that can save a huge amount of time.  I guess I'm not clear on what to disable when it comes to the VIF portion and so I was disabling all the VMUUIDs...  If this is correct, then I suppose to be safe I will have to do it for Linux manually in the pif/vif step since there's no registry?...

Link to comment
  • 0

Also, I decided to do a `tcpdump -i any -v -nn| grep -i incorrect` and found a huge amount of traffic flagged as incorrect there as well.  I began attempting to hunt things down, so I started dumping each interface.  I got to `xenbr3` and discovered that this is one of the offending interfaces.

 

How can you interpret what `xenbr3` actually is?  It's obviously a bridge, but which physical interfaces does it bridge, etc?  Is there a way to nail this down to understand what's going on?

 

`ethtool -K xenbr3 rx off tx off sg off tso off ufo off gso off gro off lro off` halted the problem, but I'm still seeing a lot of `incorrect` flags so I've got more hunting and figuring out ahead of me.

 

Also, the reason I need to understand the above is so that I can figure out how to get the pif and vif info in order to store the `xenbr3` setting.  I can't seem to figure out how to isolate the appropriate commands for permanent storage without understanding this better.

 

I did find that `vif21.4` is also having a good number of `incorrect` flags, however, `ethtool -K vif21.4 rx off tx off sg off tso off ufo off gso off gro off lro off` fails with:

Cannot change rx-checksumming
Cannot change udp-fragmentation-offload
Cannot change large-receive-offload
Could not change any device features

I suppose there's nothing I can do about that one since when I check it again it's still showing incorrect flags...

 

 

FYI, discovered a troubleshooting technique that has proven helpful in tracking down these troubled connections and thought it might be useful to know.  This allows you to not only grep through "incorrect" flags, but to have the INTERFACE also indicated at the start of the line, which will allow you to rapidly isolate what interfaces are experiencing the problem.

 

What you'll want to do is to use this script (from here, German) (version 1.3) I've uploaded (you can wget it on the server if desired): dump.sh

 

It can be executed via: 

$ dump.sh [-i interface] [tcpdump-parameters]

This can then be executed appropriately to filter through the results, example were I only look for "incorrect" entries and I cut out certain interfaces from the result:

$ sh dump.sh -v -nn | grep -i incorrect | egrep --line-buffered -v "Interface:lo:|eth3" #filters out `lo` and `eth3` interfaces

 

Link to comment
  • 0

Same issue on windows server 2022 which running as windows NAT service and SMB storage server when i build my new XS lab recently 

the rca pointed to PIF enabled the large receive offload by default.  

i checked the windows ps Get-NetAdapterLso but seems only LSO can set but RSO there is no API.  manullay set it in windows NIC driver advanced tab to disable fix the issue. 

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...