Jump to content
Welcome to our new Citrix community!
  • 0

Xenserver 6.2: Slow file copying across VM network


Question

Hi,

 

I've got a homelab and use a combination of Server 2012 R2s with Windows 8.1s and window 7 installs as virtual machines.  I also have a number of physical computers around the house.  Due to a rather catastrophic network failure, I've had to rebuild everything. I took the opportunity to create a new server to house my media and files but I've come across an issue I haven't had before and I hope someone can point me in the right direction!

 

When I copy a file between VMs, it peaks to full disk access and then drops till it bombs out at 0mb/s, it will stay like this for a few seconds before climbing again.  It continues peaking and dropping until the file has been transferred. However, if I transfer a file over the physical network from a virtual machine to a physical machine then it copies at max network speed (1gbps) until transferred.  If I transfer a file on the storage server between HBA attached HDDs then all is good and I get full speed again...

 

Is this a known issue?  Is there a setting in the network config I need to make changes to?  Local storage are SATA disks @ 7200rpm running Xenserver 6.2.

 

Any help is happily received!

 

Chris

Link to comment

Recommended Posts

  • 0

Hi Tobias, and thanks!

 

Currently I have a management network port which is what I use Xencenter on and an intel dual 1gb Ethernet card as the vm network card, I'm assuming therefore that all VM data between VMs is through the vswitch and vm data to physical is through vswitch and then external network?

 

I've copied a 10GB file across to another VM with TOP running and I've still got 1/2 the ram assigned to the dom0 still available.

Link to comment
  • 0

Unless possible to specify otherwise, traffic between VMs will flow over the primary network interface (for things like Xenmotion, you can specify alternatate networks). OK as far as dom0 memory goes, bur how about CPU load (which I assume is also not that high)? What is the total RAM and number of dom0 VCPUs you have allocated? Is your server at the latest BIOS, firmware and XenServer patch level? Do any of the NICs need driver updates? Are XenTools properly installed on your VMs? Performance on XenServer can be hard to daignose as there are so many components involved.in the overall picture.

Link to comment
  • 0

Again thank you Tobias.  With respect to the primary network interface, how do I find which one this is? As I said this is a homelab and therefore running on consumer grade hardware, so nothing special or great about it but it works fine in ESXi doing the same transfers.

 

For Dom0, I am simply running 752MB at the moment, of which 1/2 is currently in use.  There are 4vcpus assigned, with only 0.2% utilisation.  I currently have 4 VMs running, 3 Windows Server 2012 R2 installs and one windows 8.1 install.  Xenserver is running latest patch that is available to the public.

 

NICs are a realtek 8168 on the GA-990FXA-UD3 and the dual port intel gbe interface.  The realtek is my management NIC and the intels are the interface assigned to the VMs.  Xentools is installed on each VM.

 

My storage server has an LSI megaraid card passed through as an HBA adaptor and copying from this to another physical machine is at full speed.  Copying to another VM from the storage VM HDD attached to the HBA is slow and peaky as described in the original post, and is the same doing the reverse VM>VM with passthrough'd HBA adaptor.

 

I think it must be a windows thing as when I had unRAID installed, I had full speed from VMs to the unRAID VM and vice versa.

Link to comment
  • 0

So when I pass an NIC directly to the VM, and then disable the virtualised NIC everything runs fine.  Copying to and from the storage server is quick and painless, but this doesn't solve the issue of copying from VM to VM via the virtualised NICs.

 

What else can I do to investigate these issues?  As an aside, the data is generally coming from the directly passed HDDs attached to the HBA so really the main disk usage will be from the recieving or sending VM not the actual storage server VM.

Link to comment
  • 0

Realtek NICs are notoriously not good performers, so that may be the crux of it (at least for external connections). Do you have anything else you can install?

 

As to your primary network interface, you can either ssh to your XenServer or from the XenCenter console for that XenServer you can run "xsconsole" and under "status display" you will see what is designated the management network device, as well as under the "network and management interface" entry.

 

-=Tobias

Link to comment
  • 0

If it is going to and from VM on same machine, it will not traverse the external network. With this scenario, the network speed is literally at the speed of your memory. The bottleneck is likely the I/O as you read from one VMs VDI and then write it to the other VMs VDI.

 

What hardware do you have for storage? In my lab, I have a single SATA in old Dell 390. Is Early SATA and I can easily saturate the storage bandwidth.

Link to comment
  • 0

Again thank you Tobias.

 

I might need to be clearer with the NIC setup.  I have 3 NICs available in the 'server'.  1 is the built in Realtek NIC, attached to the motherboard; then I have a dual port gbe card by intel attached via PCI.  Xenserver was setup so the management nic was the realtek and was given a static ip.  Each VM (currently four are running) where then assigned one of the two available intel NICs, so effectively all communication from the VMs to the outside world where via the intel while the main xenserver management and therefore VNC connections to the servers where by the realtek.

 

Does the assigned management NIC affect the transfer of data from one VM to another?  Does the virtualised NIC only use the phyiscal intel when transferring data to and from the host?  I had always assumed (as I believe this is how ESXi works) that the virtualised NICs in each VM was connected to a vswitch and any data transferred between them simply used the switch and data was only routed to the physical NICs when going external to the host?

 

I guess you can tell that I'm not clear on how networking is handled in Xenserver.

 

Thanks,

 

Chris

Link to comment
  • 0

The host management interface (NIC 0 - Realtek) is used by the Control VM (AKA "dom 0").

 

Your guest VMs, if assigned to NIC 1 and 2 (Intel) will use that NIC for access to and from the physical machine.

 

If the 4 VMs are transferring data to each other, they will use the internal switch and never leave the physical machine via Intel NICs.

Link to comment
  • 0

If it is going to and from VM on same machine, it will not traverse the external network. With this scenario, the network speed is literally at the speed of your memory. The bottleneck is likely the I/O as you read from one VMs VDI and then write it to the other VMs VDI.

 

What hardware do you have for storage? In my lab, I have a single SATA in old Dell 390. Is Early SATA and I can easily saturate the storage bandwidth.

Hi James,

 

dc01 is the primary domain controller and is a VM running server 2012 r2.  Storage is a VM running server 2012 r2 with a LSI megaraid card passed through.  The HDDs attached to the LSI are 5400 rpm WD 3TB drives, no more than a year old.  Both VMs are on the same host.  I appreciate that 'all-in-ones' are not a favourable option, but at present it serves my needs well as I'm not using HA or anything else yet. Storage's stats in xencenter neve exceed 20%cpu and 40% memory when copying or doing activites.

 

So, some infomration:  copying across the drives attached to the LSI SAS card moves at around 125mb/s on average and the disk utilisation in Xenserver is generally less than 100kb/s for storage, that indicates to me that the server is not using the VDI for anything bar general use.

 

Copying from Storage to an external PC to the host, i.e another phyiscal machine, data transfer speed is limited by phyiscal network.  the same happens if I copy TO storage.

 

Copying from storage to DC01 which are both on the same host, connected to the same switch and assigned one of the intel NIC ports  will start at full speed (125mb/s) and then drop quickly to 0, hold there for a little while and then spike and drop, spike and drop and take a very long time to copy or move.  When doing this, the only disk usage is on the VDI that hosts the recieving or sending VM and not storage.

 

As soon as I passthrough an NIC to the storage VM, all the issues go away but then I'm obviously limited by the phyiscal network speeds as its going outside of the host through a physical switch and back in again.

 

Chris

Link to comment
  • 0

The host management interface (NIC 0 - Realtek) is used by the Control VM (AKA "dom 0").

 

Your guest VMs, if assigned to NIC 1 and 2 (Intel) will use that NIC for access to and from the physical machine.

 

If the 4 VMs are transferring data to each other, they will use the internal switch and never leave the physical machine via Intel NICs.

Hi James and thanks.  So effectively, they do not use NIC0 either as this is assigned to Dom0 and its all just Vm>vNIC>vSwitch>vNIC>VM or vice-versa?

Link to comment
  • 0

Rightio, thanks James.  So I have do have an issue with copying data from one VM to another on the same host.  I realise that you will never get the same transfer rates as phyiscal machine to physical machine but I would have expected to have a more continuous transfer rate. 

 

Certainly, on the same hardware ESXi 5.5, maxes out the transfer close to 100mb/s if I am only doing one vm to one vm (obviously the more transfers occur across multiple VMs on the same storage devices you will reach the phyiscal storage device max first) so why is xenserver struggling? Dom0 usage is very low during this time, only a single VDI is really being accessed (it is either recieving or sending data).

 

Any pointers?  I recall that unRAID (based on slackware) was full speed across all VMs even when hosted as a VM itself on the same host... Leads to be believe that this is a windows thing.  I might try a windows 7 install and see if that makes any difference.

Link to comment
  • 0

So we do have the storage intefaces between the 2 VMs, where we read and then we write. If this was Windows on bar metal and we moved a file from one folder to another, we really do not move the data, only where we reference the data. So, with 2 VMs, we have 2 VDIs, each of which is a separate data container. That is why we max out the storage sub system, as we are not modifying a pointer to data location, we actually move the data.

 

As to pointers, we need more storage bandwidth. :) I am sorry about that. Perhaps other forum users have recommendation. As to my workstation, I may rip out the SATA drive and put in a recycled RAID controller (and disks).

Link to comment
  • 0

Ok, so even if the data is contained on hdds not actually part of the vdi they still pass through a storage interface between VMs? 

 

I.e.

 

Storage Server VDI                            > Storage interface  >   DC01 VM

LSI SAS Card (Passthrough'd)

Disk 1

Disk 2... etc.

 

I'm asking this as I said before, unRAID is based on slackware (linux distribution) and when hosted as a VM in the main host with these VMs, the transfer speed is maximised with no loss of performance. Could this point then towards the paravirtualised drivers for windows?

 

I'll setup a new ubutnu install on the host, pass through the hba controller and see what speeds I get when copied from a windows vm to the linux vm.. I suspect windows to linux will be at full speed and windows to windows will stutter and struggle...

Link to comment
  • 0

Sorry, I am not familiar with unRAID. If there was issue with PV-drivers, this should have been reported through standard support process (you would need a support agreement). It is likely issue is unique to your environment (otherwise we would have tens of thousands of customers with issue). Using a VM as a storage device adds an extra layer. Ideally, we would have storage on physical media and not layered inside of a VM (if I understand your design).

Link to comment
  • 0

Hey James, that's ok I was using it as an example.  It is simply a parity aware storage system that can be expanded using any sized disks based on slackware.  I wonder how many people actually do what I am currently doing with my setup, probably not many... I know I don't want to be, but my system is at its beginnings and I want to ensure that I can scale out carefully as cash is an issue to building the perfect distributed HA setup.

 

As I'm not actually using it yet, I might try the latest creedence alpha and see if that works any differently... I suspect the chagne to 64bit Dom0 and improvements to the various subsystems might make transfers better... I shall see!

Link to comment
  • 0

Hi

 

So I think this is a windows issue when running as a VM.  I've just had a good play.

 

Just installed Ubuntu 10.10 and passthrough the LSI megaraid SAS adapter to the new install.  Did a series of copies from the Ubuntu vdi to the windows vdi.  Got a consistant 11mb/s transfer speed.  Did the same for the same file but this time from the sas adapter attached to Ubuntu to the windows vdi and I got again 11mb/s transfer speed.

 

Now I then reversed it and copied the file from the windows vdi to the Ubuntu vdi and then to the sas adapter.  Both times I hit speeds of 55mb/s when writing to the Ubuntu install.

 

There must be something wrong with the windows drivers in xentools as copying from windows vm to windows vm is very poor yet Linux to windows is better and windows to Linux is great and more inline with what I thought should happen.

 

I'll now try windows 7 to windows 8 and see if it is a windows server thing...

 

Edit:  Ok so its not just a server thing, my brand new 7 and 8.1 installs are doing the same things from windows to windows, they're not even storage devices... just trying to copy a 4gb file across the vms!

 

any other thoughts or help?

Edited by Chris Beasley
Link to comment
  • 0

I had a similar thing doing large database backup file copies last week. Copying on a 2012 server and moving file from 2008 to 2012 timed out. This was from VM to VM on the same host. Moved one VM to another host in the pool and the file copy went normal. I just wrote it off as a bizarre windows thing but it could have been XenTools.

 

Alan Lantz

SysAdmin

City of Rogers, AR

  • Like 1
Link to comment
  • 0

Hi James,

 

Will try that later...  Installed the latest nightly build of Xenserver Creedence... Had some troubles with my local attached storage but no biggie, however after it had upgraded the server fine from the current release I was then able to transfer data (4gb iso) from one windows VM to another Windows VM at the same speed as windows to Linux under 6.2.

 

Are we able to use xentools from credence in 6.2 or does that rely on the 64bit Dom0 etc?

 

Chris

Link to comment
  • 0

Afternoon everyone (well it is here in New Zealand!).  The culprit is Xen Tools.  I did a file transfer before installing anything and windows vm to windows vm 4gb iso had a sustained transfer rate of 5MB/s, probably about right for the software emulated NIC.  Installing the current version of Xen-Tools from Xenserver 6.2 produced the erractic behaviour listed above.  Removing completely and updating to the latest version in the nightly builds of credence gave a sustained transfer rate of around 55MB/s, practically the same as Linux distro's where doing... This is less than what credence can do, but given that is a wholesale update to Xenserver I am not surprised...

 

So  far haven't had any trouble so will continue to use the new xen-tools in place of the standard ones.

 

does this tie in with the report about Server 2003 and slow networks that Citrix support issued a hotfix for?

 

Chris 

 

EDIT:  Hmmm, as soon as I apply static IPs to the servers then it stops working so well... Will have to look further.

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...