Jump to content
Welcome to our new Citrix community!
  • 0

iSCSI multipathing to Nimble Array


Chris Phillips

Question

I have a Nimble iSCSI array with 2 data IP addresses and 1 discovery IP on the same subnet.  My XenServer hosts all have 2 NICs that I am using for iSCSI and I have created 2 iSCSI networks with 2 IP addresses on the same subnet as my Nimble data addresses.  I am able to connect fine to the storage, however I only have one path from each host.  I have used the discovery IP of the Nimble to connect to, however I need to get the XenServer hosts to use each of the 2 NICs to connect (giving me 2 connections per host to the storage).  How do I configure this?  I have enabled multipathing on the XenServer hosts but am unsure how to 'bind' the 2 iSCSI addreses to the iSCSI initiator (I am used to working with VMware so am assuming there is a similiar process on XenServer)

 

THanks

 

CHris

Link to comment

17 answers to this question

Recommended Posts

  • 0

I can't quite follow what you're saying here, are the data addresses two IP addresses within the same IP subnet range or are they on separate subnets? I think they need to be on different subnets to be considered different paths as otherwise the routing will always use one of them anyway.

 

You can run

multipath -ll

on the console of the XenServer and it will report back what the multipath kernel module is seeing.

also

iscsiadm -m session -P3

will show you what the iSCSI initiator is seeing. Unless the iSCSI system present two or more devices back there will be nothing for multipath to work on.

Link to comment
  • 0

Hi Mark, thanks for the quick response.  Yes the 2 data addresses are all on the same subnet.  I'll to explain what I am trying to achieve.

 

My Nimble has 2 Storage IP addresses for example 172.16.4.9 and 172.16.4.10, and an iSCSI Discovery address ie 172.16.4.11

My XenServer also has 2 IP addresses on this subnet ie 172.16.4.12 and 172.16.4.13.  The networks are connected via separate switches for redundancy / resiliency

What I am seeing is that on the Nimble I am only seeing a login from one of the XenServer IP addresses ie 172.16.4.12

I am trying to get to having 2 logins per host so on the Nimble I should see connections from 172.16.4.12 and 172.16.4.13.

I can do this on VMware by binding 2 VMK ports with IP addresses to the iSCSI adapter so traffic is passed down both paths, how do I achieve the same on XenServer?

 

Hopefull this makes a bit more sense?  I will have a look at the commands shortly and feedback the output

 

Thanks

 

Chris

Link to comment
  • 0
1 minute ago, Mark Syms said:

There may be a way but I need to check with our support people whether we would support a system with the configuration changes that would be necessary before I'm able to give you any more.

Thanks Mark, very much appreciated.  I take it this isn't an out of the box configuration (like VMware)  I have seen that the Citrix docs say to use 2 different subnets, unfortunately I can't do this as the Nimble is being actively used by VMware hosts at the moment (my Citrix Servers are new so I can make any required changes)

Link to comment
  • 0

Yes, the Citrix docs repeat the guidance from the Linux iSCSI and Multipath tools on which we're based. Having the NICs on the same subnet can suffer from ARP flux issues. To quote the Redhat Linux guidance on this
 

Quote

When a Linux box is connected to a network segment with multiple network cards, a potential problem with the link layer address to IP address mapping can occur. The machine may respond to ARP requests from both Ethernet interfaces. On the machine creating the ARP request, these multiple answers can cause confusion, or worse yet, non-deterministic population of the ARP cache. Known as ARP flux, this can lead to the possibly puzzling effect that an IP migrates non-deterministically through multiple link layer addresses. It's important to understand that ARP flux typically only affects hosts which have multiple physical connections to the same medium or broadcast domain.

 

This may be less of an issue on recent Linux kernels and so it may be possible to add configuration to the iSCSI initiator so that it will use multiple interfaces even if they're on the same subnet.

Link to comment
  • 0

bonding doesn't really give you the right semantics for iSCSI. The load balancing algorithms for bonding aren't designed to utilise the full bandwidth of the link if all the traffic is from a single source to single destination, for this you really need multipath.

Link to comment
  • 0

With iSCSI you don't want to do nic bonding. What you really need is two separate networks so you truly have two paths and XenServer/Nimble

can multipath properly. Now that being said I am able to have two paths to my Tegile using multipath on the same subnet. Maybe Nimble 

could be made to do the same.  I do this through a virtual floating IP address on each controller.

 

--Alan--

 

 

Link to comment
  • 0

Hi Chris,

 

Ignore the fact that this says Equalogic (it's technically supported for Equalogic but the same principle applies in your case). https://support.citrix.com/article/CTX138429?_ga=2.77805584.567951198.1556179279-262899172.1541432090

 

Basically what you need is the bits

 

Quote

To configure two iSCSI interfaces without binding them to a particular network interface:
#iscsiadm -m iface --op new -I c_iface

Example:
# iscsiadm –m iface –op new –I c_iface1 
New interface c_iface1 added
# iscsiadm –m iface –op new –I c_iface2
New interface c_iface2 added

Note: This configuration leads to the default interface being used for all connections. This indicates that all connections are being established using a single interface.

To configure two iSCSI interfaces and bind them to xenbr1 and xenbr2:
iscsiadm -m iface --op update -I -n iface.net_ifacename - v

Example :
# iscsiadm -m iface --op update -I c_iface1 -n iface.net_ifacename -v xenbr1 
c_iface1 updated.
# iscsiadm -m iface --op update -I c_iface2 -n iface.net_ifacename -v xenbr2 
c_iface2 updated.

and 

Quote

Run the following commands to discover and login to the sessions
iscsiadm -m discovery -t st -p <IP of SAN> 
Iscsiadm -m node -L all 

 

HTH,

 

Mark.

Link to comment
  • 0

Hi Mark, this worked great, thanks!  I have added another LUN on my storage now but am seeing 3 paths to it!  when running iscsiadm -m session what I am seeing is it is using my 2 created interfaces c_iSCSI1 and c_iSCSI2 for connections to both LUNS.  The new LUN also has a connection via the default interface.  I have tried modifying this but a unable to at all.  Do you have any ideas how I can sort this out, and why it is using this interface to connect to the one LUN but not the other?

 

THanks

 

Chris

Link to comment
  • 0

Well, I would expect you to be seeing 4 paths for both LUNs one for each combination of source and destination NIC pair but maybe you don't have a cross-switch link to allow them to cross over between the legs. As to why the new LUN is also adding a path on a different interface it all comes down to routing I guess and the iSCSI driver has found a viable route and so is using it.

Link to comment
  • 0

thanks guys.  Let me tell you what I have done so far and how the environment looks so you can hopefully tell me where I have missed something!

 

On each host I have 2 NICS that I am using for iSCSI.  Within XenCenter I have created a network for each of these NICs named iSCSI1 and iSCSI2.  These networks have the VLAN tag on them.  I have then created an IP address on each of the networks so I have 2 IP addresses on each host on the same subnet as the storage array.  Then I have gone to the console and created 2 iSCSI interfaces by running the following:

iscsiadm -m iface -o new -I c_iSCSI1

iscsiadm -m iface -o new -I c_iSCSI2

 

Then I have found the network that the IP addresses are on by running ipconfig.  (the interfaces show as xapi0 and xapi2)

Next I have bound the new interfaces to the network by running 

iscsiadm -m iface -o update -I c_iSCSI1 -n iface.net_ifacename -v xapi0

iscsiadm -m iface -o update -I c_iSCSI2 -n iface.net_ifacename -v xapi2

 

Then I have scanned the iSCSI host with 

iscsiadm -m node -t st -p <iSCSI discovery address on the Nimble>

 

Lastly I have logged in using

iscsiadm -m node -L all

 

When I look at the Nimble I am seeing a connection from the iSCSI_1 interface on the first storage IP address and one from iSCSI_2 on the second storage IP address.  I thought this was maybe how multipathing worked on Xen, but did expect to see 4 connections (2 from each iSCSI_1 to each storage IP address, and 2 from iSCSI_2 to each storage IP address), and from Mark's response above it seems this is what should be happening.  AM I doing the right thing with the binding of my new interfaces to the interfaces?  If it helps, my VMWare hosts that are connected to the same array are all seeing 4 paths and the connections are all through the same switches so I believe I should be getting 4 paths

 

Thanks

 

CHris

Link to comment
  • 0

Chris,

I've never had to do that binding step; also, i generally do the initial discovery process via XenCenter and use the wild-card connection option that should show up for the storage device; it should then see all the paths. You still may need to go through that iscsiadm discovery process on each host afterwards. Four paths should ultimately show up on XenCenter as well as if you run "multipath -ll".

 

-=Tobias

Link to comment
  • 0

Thanks Tobias.  To make sure none of my changes had broken it I have just rebuilt one of the servers, created my 2 iSCSI interfaces within XenCenter and then have connected to the iSCSI target with the * IQN but am still only getting one path.  I have since found out that the iSCSI network is actually routable so I believe that it is defaulting out of the management interface with the default iSCSI interface.  Does this sound like it could be the case?

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...