Jump to content
Welcome to our new Citrix community!
  • 0

Unable to add iscsi IP address after outage


Reg Dancy1709159709

Question

Hello,

 

Recently we had an outage and lost connectivity to my Xen pool. It turned out to be a network issue. All of my Xen hosts came back fine except one, which happens to be the pool master. The mgmt IP is fine and working but the iscsi IP addresses have disappeared. I tried to add them back via Xen Center but when I click on the configure button under the networking tab the OK button is shaded out. Meaning it's not allowing me to save my entry I click on the "add IP address" button. Any ideas?

Link to comment

13 answers to this question

Recommended Posts

  • 0

I should also add that multipath seems to be broken even though daemon is still running, however....

 

[root@xs4 rc.d]# multipath -ll

[root@xs4 rc.d]#

 

xe pbd-list sr-uuid=04bbfb22-fc69-5c9d-a7df-600ca1153e38 host-uuid=dc25b112-d348-44d1-be85-c239bf90ba4d
uuid ( RO)                  : 191ec47b-4a89-c25b-c81f-84d8394de235
             host-uuid ( RO): dc25b112-d348-44d1-be85-c239bf90ba4d
               sr-uuid ( RO): 04bbfb22-fc69-5c9d-a7df-600ca1153e38
         device-config (MRO): multihomelist: 172.31.25.101:3260,172.31.25.102:3260,172.31.25.103:3260,172.31.25.100:3260,172.31.25.105:3260,172.31.25.106:3260,172.31.25.104:3260,172.31.25.107:3260; targetIQN: iqn.2010-06.com.purestorage:flasharray.3c9bf12bd7c53cf3; target: 172.31.25.101; SCSIid: 3624a93701ab79778588c412e00011033; port: 3260
    currently-attached ( RO): false


[root@xs4 rc.d]# xe pbd-plug uuid=191ec47b-4a89-c25b-c81f-84d8394de235
Error code: SR_BACKEND_FAILURE_47
Error parameters: , The SR is not available [opterr=The SR is not available [opterr=no such volume group: VG_XenStorage-04bbfb22-fc69-5c9d-a7df-600ca1153e38]],
[root@xs4 rc.d]#

 

I've been working on this all day and I'm out of ideas...

Link to comment
  • 0

So I did an emergency network reset with xsconsole and that resolved the grayed out "ok" button. I'm now able add to add my storage IPs. I'm not able to successfully perform an iscsiadm discovery, as seen below:

 

[root@xs4 ~]# iscsiadm -m discovery -t sendtargets -p 172.32.235.101
iscsiadm: connect to 172.32.235.101 timed out
iscsiadm: connect to 172.32.235.101 timed out
iscsiadm: connect to 172.32.235.101 timed out
iscsiadm: connect to 172.32.235.101 timed out
iscsiadm: connect to 172.32.235.101 timed out
iscsiadm: connect to 172.32.235.101 timed out
iscsiadm: connection login retries (reopen_max) 5 exceeded
iscsiadm: Could not perform SendTargets discovery: encountered connection failure

 

I tried to repair via Xen Center as well as re-scan.

 

Any other suggestions?

Link to comment
  • 0

IQN is the same. I tried to plug the device and received the following:

 

xs4 ~]# xe pbd-plug uuid=191ec47b-4a89-c25b-c81f-84d8394de235
Error code: SR_BACKEND_FAILURE_47
Error parameters: , The SR is not available [opterr=The SR is not available [opterr=no such volume group: VG_XenStorage-04bbfb22-fc69-5c9d-a7df-600ca1153e38]],
[root@xs4 ~]#

 

lvdisplay returned nothing as well, ping also failed.

Link to comment
  • 0

Then your network isn't recognizing the device at all. I trust your storage device cannot see the host, either? At this point comparing a hot that works to the one that doesn't is probably the best way to try to pin down the issues.

The LVM is not going to be active on the host until it can see the storage and connect to it, hence the 47 error. Not even being able to ping isn't a good sign. Any issues perhaps with firewalls?

 

-=Tobias

Link to comment
  • 0

I have resolved the issue. As you mentioned Tobias, the network wasn't seeing the device and I was stumped as to why. At first I wasn't able to configure the network because the "Ok" button was grayed out. I resolved this by performing the emergency-network-reset". Apparently just a reboot doesn't quite cut it but the reset did. After the reset I was able to add the storage IPs. except I didn't add them in the correct order, see below:

 

root@xs4 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.214.16.1     0.0.0.0         UG    0      0        0 xenbr0
10.214.16.0     0.0.0.0         255.255.255.0   U     0      0        0 xenbr0
172.32.236.0    0.0.0.0         255.255.255.0   U     0      0        0 xapi1
172.32.235.0    0.0.0.0         255.255.255.0   U     0      0        0 xapi2
[root@xs4 ~]#iscsiadm -m discovery -t sendtargets -p 172.32.235.101

 

I kept getting connection timeouts

 

After comparing this with one of my other hosts I realized that I needed to reverse the storage IPs.

 

root@xs4 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.214.16.1     0.0.0.0         UG    0      0        0 xenbr0
10.214.16.0     0.0.0.0         255.255.255.0   U     0      0        0 xenbr0
172.32.235.0    0.0.0.0         255.255.255.0   U     0      0        0 xapi1
172.32.236.0    0.0.0.0         255.255.255.0   U     0      0        0 xapi2

 

Once I did I was able to connect to the storage array

 

[root@xs4 ~]# iscsiadm -m discovery -t sendtargets -p 172.32.235.101
172.32.235.100:3260,1 iqn.2010-06.com.purestorage:flasharray.3c9bf12bd7c53cf3
172.32.235.102:3260,1 iqn.2010-06.com.purestorage:flasharray.3c9bf12bd7c53cf3
172.32.235.104:3260,1 iqn.2010-06.com.purestorage:flasharray.3c9bf12bd7c53cf3
172.32.235.106:3260,1 iqn.2010-06.com.purestorage:flasharray.3c9bf12bd7c53cf3

 

After that I was able to "repair storage repository" via Xen Center and I'm back in business.

 

Thanks for your help!

 

Link to comment
  • 0

Great to hear and super that things are working once again! With multipathing, the order isn't always properly preserved when you do a reconstruction which is why when you add a new host to a pool it should be configured with just a single primary management interface to ensure the connection is activated properly to match that of the pool.

 

-=Tobias

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...