[ovirt-users] Host non operationnal due to an iSCSI problem
Hi, I just added a second host but it can't become operational, because it can't access to the iSCSI storage domain. My first question : is it normal or not, is RHEV really able to manage the fact an iSCSI LUN can be accessed from multiple hosts ? I don't find anything in the logs, except vdsm logs that seem to give some clue : https://bpaste.net/show/e06a11d79040 It seems there are LVM related errors, so does it mean it can access to the iSCSI ... ? By the way it's a multipath iSCSI. Any ideas ? ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Host non operationnal due to an iSCSI problem
On 19/07/16 09:52, Alexis HAUSER wrote: Hi, I just added a second host but it can't become operational, because it can't access to the iSCSI storage domain. My first question : is it normal or not, is RHEV really able to manage the fact an iSCSI LUN can be accessed from multiple hosts ? Yes, each VM disk is a Logical Volume on that LUN. Sounds like a possible networking problem. Have you assigned IP addresses to the storage interfaces on this new host? If you're using VLANs, are they set up correctly on your switch ports for the SAN network? Alex -- This message is intended only for the addressee and may contain confidential information. Unless you are that person, you may not disclose its contents or use it in any way and are requested to delete the message along with any attachments and notify us immediately. This email is not intended to, nor should it be taken to, constitute advice. The information provided is correct to our knowledge & belief and must not be used as a substitute for obtaining tax, regulatory, investment, legal or any other appropriate advice. "Transact" is operated by Integrated Financial Arrangements Ltd. 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300. (Registered office: as above; Registered in England and Wales under number: 3727592). Authorised and regulated by the Financial Conduct Authority (entered on the Financial Services Register; no. 190856). ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Host non operationnal due to an iSCSI problem
>Sounds like a possible networking problem. Have you assigned IP >addresses to the storage interfaces on this new host? hum, What do you mean by storage interfaces ? The other host on the same network can access it. > If you're using > VLANs, are they set up correctly on your switch ports for the SAN network? Yes I don't use a server to share the iSCSI storage to the hosts, (I access it directly). Do I need it ? I saw that in the RHEV doc, on first part of the iSCSI section... ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Host non operationnal due to an iSCSI problem
On 19/07/16 10:38, Alexis HAUSER wrote: Sounds like a possible networking problem. Have you assigned IP addresses to the storage interfaces on this new host? hum, What do you mean by storage interfaces ? The other host on the same network can access it. If you're using VLANs, are they set up correctly on your switch ports for the SAN network? Yes I don't use a server to share the iSCSI storage to the hosts, (I access it directly). Do I need it ? I saw that in the RHEV doc, on first part of the iSCSI section... I don't understand. iSCSI is a network storage protocol. What do you mean by "I access it directly"? When you set up the first host with an iSCSI storage domain, you would have had to point it to an IP address, "discover" the LUNs and then attach to them. This sets up the domain. On the second host, to access iSCSI storage you will have to have an interface (defined in "Networks" in oVirt) that can connect to the same IP and port the first host used. Alex -- This message is intended only for the addressee and may contain confidential information. Unless you are that person, you may not disclose its contents or use it in any way and are requested to delete the message along with any attachments and notify us immediately. This email is not intended to, nor should it be taken to, constitute advice. The information provided is correct to our knowledge & belief and must not be used as a substitute for obtaining tax, regulatory, investment, legal or any other appropriate advice. "Transact" is operated by Integrated Financial Arrangements Ltd. 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300. (Registered office: as above; Registered in England and Wales under number: 3727592). Authorised and regulated by the Financial Conduct Authority (entered on the Financial Services Register; no. 190856). ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Host non operationnal due to an iSCSI problem
> I don't understand. iSCSI is a network storage protocol. What do you
> mean by "I access it directly"? When you set up the first host with an
> iSCSI storage domain, you would have had to point it to an IP address,
> "discover" the LUNs and then attach to them. This sets up the domain.
As I explained, I don't use an iSCSI server, that's what I call accessing it
"directly".
Yes, my iSCSI storage is working on my first Host, it has been discovered
successfully, some VM are working on it etc...
The second host can discover it so I don't think it's a network issue.
>From the vdsm logs from second host ("the non working one") it looks like it
>can even see the LVM on it, right ?
Thread-32::DEBUG::2016-07-19 08:41:37,935::lvm::290::Storage.Misc.excCmd::(cmd)
FAILED: = ' Volume group "091e0526-1ff3-4ca3-863c-b911cf69277b" not
found\n Cannot process volume group 091e0526-1ff3-4ca3-863c-b911cf69277b\n';
= 5
> On the second host, to access iSCSI storage you will have to have an
> interface (defined in "Networks" in oVirt) that can connect to the same
> IP and port the first host used.
Yes I have an network interface working on the second host, which is ovirtmgmt.
I can access all other storage correctly from that host without errors. I can
discover the iSCSI.
As it is a multipath iSCSI, does it need to acces one different path for each
host ? I didn't set anything about iSCSI bonding, I use only one single
interface on each host.
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Host non operationnal due to an iSCSI problem
On 19/07/16 11:13, Alexis HAUSER wrote:
I don't understand. iSCSI is a network storage protocol. What do you
mean by "I access it directly"? When you set up the first host with an
iSCSI storage domain, you would have had to point it to an IP address,
"discover" the LUNs and then attach to them. This sets up the domain.
As I explained, I don't use an iSCSI server, that's what I call accessing it
"directly".
Yes, my iSCSI storage is working on my first Host, it has been discovered
successfully, some VM are working on it etc...
The second host can discover it so I don't think it's a network issue.
From the vdsm logs from second host ("the non working one") it looks like it
can even see the LVM on it, right ?
Thread-32::DEBUG::2016-07-19 08:41:37,935::lvm::290::Storage.Misc.excCmd::(cmd) FAILED: =
' Volume group "091e0526-1ff3-4ca3-863c-b911cf69277b" not found\n Cannot process volume
group 091e0526-1ff3-4ca3-863c-b911cf69277b\n'; = 5
It knows there there are volume groups from the database. You are
correct, in that it cannot access the VGs/LVs.
On the second host, to access iSCSI storage you will have to have an
interface (defined in "Networks" in oVirt) that can connect to the same
IP and port the first host used.
Yes I have an network interface working on the second host, which is ovirtmgmt.
I can access all other storage correctly from that host without errors. I can
discover the iSCSI.
As it is a multipath iSCSI, does it need to acces one different path for each
host ? I didn't set anything about iSCSI bonding, I use only one single
interface on each host.
I'm still finding this hard to understand. If you are using iSCSI, you
/are/ using a server (called the "Target" in SCSI speak). Is the iSCSI
storage actually on the first host? How did you actually do the
discovery and assign the LUNs? In the storage domain properties you
should be able to see the IP and port of the Targets, something like
"iqn.2012-02:foo-target1,192.168.10.10,3260", and you need to ensure the
second host can reach that IP and port to be able to see the storage.
Multipath should not make any difference right now, but in order to use
it effectively you should probably set up an iSCSI bond. The requirement
for multipath to work properly is that the two physical interfaces on
the host and initiator are in different IP subnets (and should ideally
travel via different switches but that is not a hard requirement).
If you only have one physical interface on each host, there's not much
point doing multipath, as you don't stand to gain any performance or
resilience.
Cheers
Alex
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.
"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Host non operationnal due to an iSCSI problem
>I'm still finding this hard to understand. If you are using iSCSI, you >/are/ using a server (called the "Target" in SCSI speak). Is the iSCSI >storage actually on the first host? It's a Dell bay (or "storage array", I think that's the correct name in english...) > How did you actually do the >discovery and assign the LUNs? In the storage domain properties you >should be able to see the IP and port of the Targets, something like >"iqn.2012-02:foo-target1,192.168.10.10,3260", and you need to ensure the >second host can reach that IP and port to be able to see the storage. Actually I jutt made a test : authorize access only to the second host (on the Dell bay), it works but only after setting it to maintenance mode and reactivate it. Then authorizing both of the hosts (as initially) make them both working now...It doesn't really makes sense... It is a very strange behavior. Maybe the second host needed to be set in maintenance mode then reactivated ? >If you only have one physical interface on each host, there's not much >point doing multipath, as you don't stand to gain any performance or >resilience. I didn't choose if it was multipath or not, someone only gave me access to this storage, but I understand what you mean. However, I'll certainly add bonding later. Do you have any idea what setting maintenance mode and reactivating does on a host ? Does it restart some services ? I don't really understand what just happened actually... All I know is that it is used for backup, reinstall and update. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Host non operationnal due to an iSCSI problem
On 19/07/16 15:22, Alexis HAUSER wrote: I'm still finding this hard to understand. If you are using iSCSI, you /are/ using a server (called the "Target" in SCSI speak). Is the iSCSI storage actually on the first host? It's a Dell bay (or "storage array", I think that's the correct name in english...) If it offers iSCSI, it is indeed a "Target". It's effectively a server as it's offering up a service on the network! That does explain my confusion though... How did you actually do the discovery and assign the LUNs? In the storage domain properties you should be able to see the IP and port of the Targets, something like "iqn.2012-02:foo-target1,192.168.10.10,3260", and you need to ensure the second host can reach that IP and port to be able to see the storage. Actually I jutt made a test : authorize access only to the second host (on the Dell bay), it works but only after setting it to maintenance mode and reactivate it. Then authorizing both of the hosts (as initially) make them both working now...It doesn't really makes sense... It is a very strange behavior. Maybe the second host needed to be set in maintenance mode then reactivated ? Normally you should not have to do that. It could be that it was not allowed access, and you'd have to leave it a while for the host to retry. At least you have it working now! If you only have one physical interface on each host, there's not much point doing multipath, as you don't stand to gain any performance or resilience. I didn't choose if it was multipath or not, someone only gave me access to this storage, but I understand what you mean. However, I'll certainly add bonding later. Do you have any idea what setting maintenance mode and reactivating does on a host ? Does it restart some services ? I don't really understand what just happened actually... All I know is that it is used for backup, reinstall and update. Maintenance mode will migrate any running VMs off that host and enable you to do some tasks (including the ones you mention) that you can't do when it's running VMs. I believe it stops certain services as well, not sure which ones. It's perfectly safe and routine thing to do in RHEV/oVirt. Cheers Alex -- This message is intended only for the addressee and may contain confidential information. Unless you are that person, you may not disclose its contents or use it in any way and are requested to delete the message along with any attachments and notify us immediately. This email is not intended to, nor should it be taken to, constitute advice. The information provided is correct to our knowledge & belief and must not be used as a substitute for obtaining tax, regulatory, investment, legal or any other appropriate advice. "Transact" is operated by Integrated Financial Arrangements Ltd. 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300. (Registered office: as above; Registered in England and Wales under number: 3727592). Authorised and regulated by the Financial Conduct Authority (entered on the Financial Services Register; no. 190856). ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Host non operationnal due to an iSCSI problem
>Normally you should not have to do that. It could be that it was not >allowed access, and you'd have to leave it a while for the host to retry. It looped on retrying to make it work for 24 hours...It was authorized. But actually when I first add the host, it didn't have authorization. I added authorization, then it looped on retrying without success...Until I did what I told you. >At least you have it working now! Yes thank you, but I'm a bit disappointed by this instability and I'd really like to understand what happened...In case it reproduce again. > > > >> If you only have one physical interface on each host, there's not much >> point doing multipath, as you don't stand to gain any performance or >> resilience. > I didn't choose if it was multipath or not, someone only gave me access to > this storage, but I understand what you mean. However, I'll certainly add > bonding later. > > > Do you have any idea what setting maintenance mode and reactivating does on a > host ? Does it restart some services ? I don't really understand what just > happened actually... > All I know is that it is used for backup, reinstall and update. Maintenance mode will migrate any running VMs off that host and enable you to do some tasks (including the ones you mention) that you can't do when it's running VMs. I believe it stops certain services as well, not sure which ones. It's perfectly safe and routine thing to do in RHEV/oVirt. Thanks, that's certainly what solved the issue then...Or the Dell Bay doing weird random stuff who knows ^^ I've often heard people telling others setting up a host or engine to maintenance mode then reactivate in case of a lot of various issues, it seems to be a method to keep in mind. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
