RE: iscsid issue in container env
It looks like you were right :( iscsid.socket gets activated right after the yum install. I will re-run our tests to confirm that disabling iscsid.socket fixes the container's iscsid process and does not break anything else on the host. Thank you very much for your help Serguei -Original Message- From: Chris Leech [mailto:cle...@redhat.com] Sent: Thursday, January 26, 2017 3:24 PM To: open-iscsi@googlegroups.com; Serguei Bezverkhi (sbezverk) Subject: Re: iscsid issue in container env - Original Message - > For sure there is no any other iscsd process running. What I could > see, that even if iscsi-initiator-utils are just installed on the > host, the process is not active, it still does not work. To make it > work it needs to be yum removed.. > Is there anything other than just service starting scripts are created > during the installation? Maybe there are some system shares which are > not the same and conflict one with the other? Anything you could think of > please. Could be systemd socket activation enabled by default on the host? That would also block the IPC socket. Check the status of the iscsid.socket unit and disable it if needed? > Thank you > > Serguei > > -Original Message- > From: Chris Leech [mailto:cle...@redhat.com] > Sent: Thursday, January 26, 2017 2:00 PM > To: open-iscsi@googlegroups.com; Serguei Bezverkhi (sbezverk) > > Subject: Re: iscsid issue in container env > > - Original Message - > > Hi Chris, > > > > Thank you for your reply. Our iscsid container runs in host > > namespace as well it has visibility to host pids running, and host > > networking. > > So in our case container looks more like a nice wrapper for iscsd > > process. Iscsid process has full privileged access to the host's > > components Please let me know if it changes the problem definition > > from your perspective. > > Well, if it's truly running in the root network namespace (not bridged > to > host) then it's not what I was thinking. I know that iscsid runs > under docker with net=host. Could there be another iscsid process > running outside of the container? That would block binding to the > unix ipc socket, and seems to be what's mentioned in the issue linked to in > your launchpad bug. > Right now there can be only one iscsid, and it must be in the > root/default/global netns (what's the right terminology here?). > > > You are right about more attention as it is not just docker, but > > openstack and kubernetes which all use docker images are interested > > to have working iscsi solution. > > > > Thank you > > > > Serguei > > > > -Original Message- > > From: Chris Leech [mailto:cle...@redhat.com] > > Sent: Thursday, January 26, 2017 11:27 AM > > To: open-iscsi@googlegroups.com; Serguei Bezverkhi (sbezverk) > > > > Subject: Re: iscsid issue in container env > > > > Hi, > > > > The issue with containers has been that the interface between iscsid > > and the kernel is not aware of network namespaces, and will not work > > if not using the root/host network. I have a partial fix (kernel > > patch set) that needs finishing, making the iSCSI netlink protocol > > namespace aware and limiting visibility of iSCSI sysfs objects by > > network namespace. The flash-node sysfs interface needs migrating > > from bus to class code to take advantage of the namespace filtering, > > and this would only work for iscsi_tcp and maybe iser without > > additional work to migrate iscsi_host devices between network namespaces. > > > > This seems to be getting more attention lately, I'll see if I can't > > make it more of a priority. > > > > Chris > > > > - Original Message - > > > Hello, > > > > > > As per Mike's suggestion I am posting my issue at this mailer. > > > > > > We (kolla project) hit a very very strange problem. Kolla builds > > > and runs openstack as set of containers. One of the containers > > > requires iscsid which we install onto the container image. What we > > > have discovered and so far no explanation was found, if by some > > > reason iscsi-initiator-utils were installed on the host where the > > > container runs, without even activating them, iscsd dame fails to > > > start in the container. As soon as we yum remove iscsi-initiator > > > from the host, iscsid in the container is happy and runs perfectly. > > > We even have a bug filed to track this issue: > > > https://bu
Re: iscsid issue in container env
- Original Message - > For sure there is no any other iscsd process running. What I could see, that > even if iscsi-initiator-utils are just installed on the host, the process is > not active, it still does not work. To make it work it needs to be yum > removed.. > Is there anything other than just service starting scripts are created during > the installation? Maybe there are some system shares which are not the same > and conflict one with the other? Anything you could think of please. Could be systemd socket activation enabled by default on the host? That would also block the IPC socket. Check the status of the iscsid.socket unit and disable it if needed? > Thank you > > Serguei > > -Original Message- > From: Chris Leech [mailto:cle...@redhat.com] > Sent: Thursday, January 26, 2017 2:00 PM > To: open-iscsi@googlegroups.com; Serguei Bezverkhi (sbezverk) > > Subject: Re: iscsid issue in container env > > - Original Message - > > Hi Chris, > > > > Thank you for your reply. Our iscsid container runs in host namespace > > as well it has visibility to host pids running, and host networking. > > So in our case container looks more like a nice wrapper for iscsd > > process. Iscsid process has full privileged access to the host's > > components Please let me know if it changes the problem definition from > > your perspective. > > Well, if it's truly running in the root network namespace (not bridged to > host) then it's not what I was thinking. I know that iscsid runs under > docker with net=host. Could there be another iscsid process running outside > of the container? That would block binding to the unix ipc socket, and > seems to be what's mentioned in the issue linked to in your launchpad bug. > Right now there can be only one iscsid, and it must be in the > root/default/global netns (what's the right terminology here?). > > > You are right about more attention as it is not just docker, but > > openstack and kubernetes which all use docker images are interested to > > have working iscsi solution. > > > > Thank you > > > > Serguei > > > > -Original Message----- > > From: Chris Leech [mailto:cle...@redhat.com] > > Sent: Thursday, January 26, 2017 11:27 AM > > To: open-iscsi@googlegroups.com; Serguei Bezverkhi (sbezverk) > > > > Subject: Re: iscsid issue in container env > > > > Hi, > > > > The issue with containers has been that the interface between iscsid > > and the kernel is not aware of network namespaces, and will not work > > if not using the root/host network. I have a partial fix (kernel > > patch set) that needs finishing, making the iSCSI netlink protocol > > namespace aware and limiting visibility of iSCSI sysfs objects by > > network namespace. The flash-node sysfs interface needs migrating > > from bus to class code to take advantage of the namespace filtering, > > and this would only work for iscsi_tcp and maybe iser without > > additional work to migrate iscsi_host devices between network namespaces. > > > > This seems to be getting more attention lately, I'll see if I can't > > make it more of a priority. > > > > Chris > > > > - Original Message - > > > Hello, > > > > > > As per Mike's suggestion I am posting my issue at this mailer. > > > > > > We (kolla project) hit a very very strange problem. Kolla builds > > > and runs openstack as set of containers. One of the containers > > > requires iscsid which we install onto the container image. What we > > > have discovered and so far no explanation was found, if by some > > > reason iscsi-initiator-utils were installed on the host where the > > > container runs, without even activating them, iscsd dame fails to > > > start in the container. As soon as we yum remove iscsi-initiator > > > from the host, iscsid in the container is happy and runs perfectly. > > > We even have a bug filed to track this issue: > > > https://bugs.launchpad.net/kolla/+bug/1640304 > > > > > > Could you suggest some ideas for further troubleshooting, as > > > unfortunately we ran out of ours. I can easily reproduce the issue > > > if any additional information could be useful. > > > > > > Thank you > > > > > > Serguei > > > > > > > -- > > You received this message because you are subscribed to the Google > > Groups "open-iscsi" group. > > To unsubscribe from this group and stop receivi
RE: iscsid issue in container env
For sure there is no any other iscsd process running. What I could see, that even if iscsi-initiator-utils are just installed on the host, the process is not active, it still does not work. To make it work it needs to be yum removed.. Is there anything other than just service starting scripts are created during the installation? Maybe there are some system shares which are not the same and conflict one with the other? Anything you could think of please. Thank you Serguei -Original Message- From: Chris Leech [mailto:cle...@redhat.com] Sent: Thursday, January 26, 2017 2:00 PM To: open-iscsi@googlegroups.com; Serguei Bezverkhi (sbezverk) Subject: Re: iscsid issue in container env - Original Message - > Hi Chris, > > Thank you for your reply. Our iscsid container runs in host namespace > as well it has visibility to host pids running, and host networking. > So in our case container looks more like a nice wrapper for iscsd > process. Iscsid process has full privileged access to the host's > components Please let me know if it changes the problem definition from your > perspective. Well, if it's truly running in the root network namespace (not bridged to host) then it's not what I was thinking. I know that iscsid runs under docker with net=host. Could there be another iscsid process running outside of the container? That would block binding to the unix ipc socket, and seems to be what's mentioned in the issue linked to in your launchpad bug. Right now there can be only one iscsid, and it must be in the root/default/global netns (what's the right terminology here?). > You are right about more attention as it is not just docker, but > openstack and kubernetes which all use docker images are interested to > have working iscsi solution. > > Thank you > > Serguei > > -Original Message- > From: Chris Leech [mailto:cle...@redhat.com] > Sent: Thursday, January 26, 2017 11:27 AM > To: open-iscsi@googlegroups.com; Serguei Bezverkhi (sbezverk) > > Subject: Re: iscsid issue in container env > > Hi, > > The issue with containers has been that the interface between iscsid > and the kernel is not aware of network namespaces, and will not work > if not using the root/host network. I have a partial fix (kernel > patch set) that needs finishing, making the iSCSI netlink protocol > namespace aware and limiting visibility of iSCSI sysfs objects by > network namespace. The flash-node sysfs interface needs migrating > from bus to class code to take advantage of the namespace filtering, > and this would only work for iscsi_tcp and maybe iser without > additional work to migrate iscsi_host devices between network namespaces. > > This seems to be getting more attention lately, I'll see if I can't > make it more of a priority. > > Chris > > - Original Message - > > Hello, > > > > As per Mike's suggestion I am posting my issue at this mailer. > > > > We (kolla project) hit a very very strange problem. Kolla builds > > and runs openstack as set of containers. One of the containers > > requires iscsid which we install onto the container image. What we > > have discovered and so far no explanation was found, if by some > > reason iscsi-initiator-utils were installed on the host where the > > container runs, without even activating them, iscsd dame fails to > > start in the container. As soon as we yum remove iscsi-initiator > > from the host, iscsid in the container is happy and runs perfectly. > > We even have a bug filed to track this issue: > > https://bugs.launchpad.net/kolla/+bug/1640304 > > > > Could you suggest some ideas for further troubleshooting, as > > unfortunately we ran out of ours. I can easily reproduce the issue > > if any additional information could be useful. > > > > Thank you > > > > Serguei > > > > -- > You received this message because you are subscribed to the Google > Groups "open-iscsi" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to open-iscsi+unsubscr...@googlegroups.com. > To post to this group, send email to open-iscsi@googlegroups.com. > Visit this group at https://groups.google.com/group/open-iscsi. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at https://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
Re: iscsid issue in container env
- Original Message - > Hi Chris, > > Thank you for your reply. Our iscsid container runs in host namespace as well > it has visibility to host pids running, and host networking. So in our case > container looks more like a nice wrapper for iscsd process. Iscsid process > has full privileged access to the host's components Please let me know if it > changes the problem definition from your perspective. Well, if it's truly running in the root network namespace (not bridged to host) then it's not what I was thinking. I know that iscsid runs under docker with net=host. Could there be another iscsid process running outside of the container? That would block binding to the unix ipc socket, and seems to be what's mentioned in the issue linked to in your launchpad bug. Right now there can be only one iscsid, and it must be in the root/default/global netns (what's the right terminology here?). > You are right about more attention as it is not just docker, but openstack > and kubernetes which all use docker images are interested to have working > iscsi solution. > > Thank you > > Serguei > > -Original Message- > From: Chris Leech [mailto:cle...@redhat.com] > Sent: Thursday, January 26, 2017 11:27 AM > To: open-iscsi@googlegroups.com; Serguei Bezverkhi (sbezverk) > > Subject: Re: iscsid issue in container env > > Hi, > > The issue with containers has been that the interface between iscsid and the > kernel is not aware of network namespaces, and will not work if not using > the root/host network. I have a partial fix (kernel patch set) that needs > finishing, making the iSCSI netlink protocol namespace aware and limiting > visibility of iSCSI sysfs objects by network namespace. The flash-node > sysfs interface needs migrating from bus to class code to take advantage of > the namespace filtering, and this would only work for iscsi_tcp and maybe > iser without additional work to migrate iscsi_host devices between network > namespaces. > > This seems to be getting more attention lately, I'll see if I can't make it > more of a priority. > > Chris > > - Original Message - > > Hello, > > > > As per Mike's suggestion I am posting my issue at this mailer. > > > > We (kolla project) hit a very very strange problem. Kolla builds and > > runs openstack as set of containers. One of the containers requires > > iscsid which we install onto the container image. What we have > > discovered and so far no explanation was found, if by some reason > > iscsi-initiator-utils were installed on the host where the container > > runs, without even activating them, iscsd dame fails to start in the > > container. As soon as we yum remove iscsi-initiator from the host, > > iscsid in the container is happy and runs perfectly. We even have a > > bug filed to track this issue: > > https://bugs.launchpad.net/kolla/+bug/1640304 > > > > Could you suggest some ideas for further troubleshooting, as > > unfortunately we ran out of ours. I can easily reproduce the issue if > > any additional information could be useful. > > > > Thank you > > > > Serguei > > > > -- > You received this message because you are subscribed to the Google Groups > "open-iscsi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to open-iscsi+unsubscr...@googlegroups.com. > To post to this group, send email to open-iscsi@googlegroups.com. > Visit this group at https://groups.google.com/group/open-iscsi. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at https://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
RE: iscsid issue in container env
Hi Chris, Thank you for your reply. Our iscsid container runs in host namespace as well it has visibility to host pids running, and host networking. So in our case container looks more like a nice wrapper for iscsd process. Iscsid process has full privileged access to the host's components Please let me know if it changes the problem definition from your perspective. You are right about more attention as it is not just docker, but openstack and kubernetes which all use docker images are interested to have working iscsi solution. Thank you Serguei -Original Message- From: Chris Leech [mailto:cle...@redhat.com] Sent: Thursday, January 26, 2017 11:27 AM To: open-iscsi@googlegroups.com; Serguei Bezverkhi (sbezverk) Subject: Re: iscsid issue in container env Hi, The issue with containers has been that the interface between iscsid and the kernel is not aware of network namespaces, and will not work if not using the root/host network. I have a partial fix (kernel patch set) that needs finishing, making the iSCSI netlink protocol namespace aware and limiting visibility of iSCSI sysfs objects by network namespace. The flash-node sysfs interface needs migrating from bus to class code to take advantage of the namespace filtering, and this would only work for iscsi_tcp and maybe iser without additional work to migrate iscsi_host devices between network namespaces. This seems to be getting more attention lately, I'll see if I can't make it more of a priority. Chris - Original Message - > Hello, > > As per Mike's suggestion I am posting my issue at this mailer. > > We (kolla project) hit a very very strange problem. Kolla builds and > runs openstack as set of containers. One of the containers requires > iscsid which we install onto the container image. What we have > discovered and so far no explanation was found, if by some reason > iscsi-initiator-utils were installed on the host where the container > runs, without even activating them, iscsd dame fails to start in the > container. As soon as we yum remove iscsi-initiator from the host, > iscsid in the container is happy and runs perfectly. We even have a > bug filed to track this issue: > https://bugs.launchpad.net/kolla/+bug/1640304 > > Could you suggest some ideas for further troubleshooting, as > unfortunately we ran out of ours. I can easily reproduce the issue if > any additional information could be useful. > > Thank you > > Serguei > -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at https://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
Re: iscsid issue in container env
Hi, The issue with containers has been that the interface between iscsid and the kernel is not aware of network namespaces, and will not work if not using the root/host network. I have a partial fix (kernel patch set) that needs finishing, making the iSCSI netlink protocol namespace aware and limiting visibility of iSCSI sysfs objects by network namespace. The flash-node sysfs interface needs migrating from bus to class code to take advantage of the namespace filtering, and this would only work for iscsi_tcp and maybe iser without additional work to migrate iscsi_host devices between network namespaces. This seems to be getting more attention lately, I'll see if I can't make it more of a priority. Chris - Original Message - > Hello, > > As per Mike's suggestion I am posting my issue at this mailer. > > We (kolla project) hit a very very strange problem. Kolla builds and > runs openstack as set of containers. One of the containers requires > iscsid which we install onto the container image. What we have > discovered and so far no explanation was found, if by some reason > iscsi-initiator-utils were installed on the host where the container > runs, without even activating them, iscsd dame fails to start in the > container. As soon as we yum remove iscsi-initiator from the host, > iscsid in the container is happy and runs perfectly. We even have a > bug filed to track this issue: > https://bugs.launchpad.net/kolla/+bug/1640304 > > Could you suggest some ideas for further troubleshooting, as > unfortunately we ran out of ours. I can easily reproduce the issue if > any additional information could be useful. > > Thank you > > Serguei > -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at https://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
iscsid issue in container env
Hello, As per Mike's suggestion I am posting my issue at this mailer. We (kolla project) hit a very very strange problem. Kolla builds and runs openstack as set of containers. One of the containers requires iscsid which we install onto the container image. What we have discovered and so far no explanation was found, if by some reason iscsi-initiator-utils were installed on the host where the container runs, without even activating them, iscsd dame fails to start in the container. As soon as we yum remove iscsi-initiator from the host, iscsid in the container is happy and runs perfectly. We even have a bug filed to track this issue: https://bugs.launchpad.net/kolla/+bug/1640304 Could you suggest some ideas for further troubleshooting, as unfortunately we ran out of ours. I can easily reproduce the issue if any additional information could be useful. Thank you Serguei -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at https://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
