bind_src_by_address() is disabled?
Hi folks, I spent some time browsing through this forum but I was unable to find an explanation for this comment referring to the disabled bind_src_by_address() function in io.c: *This is not supported for now, because it is not exactly what we want.* *It also turns out that targets will send packets to other interfaces* *causing all types of weird things to happen.* I found several posts from people referring to this specific comment but I did not find an explanation. Is it possible that the author of this comment was referring to the ARP flux issue, which may cause a target to associate the bound IP address with the MAC address from an interface other than the one specified with SO_BINDTODEVICE? If so, I don't see how avoiding the call to bind() solves this problem. I would appreciate a reply from anyone who might know what "weird things" means in this context. Thanks! Tom.III -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
Re: [Lsf-pc] [LSF/MM TOPIC] iSCSI MQ adoption via MCS discussion
On 01/09/2015 12:28 PM, Hannes Reinecke wrote: > On 01/09/2015 07:00 PM, Michael Christie wrote: >> >> On Jan 8, 2015, at 11:03 PM, Nicholas A. Bellinger >> wrote: >> >>> On Thu, 2015-01-08 at 15:22 -0800, James Bottomley wrote: On Thu, 2015-01-08 at 14:57 -0800, Nicholas A. Bellinger wrote: > On Thu, 2015-01-08 at 14:29 -0800, James Bottomley wrote: >> On Thu, 2015-01-08 at 14:16 -0800, Nicholas A. Bellinger wrote: >>> >>> >>> > The point is that a simple session wide counter for command sequence > number assignment is significantly less overhead than all of the > overhead associated with running a full multipath stack atop multiple > sessions. I don't see how that's relevant to issue speed, which was the measure we were using: The layers above are just a hopper. As long as they're loaded, the MQ lower layer can issue at full speed. So as long as the multipath hopper is efficient enough to keep the queues loaded there's no speed degradation. The problem with a sequence point inside the MQ issue layer is that it can cause a stall that reduces the issue speed. so the counter sequence point causes a degraded issue speed over the multipath hopper approach above even if the multipath approach has a higher CPU overhead. Now, if the system is close to 100% cpu already, *then* the multipath overhead will try to take CPU power we don't have and cause a stall, but it's only in the flat out CPU case. > Not to mention that our iSCSI/iSER initiator is already taking a session > wide lock when sending outgoing PDUs, so adding a session wide counter > isn't adding any additional synchronization overhead vs. what's already > in place. I'll leave it up to the iSER people to decide whether they're redoing this as part of the MQ work. >>> >>> Session wide command sequence number synchronization isn't something to >>> be removed as part of the MQ work. It's a iSCSI/iSER protocol >>> requirement. >>> >>> That is, the expected + maximum sequence numbers are returned as part of >>> every response PDU, which the initiator uses to determine when the >>> command sequence number window is open so new non-immediate commands may >>> be sent to the target. >>> >>> So, given some manner of session wide synchronization is required >>> between different contexts for the existing single connection case to >>> update the command sequence number and check when the window opens, it's >>> a fallacy to claim MC/S adds some type of new initiator specific >>> synchronization overhead vs. single connection code. >> >> I think you are assuming we are leaving the iscsi code as it is today. >> >> For the non-MCS mq session per CPU design, we would be allocating and >> binding the session and its resources to specific CPUs. They would only >> be accessed by the threads on that one CPU, so we get our >> serialization/synchronization from that. That is why we are saying we >> do not need something like atomic_t/spin_locks for the sequence number >> handling for this type of implementation. >> > Wouldn't that need to be coordinated with the networking layer? Yes. > Doesn't it do the same thing, matching TX/RX queues to CPUs? Yes. > If so, wouldn't we decrease bandwidth by restricting things to one CPU? We have a session or connection per CPU though, so we end up hitting the same problem you talked about last year where one hctx (iscsi session or connection's socket or nic hw queue) could get overloaded. This is what I meant in my original mail where iscsi would rely on whatever blk/mq load balancers we end up implementing at that layer to balance requests across hctxs. -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
Re: [Lsf-pc] [LSF/MM TOPIC] iSCSI MQ adoption via MCS discussion
On Fri, 2015-01-09 at 19:28 +0100, Hannes Reinecke wrote: [...] > > I think you are assuming we are leaving the iscsi code as it is today. > > > > For the non-MCS mq session per CPU design, we would be allocating and > > binding the session and its resources to specific CPUs. They would only > > be accessed by the threads on that one CPU, so we get our > > serialization/synchronization from that. That is why we are saying we > > do not need something like atomic_t/spin_locks for the sequence number > > handling for this type of implementation. > > > Wouldn't that need to be coordinated with the networking layer? > Doesn't it do the same thing, matching TX/RX queues to CPUs? > If so, wouldn't we decrease bandwidth by restricting things to one CPU? So this is actually one of the fascinating questions on multi-queue. Long ago, when I worked for the NCR OS group and we were bringing up the first SMP systems, we actually found that the SCSI stack went faster when bound to a single CPU. The problem in those days was lock granularity and contention, so single CPU binding eliminated that overhead. However, nowadays with modern multi-tiered caching and huge latencies for cache line bouncing, we're approaching the point where the fineness of our lock granularity is hurting performance, so it's worth re-asking the question of whether just dumping all the lock latency by single CPU binding is a worthwhile exercise. James -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
Re: [Lsf-pc] [LSF/MM TOPIC] iSCSI MQ adoption via MCS discussion
On 01/09/2015 07:00 PM, Michael Christie wrote: > > On Jan 8, 2015, at 11:03 PM, Nicholas A. Bellinger > wrote: > >> On Thu, 2015-01-08 at 15:22 -0800, James Bottomley wrote: >>> On Thu, 2015-01-08 at 14:57 -0800, Nicholas A. Bellinger wrote: On Thu, 2015-01-08 at 14:29 -0800, James Bottomley wrote: > On Thu, 2015-01-08 at 14:16 -0800, Nicholas A. Bellinger wrote: >> >> >> The point is that a simple session wide counter for command sequence number assignment is significantly less overhead than all of the overhead associated with running a full multipath stack atop multiple sessions. >>> >>> I don't see how that's relevant to issue speed, which was the measure we >>> were using: The layers above are just a hopper. As long as they're >>> loaded, the MQ lower layer can issue at full speed. So as long as the >>> multipath hopper is efficient enough to keep the queues loaded there's >>> no speed degradation. >>> >>> The problem with a sequence point inside the MQ issue layer is that it >>> can cause a stall that reduces the issue speed. so the counter sequence >>> point causes a degraded issue speed over the multipath hopper approach >>> above even if the multipath approach has a higher CPU overhead. >>> >>> Now, if the system is close to 100% cpu already, *then* the multipath >>> overhead will try to take CPU power we don't have and cause a stall, but >>> it's only in the flat out CPU case. >>> Not to mention that our iSCSI/iSER initiator is already taking a session wide lock when sending outgoing PDUs, so adding a session wide counter isn't adding any additional synchronization overhead vs. what's already in place. >>> >>> I'll leave it up to the iSER people to decide whether they're redoing >>> this as part of the MQ work. >>> >> >> Session wide command sequence number synchronization isn't something to >> be removed as part of the MQ work. It's a iSCSI/iSER protocol >> requirement. >> >> That is, the expected + maximum sequence numbers are returned as part of >> every response PDU, which the initiator uses to determine when the >> command sequence number window is open so new non-immediate commands may >> be sent to the target. >> >> So, given some manner of session wide synchronization is required >> between different contexts for the existing single connection case to >> update the command sequence number and check when the window opens, it's >> a fallacy to claim MC/S adds some type of new initiator specific >> synchronization overhead vs. single connection code. > > I think you are assuming we are leaving the iscsi code as it is today. > > For the non-MCS mq session per CPU design, we would be allocating and > binding the session and its resources to specific CPUs. They would only > be accessed by the threads on that one CPU, so we get our > serialization/synchronization from that. That is why we are saying we > do not need something like atomic_t/spin_locks for the sequence number > handling for this type of implementation. > Wouldn't that need to be coordinated with the networking layer? Doesn't it do the same thing, matching TX/RX queues to CPUs? If so, wouldn't we decrease bandwidth by restricting things to one CPU? Cheers, Hannes -- Dr. Hannes Reinecke zSeries & Storage h...@suse.de +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg) -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
Re: [Lsf-pc] [LSF/MM TOPIC] iSCSI MQ adoption via MCS discussion
On Jan 8, 2015, at 11:03 PM, Nicholas A. Bellinger wrote: > On Thu, 2015-01-08 at 15:22 -0800, James Bottomley wrote: >> On Thu, 2015-01-08 at 14:57 -0800, Nicholas A. Bellinger wrote: >>> On Thu, 2015-01-08 at 14:29 -0800, James Bottomley wrote: On Thu, 2015-01-08 at 14:16 -0800, Nicholas A. Bellinger wrote: > > > >>> The point is that a simple session wide counter for command sequence >>> number assignment is significantly less overhead than all of the >>> overhead associated with running a full multipath stack atop multiple >>> sessions. >> >> I don't see how that's relevant to issue speed, which was the measure we >> were using: The layers above are just a hopper. As long as they're >> loaded, the MQ lower layer can issue at full speed. So as long as the >> multipath hopper is efficient enough to keep the queues loaded there's >> no speed degradation. >> >> The problem with a sequence point inside the MQ issue layer is that it >> can cause a stall that reduces the issue speed. so the counter sequence >> point causes a degraded issue speed over the multipath hopper approach >> above even if the multipath approach has a higher CPU overhead. >> >> Now, if the system is close to 100% cpu already, *then* the multipath >> overhead will try to take CPU power we don't have and cause a stall, but >> it's only in the flat out CPU case. >> >>> Not to mention that our iSCSI/iSER initiator is already taking a session >>> wide lock when sending outgoing PDUs, so adding a session wide counter >>> isn't adding any additional synchronization overhead vs. what's already >>> in place. >> >> I'll leave it up to the iSER people to decide whether they're redoing >> this as part of the MQ work. >> > > Session wide command sequence number synchronization isn't something to > be removed as part of the MQ work. It's a iSCSI/iSER protocol > requirement. > > That is, the expected + maximum sequence numbers are returned as part of > every response PDU, which the initiator uses to determine when the > command sequence number window is open so new non-immediate commands may > be sent to the target. > > So, given some manner of session wide synchronization is required > between different contexts for the existing single connection case to > update the command sequence number and check when the window opens, it's > a fallacy to claim MC/S adds some type of new initiator specific > synchronization overhead vs. single connection code. I think you are assuming we are leaving the iscsi code as it is today. For the non-MCS mq session per CPU design, we would be allocating and binding the session and its resources to specific CPUs. They would only be accessed by the threads on that one CPU, so we get our serialization/synchronization from that. That is why we are saying we do not need something like atomic_t/spin_locks for the sequence number handling for this type of implementation. If we just tried to do this with the old code where the session could be accessed on multiple CPUs then you are right, we need locks/atomics like how we do in the MCS case. -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
2 iscsid processes started
Hi, I have two questions regarding open-iscsi. 1) I experience the same as exposed by the person in this post : http://forum.proxmox.com/threads/6415-iscsid-it-starts-two-process When I execute iscsid by hand or startup the service, two processes are being spawned. As a result when I stop the service only one process is killed the other stays there. I have to pkill iscsid to clean this process. Only one process has an associated iscsid.pid file, the second judging by pid ordering. The second only has an unix socket openned. If I start iscsid in foreground with -f option, only one process is spawn. Is this a normal behaviour and if yes, how are we suppose to cleanly stop the service ? 2) After having done some iscsiadm operations I don't remember, Each time I enter an iscsiadm command, I get error messages regarding unfound sessions. The normal output follows but I cannot discard these error messages. Here is an example of these error messages : iscsiadm: could not read session targetname: 5 iscsiadm: could not find session info for session11 iscsiadm: could not read session targetname: 5 iscsiadm: could not find session info for session12 iscsiadm: could not read session targetname: 5 iscsiadm: could not find session info for session13 iscsiadm: could not read session targetname: 5 iscsiadm: could not find session info for session14 iscsiadm: could not read session targetname: 5 iscsiadm: could not find session info for session15 iscsiadm: could not read session targetname: 5 iscsiadm: could not find session info for session16 iscsiadm: could not read session targetname: 5 iscsiadm: could not find session info for session24 iscsiadm: could not read session targetname: 5 iscsiadm: could not find session info for session25 iscsiadm: could not read session targetname: 5 iscsiadm: could not find session info for session26 iscsiadm: could not read session targetname: 5 iscsiadm: could not find session info for session30 iscsiadm: could not read session targetname: 5 iscsiadm: could not find session info for session31 The only way I found to remove these messages was to reboot the machine. Is there another way to clean these states ? Thanks for your help. -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
Re: [LSF/MM TOPIC] iSCSI MQ adoption via MCS discussion
On 1/8/2015 4:11 PM, Bart Van Assche wrote: On 01/08/15 14:45, Sagi Grimberg wrote: Actually I started with that approach, but the independent connections under a single session (I-T-Nexus) violates the command ordering requirement. Plus, such a solution is specific to iSER... Hello Sagi, Which command ordering requirement are you referring to ? The Linux storage stack does not guarantee that block layer or SCSI commands will be processed in the same order as these commands have been submitted. I was referring to the iSCSI session requirement. I initially thought of an approach to maintain multiple iSER connections under a single session but pretty soon I realized that preserving commands ordering this way is not feasible. So independent iSER connections means independent iSCSI sessions (each with a single connection). This is indeed another choice, which we are clearly debating on... I'm just wandering if we are not trying to force-fit this model. How would this model look like? We will need to define another entity to track and maintain the sessions and to allocate the scsi_host. Will that be communicated to user-space? How will error recovery look like? Sagi. -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
Re: [LSF/MM TOPIC] iSCSI MQ adoption via MCS discussion
On 1/9/2015 1:26 AM, Mike Christie wrote: I am not sure if we want this to be a deciding factor. I think the session wide lock is something that can be removed in the main IO paths. A lot of what it is used for now is cmd/task related handling like list accesses. When we have the scsi layer alloc/free/manage that, then we can simplify that a lot for iser/bnx2i/cxgb*i since there send path is less complicated than software iscsi. Completely agree. We should assume that other than session-wide command sequence numbers nothing is synced across connections. It is also used for the state check but I think that is overkill. I have a piped patch to remove this completely redundant spin_lock that protects a check on a state that can change right after the spin_unlock. -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
