Re: iSCSI Multiqueue
> On Jan 24, 2020, at 3:43 AM, Vladislav Bolkhovitin wrote: > > >> ... > > From my old iSCSI target development days, MS is fundamentally not > friendly to multi-queue, because it requires by the iSCSI spec to > preserve order of commands inside the session across multiple > connections. Commands serialization => shared lock or atomic => no > multi-queue benefits. > > Hence, usage of MS for multi-queue would be beneficial only if to drop > (aka violate) this iSCSI spec requirement. > > Just a small reminder. I have not looked in the updated iSCSI spec for a > while, but don't remember this requirement was anyhow eased there. > > In any case, multiple iSCSI sessions per block level "session" would > always be another alternative that would require virtually zero changes > in open-iscsi and in-kernel iSCSI driver[1] as opposed to complex > changes required to start supporting MS in it as well as in many iSCSI > targets around that currently do not[2]. If I would be working on iSCSI > MQ, I would consider this as the first and MUCH more preferable option. > > Vlad > > 1. Most likely, completely zero. > 2. Where requirement to preserve commands order would similarly kill all > the MQ performance benefits. My reaction, from a similar background, matches yours. iSCSI makes things quite hard by requiring ordering across the connections that make up a session. That discourages implementation of multi-connection support in targets (it's optional). In some cases, it entirely rules it out; for example, in the EqualLogic storage arrays it would be pretty useless to support multi-connection since the connections could not be spread over multiple arrays, and for that reason we ruled out that feature. By contrast, MPIO (several independent sessions used by the storage stack as a wider and/or more fault tolerant pipe to the storage) requires essentially no work at the target and gives at least as much benefit as MCS for a lot less work. paul -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/open-iscsi/B7F71EA1-6E05-43A5-BADA-4F29550424D6%40comcast.net.
Re: iSCSI Multiqueue
On 1/24/20 12:43 AM, Vladislav Bolkhovitin wrote: > > On 1/23/20 1:51 PM, The Lee-Man wrote: >> On Wednesday, January 15, 2020 at 7:16:48 AM UTC-8, Bobby wrote: >> >> >> Hi all, >> >> I have a question regarding multi-queue in iSCSI. AFAIK, *scsi-mq* >> has been functional in kernel since kernel 3.17. Because earlier, >> the block layer was updated to multi-queue *blk-mq* from >> single-queue. So the current kernel has full-fledged *multi-queues*. >> >> The question is: >> >> How an iSCSI initiator uses multi-queue? Does it mean having >> multiple connections? I would like >> to see where exactly that is achieved in the code, if someone can >> please me give me a hint. Thanks in advance :) >> >> Regards >> >> >> open-iscsi does not use multi-queue specifically, though all of the >> block layer is now converted to using multi-queue. If I understand >> correctly, there is no more single-queue, but there is glue that allows >> existing single-queue drivers to continue on, mapping their use to >> multi-queue. (Someone please correct me if I'm wrong.) >> >> The only time multi-queue might be useful for open-iscsi to use would be >> for MCS -- multiple connections per session. But the implementation of >> multi-queue makes using it for MCS problematic. Because each queue is on >> a different CPU, open-iscsi would have to coordinate the multiple >> connections across multiple CPUs, making things like ensuring correct >> sequence numbers difficult. >> >> Hope that helps. I _believe_ there is still an effort to map open-iscsi >> MCS to multi-queue, but nobody has tried to actually do it yet that I >> know of. The goal, of course, is better throughput using MCS. > > From my old iSCSI target development days, MS is fundamentally not > friendly to multi-queue, because it requires by the iSCSI spec to > preserve order of commands inside the session across multiple > connections. Commands serialization => shared lock or atomic => no > multi-queue benefits. > > Hence, usage of MS for multi-queue would be beneficial only if to drop > (aka violate) this iSCSI spec requirement. > > Just a small reminder. I have not looked in the updated iSCSI spec for a > while, but don't remember this requirement was anyhow eased there. > > In any case, multiple iSCSI sessions per block level "session" would > always be another alternative that would require virtually zero changes > in open-iscsi and in-kernel iSCSI driver[1] as opposed to complex > changes required to start supporting MS in it as well as in many iSCSI > targets around that currently do not[2]. If I would be working on iSCSI > MQ, I would consider this as the first and MUCH more preferable option. > > Vlad > > 1. Most likely, completely zero. > 2. Where requirement to preserve commands order would similarly kill all > the MQ performance benefits. Oops, 'MCS' must be everywhere instead of 'MS'. Something "corrected" this "for me" behind my back. Sorry, Vlad -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/open-iscsi/846665ef-d238-8abd-8d1b-72e494af9dd1%40vlnb.net.
Re: iSCSI Multiqueue
On Wednesday, January 15, 2020 at 7:16:48 AM UTC-8, Bobby wrote: > > > Hi all, > > I have a question regarding multi-queue in iSCSI. AFAIK, *scsi-mq* has > been functional in kernel since kernel 3.17. Because earlier, > the block layer was updated to multi-queue *blk-mq* from single-queue. So > the current kernel has full-fledged *multi-queues*. > > The question is: > > How an iSCSI initiator uses multi-queue? Does it mean having multiple > connections? I would like > to see where exactly that is achieved in the code, if someone can please > me give me a hint. Thanks in advance :) > > Regards > open-iscsi does not use multi-queue specifically, though all of the block layer is now converted to using multi-queue. If I understand correctly, there is no more single-queue, but there is glue that allows existing single-queue drivers to continue on, mapping their use to multi-queue. (Someone please correct me if I'm wrong.) The only time multi-queue might be useful for open-iscsi to use would be for MCS -- multiple connections per session. But the implementation of multi-queue makes using it for MCS problematic. Because each queue is on a different CPU, open-iscsi would have to coordinate the multiple connections across multiple CPUs, making things like ensuring correct sequence numbers difficult. Hope that helps. I _believe_ there is still an effort to map open-iscsi MCS to multi-queue, but nobody has tried to actually do it yet that I know of. The goal, of course, is better throughput using MCS. -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/open-iscsi/8f236c4a-a207-4a0e-8dff-ad14a74e57dc%40googlegroups.com.
iSCSI Multiqueue
Hi all, I have a question regarding multi-queue in iSCSI. AFAIK, *scsi-mq* has been functional in kernel since kernel 3.17. Because earlier, the block layer was updated to multi-queue *blk-mq* from single-queue. So the current kernel has full-fledged *multi-queues*. The question is: How an iSCSI initiator uses multi-queue? Does it mean having multiple connections? I would like to see where exactly that is achieved in the code, if someone can please me give me a hint. Thanks in advance :) Regards -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/open-iscsi/fa6d0ccd-0fdc-4fa2-bdee-7a8cf22f18b1%40googlegroups.com.
Re: May I do some contributions for iSCSI multiqueue
Hi, This is my new account based on my SUSE email, I think I should use this in future. Thanks Zhu Lingshan 在 2015年6月26日星期五 UTC+8下午2:31:26，LSZhu写道： Hi, I have been working for iSCSI in SUSE for half a year, I have some basic knowledge of iSCSI. I did some debug and performance analyze work before.I am quite interested in iSCSI-mq, I am not a expert here, but may I do some contributions for iSCSI-mq? If you need me in somewhere, please let me know. In my view, there seems such works need to be done: (1) open-iscsi should simulate a multi-queue block device on the initiator side, I mean, /dev/sdc, sdd which simulated by open-iscsi should be multi-queue, just like we want a multi-queue hardware device as a backstore. (2) I/O scheduler is needed in the block layer for multi-queue, for the simulated device mentioned above. (3) open-iscsi should establish more than one connections to the target in a session, and a I/O scheduler is needed here. (4) some performance improve work like how to manage multi-queue threads on multiple CPU cores, how to reduce latency, how to create a better pipeline. I have heard that on the target LIO side, multi-queue work is done. If I am wrong somewhere, please tell me, I would appreciate for that. I know you have did a lot of work for multi-queue, so if you need me somewhere, or I can help in some work, please let me know. Thanks BR Zhu Lingshan -- You received this message because you are subscribed to the Google Groups open-iscsi group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
Re: May I do some contributions for iSCSI multiqueue
On 06/25/2015 11:31 PM, LSZhu wrote: I have been working for iSCSI in SUSE for half a year, I have some basic knowledge of iSCSI. I did some debug and performance analyze work before.I am quite interested in iSCSI-mq, I am not a expert here, but may I do some contributions for iSCSI-mq? If you need me in somewhere, please let me know. In my view, there seems such works need to be done: (1) open-iscsi should simulate a multi-queue block device on the initiator side, I mean, /dev/sdc, sdd which simulated by open-iscsi should be multi-queue, just like we want a multi-queue hardware device as a backstore. (2) I/O scheduler is needed in the block layer for multi-queue, for the simulated device mentioned above. (3) open-iscsi should establish more than one connections to the target in a session, and a I/O scheduler is needed here. (4) some performance improve work like how to manage multi-queue threads on multiple CPU cores, how to reduce latency, how to create a better pipeline. I have heard that on the target LIO side, multi-queue work is done. If I am wrong somewhere, please tell me, I would appreciate for that. I know you have did a lot of work for multi-queue, so if you need me somewhere, or I can help in some work, please let me know. Hello Zhu, The most recent discussion about this topic I am aware of can be found at http://thread.gmane.org/gmane.linux.scsi/98199. Please note that I'm not an iSCSI expert. Bart. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
Re: May I do some contributions for iSCSI multiqueue
Hi, This is my new account based on my SUSE email, I would use this one in future. Thanks BR Zhu Lingshan 在 2015年6月26日星期五 UTC+8下午2:31:26，LSZhu写道： Hi, I have been working for iSCSI in SUSE for half a year, I have some basic knowledge of iSCSI. I did some debug and performance analyze work before.I am quite interested in iSCSI-mq, I am not a expert here, but may I do some contributions for iSCSI-mq? If you need me in somewhere, please let me know. In my view, there seems such works need to be done: (1) open-iscsi should simulate a multi-queue block device on the initiator side, I mean, /dev/sdc, sdd which simulated by open-iscsi should be multi-queue, just like we want a multi-queue hardware device as a backstore. (2) I/O scheduler is needed in the block layer for multi-queue, for the simulated device mentioned above. (3) open-iscsi should establish more than one connections to the target in a session, and a I/O scheduler is needed here. (4) some performance improve work like how to manage multi-queue threads on multiple CPU cores, how to reduce latency, how to create a better pipeline. I have heard that on the target LIO side, multi-queue work is done. If I am wrong somewhere, please tell me, I would appreciate for that. I know you have did a lot of work for multi-queue, so if you need me somewhere, or I can help in some work, please let me know. Thanks BR Zhu Lingshan -- You received this message because you are subscribed to the Google Groups open-iscsi group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
May I do some contributions for iSCSI multiqueue
Hi, I have been working for iSCSI in SUSE for half a year, I have some basic knowledge of iSCSI. I did some debug and performance analyze work before.I am quite interested in iSCSI-mq, I am not a expert here, but may I do some contributions for iSCSI-mq? If you need me in somewhere, please let me know. In my view, there seems such works need to be done: (1) open-iscsi should simulate a multi-queue block device on the initiator side, I mean, /dev/sdc, sdd which simulated by open-iscsi should be multi-queue, just like we want a multi-queue hardware device as a backstore. (2) I/O scheduler is needed in the block layer for multi-queue, for the simulated device mentioned above. (3) open-iscsi should establish more than one connections to the target in a session, and a I/O scheduler is needed here. (4) some performance improve work like how to manage multi-queue threads on multiple CPU cores, how to reduce latency, how to create a better pipeline. I have heard that on the target LIO side, multi-queue work is done. If I am wrong somewhere, please tell me, I would appreciate for that. I know you have did a lot of work for multi-queue, so if you need me somewhere, or I can help in some work, please let me know. Thanks BR Zhu Lingshan -- You received this message because you are subscribed to the Google Groups open-iscsi group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
