Re: [Cbe-oss-dev] LIO Target iSCSI/SE PS3-Linux / FC8 builds
k/target/target/../include/iscsi_linux_defs.h:36, > > from /usr/src/linux-iscsi/trunk/target/target/../include/iscsi_crc.h:21, > > from /usr/src/linux-iscsi/trunk/target/target/../common/iscsi_crc.c:35: > include/asm/uaccess.h: In function copy_from_user: > include/asm/uaccess.h:342: warning: integer constant is too large > for unsigned long type > include/asm/uaccess.h:342: warning: integer constant is too large > for unsigned long type > include/asm/uaccess.h:343: warning: integer constant is too large > for unsigned long type > include/asm/uaccess.h:343: warning: integer constant is too large > for unsigned long type > include/asm/uaccess.h: In function copy_to_user: > include/asm/uaccess.h:357: warning: integer constant is too large > for unsigned long type > include/asm/uaccess.h:357: warning: integer constant is too large > for unsigned long type > include/asm/uaccess.h:358: warning: integer constant is too large > for unsigned long type > include/asm/uaccess.h:358: warning: integer constant is too large > for unsigned long type > include/asm/uaccess.h: In function clear_user: > include/asm/uaccess.h:452: warning: integer constant is too large > for unsigned long type > include/asm/uaccess.h:452: warning: integer constant is too large > for unsigned long type > include/asm/uaccess.h:453: warning: integer constant is too large > for unsigned long type > include/asm/uaccess.h:453: warning: integer constant is too large > for unsigned long type > make[3]: *** [/usr/src/linux-iscsi/trunk/target/target/../common/iscsi_crc.o] > Error 1 > make[2]: *** [_module_/usr/src/linux-iscsi/trunk/target/target] Error 2 > make[2]: Leaving directory `/usr/src/ps3-linux' > make[1]: *** [all] Error 2 > make[1]: Leaving directory `/usr/src/linux-iscsi/trunk/target/target' > make: *** [kernel] Error 2 > > thanks > > Marc > > Am Montag 04 Februar 2008 15:17:03 schrieb Nicholas A. Bellinger: > > Greetings all, > > > > I have updated the wiki at: > > > > http://linux-iscsi.org/index.php/Playstation3/iSCSI > > > > and posted the first LIO target builds on the LIO Cluster: > > > > http://linux-iscsi.org/builds/ps3-linux/ > > > > I will adding the documentation for both LIO SE on PS3-Linux, and the > > FC8 upgrade process, as the latter can still be a bit challenging for > > new users. > > > > Here is the info from the README: > > > > LIO Target iSCSI/SE for PS3-Linux v2.9.0.209 > > > > I) Kernel module package > > > > iscsi-target-module-2.6.24-2.9.0.209-1.powerpc.rpm > > > > This modules are built for ppc64 and built with the toolkit for Fedora Core > > 8 ppc. This is gcc (GCC) 4.1.2 20070925 (Red Hat 4.1.2-33) > > > > This module has been tested with the latest ps3-linux.git and built against > > arch/powerpc/configs/ps3_defconfig. To use the BD-ROM, your 2.6.24 kernel > > must contain ps3rom-use-128-max-sector.diff. Please see: > > > > http://git.kernel.org/?p=linux/kernel/git/geoff/ps3-linux.git;a=commit;h=e8 > >2112af66a39d11bcb484de9cfa45f0d214c97f > > > > This module package should work with kernel-2.6.24-20080131.ppc64.rpm from > > CELL-Linux-CL_20080201-ADDON.iso, but the BD-ROM will throw an exception > > without ps3rom-use-128-max-sector.diff. Please see the following link for > > more information about the ADDON CD, and watch for an updated kernel > > package soon.. > > > > http://www.kernel.org/pub/linux/kernel/people/geoff/cell/ > > > > II) Userspace packages > > > > iscsi-target-tools-2.9.0.209-1.ppc.rpm > > sbe-mibs-2.9.0.209-1.ppc.rpm > > > > Note that these are 32-bit and built on Fedora Core 8. > > > > Have fun! > > > > --nab > > > > ___ > > cbe-oss-dev mailing list > > [EMAIL PROTECTED] > > https://ozlabs.org/mailman/listinfo/cbe-oss-dev > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: [Cbe-oss-dev] LIO Target iSCSI/SE PS3-Linux / FC8 builds
Hi Again, Almost forgot, I put the following RPMs that iscsi userspace tools rpms depends for LIO-console.pl and LIO-demo.sh: perl-Curses-1.15-1.fc6.ppc.rpm perl-IO-All-0.33-3.fc5.noarch.rpm perl-IO-String-1.08-1.2.fc5.rf.noarch.rpm http://linux-iscsi.org/builds/ps3-linux/ These should work, let me know if you are missing any when you install iscsi-target-tools. Thanks, --nab On Mon, 2008-02-04 at 07:52 -0800, Nicholas A. Bellinger wrote: > Hi Marc, > > You can generate the kernel RPM with 'make kernel ARCH=powerpc'. > > Also, while module-assistant is supported on debian/ubuntu, > trunk/buildtools/ currently does not support generating kernel module > source rpms. If you want to send a patch, I would be more than happy to > take it. > > --nab > > > On Mon, 2008-02-04 at 16:47 +0100, Marc Dietrich wrote: > > Hi Nicholas, > > > > can you please also upload a src.rpm? I'm having toubles compiling the > > kernel > > code: > > > > # cd target/ ; ./autoconfig --write-to-file ; cat .make_autoconfig ; make > > kernel > > /usr/src/linux-iscsi/trunk/target/.make_autoconfig > > ARCH?=ppc > > AUTO_CFLAGS?= -DHAS_UTS_RELEASE -DUSE_SCSI_H > > -I/lib/modules/2.6.24-06289-g144de36/source/drivers/scsi -DUSE_MSLEEP > > -DUSE_COMPAT_IOCTL -Dscsi_execute_async_address > > -DPYX_ISCSI_VENDOR='"Linux-iSCSI.org"' > > -DIQN_PREFIX='"iqn.2003-01.org.linux-iscsi"' -DLINUX > > -DLINUX_SCATTERLIST_HAS_PAGE -DSVN_VSN=\"209\" > > BASENAME?=FedoraCore-R8-Werewolf.ppc > > DISTRO?=FEDORA > > KERNEL?=26 > > KERNEL_DIR?=/lib/modules/2.6.24-06289-g144de36/build > > KERNEL_INCLUDE_DIR?=/lib/modules/2.6.24-06289-g144de36/source/include > > KERNEL_SOURCE_DIR?=/lib/modules/2.6.24-06289-g144de36/source > > KERNEL_VERSION_INFO?=LINUX_KERNEL_26 > > OSTYPE?=LINUX > > PYX_ISCSI_VERSION?=2.9.0.209 > > RELEASE?=2.6.24-06289-g144de36 > > RELEASES?=ARRAY(0x102052d4) > > RPM_DIR?=/usr/src/redhat > > SNMP?=0 > > SYSTEM?=FedoraCore-R8-Werewolf > > VERSION_IPYXD?=2.9.0.209 > > make -C target clean all > > make[1]: Entering directory `/usr/src/linux-iscsi/trunk/target/target' > > rm -f /usr/src/linux-iscsi/trunk/target/target/../common/iscsi_crc.o > > /usr/src/linux-iscsi/trunk/target/target/../common/iscsi_debug_opcodes.o > > /usr/src/linux-iscsi/trunk/target/target/../common/iscsi_parameters.o > > /usr/src/linux-iscsi/trunk/target/target/../common/iscsi_seq_and_pdu_list.o > > /usr/src/linux-iscsi/trunk/target/target/../common/iscsi_serial.o > > /usr/src/linux-iscsi/trunk/target/target/../common/iscsi_thread_queue.o > > /usr/src/linux-iscsi/trunk/target/target/iscsi_target_datain_values.o > > /usr/src/linux-iscsi/trunk/target/target/iscsi_target_device.o > > /usr/src/linux-iscsi/trunk/target/target/iscsi_target_discovery.o > > /usr/src/linux-iscsi/trunk/target/target/iscsi_target_erl0.o > > /usr/src/linux-iscsi/trunk/target/target/iscsi_target_erl1.o > > /usr/src/linux-iscsi/trunk/target/target/iscsi_target_erl2.o > > /usr/src/linux-iscsi/trunk/target/target/iscsi_target_feature_obj.o > > /usr/src/linux-iscsi/trunk/target/target/iscsi_target_hba.o > > /usr/src/linux-iscsi/trunk/target/target/iscsi_target_info.o > > /usr/src/linux-iscsi/trunk/target/target/iscsi_target_ioctl.o > > /usr/src/linux-iscsi/trunk/target/target/iscsi_target_linux_proc.o > > /usr/src/linux-iscsi/trunk/target/target/iscsi_target_login.o > > /usr/src/linux-iscsi/trunk/target/target/iscsi_target_nego.o > > /usr/src/linux-iscsi/trunk/target/target/iscsi_target_nodeattrib.o > > /usr/src/linux-iscsi/trunk/target/target/iscsi_target_plugin.o > > /usr/src/linux-iscsi/trunk/target/target/iscsi_target_reportluns.o > > /usr/src/linux-iscsi/trunk/target/target/iscsi_target_scdb.o > > /usr/src/linux-iscsi/trunk/target/target/iscsi_target_seobj.o > > /usr/src/linux-iscsi/trunk/target/target/iscsi_target_tmr.o > > /usr/src/linux-iscsi/trunk/target/target/iscsi_target_tpg.o > > /usr/src/linux-iscsi/trunk/target/target/iscsi_target_transport.o > > /usr/src/linux-iscsi/trunk/target/target/iscsi_target_util.o > > /usr/src/linux-iscsi/trunk/target/target/iscsi_target.o > > /usr/src/linux-iscsi/trunk/target/target/div64.o > > /usr/src/linux-iscsi/trunk/target/target/iscsi_target_raid.o > > /usr/src/linux-iscsi/trunk/target/target/iscsi_target_repl.o > > /usr/src/linux-iscsi/trunk/target/target/iscsi_target_iblock.o > > /usr/src/linux-iscsi/trunk/target/target/iscsi_target_pscsi.o > > /usr/src/linux-iscsi/trunk/target/target/iscsi_t
LIO Target iSCSI/SE PS3-Linux / FC8 builds
Greetings all, I have updated the wiki at: http://linux-iscsi.org/index.php/Playstation3/iSCSI and posted the first LIO target builds on the LIO Cluster: http://linux-iscsi.org/builds/ps3-linux/ I will adding the documentation for both LIO SE on PS3-Linux, and the FC8 upgrade process, as the latter can still be a bit challenging for new users. Here is the info from the README: LIO Target iSCSI/SE for PS3-Linux v2.9.0.209 I) Kernel module package iscsi-target-module-2.6.24-2.9.0.209-1.powerpc.rpm This modules are built for ppc64 and built with the toolkit for Fedora Core 8 ppc. This is gcc (GCC) 4.1.2 20070925 (Red Hat 4.1.2-33) This module has been tested with the latest ps3-linux.git and built against arch/powerpc/configs/ps3_defconfig. To use the BD-ROM, your 2.6.24 kernel must contain ps3rom-use-128-max-sector.diff. Please see: http://git.kernel.org/?p=linux/kernel/git/geoff/ps3-linux.git;a=commit;h=e82112af66a39d11bcb484de9cfa45f0d214c97f This module package should work with kernel-2.6.24-20080131.ppc64.rpm from CELL-Linux-CL_20080201-ADDON.iso, but the BD-ROM will throw an exception without ps3rom-use-128-max-sector.diff. Please see the following link for more information about the ADDON CD, and watch for an updated kernel package soon.. http://www.kernel.org/pub/linux/kernel/people/geoff/cell/ II) Userspace packages iscsi-target-tools-2.9.0.209-1.ppc.rpm sbe-mibs-2.9.0.209-1.ppc.rpm Note that these are 32-bit and built on Fedora Core 8. Have fun! --nab --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
LIO-Target Core v3.0.0 imported in k.o git
Greetings all, The LIO-Target Core v3.0.0 tree has been imported from v2.9-STABLE from Linux-iSCSI.org source tree repositories into kernel.org git, and is building w/ v2.6.26-rc3. It can be found at: http://git.kernel.org/?p=linux/kernel/git/nab/lio-core-2.6.git;a=summary I will be continuing the cleanup activies for upstream, which so far has included the removal of legacy unused engine level mirroring/replication bits (as we are using LIO-DRBD or LIO-NR1 w/ MD for this now), and a few LINUX_VERSION_CODE removals. [EMAIL PROTECTED] lio-core]$ wc -l *.c *.h | tail -n 1 57879 total The goal now will be to seperate out the LIO-Target / LIO-Core pieces for moving the latter upstream initially (eg a working passthrough, and then v3.0 LIO-Target using traditional iSCSI, then iWARP and iSER. This will all be going into the roadmap on Linux-iSCSI.org for reference. I invite interested parties to have a look, and please contact me on the LIO-Target devel list, or privately if you would like to get involved looking at some code that are in line with your knowledge/interests/projects. Many thanks for your most valuable of time, --nab --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Open/iSCSI + Logout Response Timeout + replacement_timeout firing
Greetings Mike, Hannes, and all; So, and Jerome and Myself have been pushing VHACS (please see towards production, we have begin to run into a particular issue while during the 'vhacs cluster -I' (eg: cluster initilization) routine when we had a bunch of VHACS server and client clouds active, and need to bring them all down. Before explaining the case, here is a little background: DRBD_TARGET: LIO-Target + DRBD export Cluster RA (Server side VHACS) ISCSI_MOUNT: Open/iSCSI + Ext3 Mount Cluster RA (Client side VHACS) Here is the scenario using 'node.session.timeo.replacement_timeout = 120': We have been experiencing a case where sometimes the DRBD_TARGET RA for a particular VHACS cloud gets shutdown BEFORE the ISCSI_MOUNT RA. This of course means that all of the outstanding I/Os from Open/iSCSI with regard to the mounted filesystem from the DRBD_TARGET that has gone way need to be failed back to SCSI with a DID_ERROR status. The problem is that the failure of the outstanding I/Os does not seem to be occuring in all cases. In particular, a iscsiadm --logout I believe is getting issued, and said logout request failing/timing out because DRBD_TARGET has been released. It is at this point where umount for the ext3 mount and/or sync hangs indefinately. When the problem occurs, it looks like this from the kernel ring buffer: iscsi_deallocate_extra_thread_sets:285: ***OPS*** Stopped 1 thread set(s) (2 total threads). iscsi_deallocate_extra_thread_sets:285: ***OPS*** Stopped 2 thread set(s) (4 total threads). session10: iscsi: session recovery timed out after 120 secs sd 51:0:0:0: scsi: Device offlined - not ready after error recovery sd 51:0:0:0: [sdg] Result: hostbyte=DID_BUS_BUSY driverbyte=DRIVER_OK,SUGGEST_OK end_request: I/O error, dev sdg, sector 0 Buffer I/O error on device sdg, logical block 0 lost page write due to I/O error on sdg I should mention that we are not doing any I/O to said iSCSI LUN via Open/iSCSI other than the filesystem metadata for ext3 umount and SYNCHRONIZE_CACHE CDB during struct scsi_device deregistration. From experience with Core-iSCSI, I know the logout path is tricky wrt exceptions (I spent months on it to handle all cases with Immediate and Non Immediate Logout, as well as doing logouts on the fly from the same connection in MC/S and different connections in MC/S :-) So the question is: I) When a ISCSI_INIT_LOGOUT_REQ is not returned with a ISCSI_TARGET_LOGOUT_RSP and replacement_timeout fires, are all outstanding I/Os for that particular session being completed with an non-recoveryable exception..? Has anyone ever run into this case and/or tested it..? Not being exquisitely fimilar with Open/iSCSI source code, if either of you gents could have a quick look and see if this is indeed broken and hopefully provide a simple fix (I will take care of drinks at the Kernel Summit :-), or if not could point me in the right direction it would be much apperciated. --nab --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Open/iSCSI + Logout Response Timeout + replacement_timeout firing
On Fri, 2008-06-27 at 16:00 -0700, Nicholas A. Bellinger wrote: > Greetings Mike, Hannes, and all; > > So, and Jerome and Myself have been pushing VHACS (please see towards Whoops, forgot to include the VHACS link. :-) http://linux-iscsi.org/index.php/VHACS --nab > production, we have begin to run into a particular issue while during > the 'vhacs cluster -I' (eg: cluster initilization) routine when we had a > bunch of VHACS server and client clouds active, and need to bring them > all down. Before explaining the case, here is a little background: > > DRBD_TARGET: LIO-Target + DRBD export Cluster RA (Server side VHACS) > ISCSI_MOUNT: Open/iSCSI + Ext3 Mount Cluster RA (Client side VHACS) > > Here is the scenario using 'node.session.timeo.replacement_timeout = > 120': > > We have been experiencing a case where sometimes the DRBD_TARGET RA for > a particular VHACS cloud gets shutdown BEFORE the ISCSI_MOUNT RA. This > of course means that all of the outstanding I/Os from Open/iSCSI with > regard to the mounted filesystem from the DRBD_TARGET that has gone way > need to be failed back to SCSI with a DID_ERROR status. > > The problem is that the failure of the outstanding I/Os does not seem to > be occuring in all cases. In particular, a iscsiadm --logout I believe > is getting issued, and said logout request failing/timing out because > DRBD_TARGET has been released. It is at this point where umount for the > ext3 mount and/or sync hangs indefinately. When the problem occurs, it > looks like this from the kernel ring buffer: > > iscsi_deallocate_extra_thread_sets:285: ***OPS*** Stopped 1 thread set(s) (2 > total threads). > iscsi_deallocate_extra_thread_sets:285: ***OPS*** Stopped 2 thread set(s) (4 > total threads). > session10: iscsi: session recovery timed out after 120 secs > sd 51:0:0:0: scsi: Device offlined - not ready after error recovery > sd 51:0:0:0: [sdg] Result: hostbyte=DID_BUS_BUSY > driverbyte=DRIVER_OK,SUGGEST_OK > end_request: I/O error, dev sdg, sector 0 > Buffer I/O error on device sdg, logical block 0 > lost page write due to I/O error on sdg > > I should mention that we are not doing any I/O to said iSCSI LUN via > Open/iSCSI other than the filesystem metadata for ext3 umount and > SYNCHRONIZE_CACHE CDB during struct scsi_device deregistration. From > experience with Core-iSCSI, I know the logout path is tricky wrt > exceptions (I spent months on it to handle all cases with Immediate and > Non Immediate Logout, as well as doing logouts on the fly from the same > connection in MC/S and different connections in MC/S :-) > > So the question is: > > I) When a ISCSI_INIT_LOGOUT_REQ is not returned with a > ISCSI_TARGET_LOGOUT_RSP and replacement_timeout fires, are all > outstanding I/Os for that particular session being completed with an > non-recoveryable exception..? Has anyone ever run into this case and/or > tested it..? > > Not being exquisitely fimilar with Open/iSCSI source code, if either of > you gents could have a quick look and see if this is indeed broken and > hopefully provide a simple fix (I will take care of drinks at the Kernel > Summit :-), or if not could point me in the right direction it would be > much apperciated. > > --nab > > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Open/iSCSI + Logout Response Timeout + replacement_timeout firing
Hi Mike! On Sat, 2008-06-28 at 15:33 -0500, Mike Christie wrote: > What version of open-iscsi and kernel are you using? And are you using > the kernel modules with open-iscsi or the ones that come with the kernel? > Whoops, forgot to include that tid-bit: open-iscsi: 2.0.730-1etch1 kernel: I am using v2.6.22.19-kdb, and Jerome is using v2.6.22-4-vserver-amd64 > Nicholas A. Bellinger wrote: > >> > >> The problem is that the failure of the outstanding I/Os does not seem to > >> be occuring in all cases. In particular, a iscsiadm --logout I believe > >> is getting issued, and said logout request failing/timing out because > >> DRBD_TARGET has been released. It is at this point where umount for the > >> ext3 mount and/or sync hangs indefinately. When the problem occurs, it > >> looks like this from the kernel ring buffer: > >> > >> iscsi_deallocate_extra_thread_sets:285: ***OPS*** Stopped 1 thread set(s) > >> (2 total threads). > >> iscsi_deallocate_extra_thread_sets:285: ***OPS*** Stopped 2 thread set(s) > >> (4 total threads). > >> session10: iscsi: session recovery timed out after 120 secs > >> sd 51:0:0:0: scsi: Device offlined - not ready after error recovery > > If you see this then any and all that was sent the device and any new IO > should be failed to the FS and block layer like below. There is a bug in > some kernels though, where if you were to run a iscsiadm logout command > it can hang and lead to weird problems, because the scsi layer is > broken. If you use open-iscsi 869.2's kernel modules or the iscsi > modules in 2.6.25.6 or newer then this is fixed. Ok boss, I will upgrade the tools and jump from v2.6.22.19-kdb to v2.6.25.9-kdb on my VHACs nodes. > Not sure if that is > what you are seeing, because we see IO failed upwards here. Also once we > see "Device offlined", the scsi layer is going to fail the IO when it > hits the scsi prep functions and is never even reaches us. If there is > IO stuck in the driver you could do > cat /sys/class/scsi_host/hostX/host_busy > > to check (that file prints the number of commands the scsi layer has > sent the driver and the driver has not yet returned back (ok so I mean > how many commands is outsatnding)). > > , got it. > >> sd 51:0:0:0: [sdg] Result: hostbyte=DID_BUS_BUSY > >> driverbyte=DRIVER_OK,SUGGEST_OK > >> end_request: I/O error, dev sdg, sector 0 > >> Buffer I/O error on device sdg, logical block 0 > >> lost page write due to I/O error on sdg > > > > > >> > >> I should mention that we are not doing any I/O to said iSCSI LUN via > >> Open/iSCSI other than the filesystem metadata for ext3 umount and > >> SYNCHRONIZE_CACHE CDB during struct scsi_device deregistration. From > >> experience with Core-iSCSI, I know the logout path is tricky wrt > >> exceptions (I spent months on it to handle all cases with Immediate and > >> Non Immediate Logout, as well as doing logouts on the fly from the same > >> connection in MC/S and different connections in MC/S :-) > >> > >> So the question is: > >> > >> I) When a ISCSI_INIT_LOGOUT_REQ is not returned with a > >> ISCSI_TARGET_LOGOUT_RSP and replacement_timeout fires, are all > >> outstanding I/Os for that particular session being completed with an > >> non-recoveryable exception..? Has anyone ever run into this case and/or > >> tested it..? > > If the connection is down when you run iscsiadm logout, we will not send > a logout and the replacement_timeout does not come into play. We just > fast fail the connection and just cleanup the commands and kernel > resrouces and iscsiadm returns (yeah pretty bad I know - it is on the TODO). > > If the connection is up when you run iscsiadm logout, and while the > logout is floating around the connection drops, we are again lazy and > just fail and cleanup and return right away. The replacement_timeout > does not come into play for this and we just fail right away. > > > If you run 869.2 from open-iscsi.org and build with > > make DEBUG_SCSI=1 DEBUG_TCP=1 > make DEBUG_SCSI=1 DEBUG_TCP=1 install > and send all the log output I can tell you better what is going on. > Ok, I will get into this at the office tomorrow afternoon and let you know what I find. Thanks for the info. Many thanks for your most valuable of time, --nab --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Open/iSCSI + Logout Response Timeout + replacement_timeout firing
On Sun, 2008-06-29 at 01:36 -0500, Mike Christie wrote: > Nicholas A. Bellinger wrote: > > Hi Mike! > > > > Hey, looks like you doing some more cool stuff. Don't you have a job > where you have to hack on boring stuff like the rest of us :) > What Jerome said. :-) > > > On Sat, 2008-06-28 at 15:33 -0500, Mike Christie wrote: > >> What version of open-iscsi and kernel are you using? And are you using > >> the kernel modules with open-iscsi or the ones that come with the kernel? > >> > > > > Whoops, forgot to include that tid-bit: > > > > open-iscsi: 2.0.730-1etch1 > > > > kernel: I am using v2.6.22.19-kdb, and Jerome is using > > v2.6.22-4-vserver-amd64 > > > > Ah if your disk are using write back cache then you are going to hit > some problems. So if you see this in /var/log/messages when you loging: > > kernel: sd 9:0:0:1: [sdb] Write cache: enabled, > > then later when you run iscsiadm to log out you see: > > kernel: sd 9:0:0:1: [sdb] Synchronizing SCSI cache > > Then you are going to hit problems due to the scsi sysfs interface > changing on us. iscsiadm is going to hang. IO is going to hang. You > basically have to reboot the box by hand. > Yep, so the LIO-CORE does NOT emulate Write Cache Enable (although there is some Read Cache code :-) bit in the caching mode page in the virtual storage object case (IBLOCK, FILE, etc). We are using the LIO-Core IBLOCK plugin for export DRBD's struct block_device, which uses the generic emulation of MODE_SENSE*, which leaves buf[2] untouched in caching mode page. For the PSCSI subsystem plugin, these mode pages obviously come from the underlying hardware. I even recall that libata does emulate this down to SATA/PATA bits for us. (thanks jgarzik and co) On a related note, did you and Tomo ever get around to implement write/read caching emulation in STGT for virtual devices..? Its definately something that is on my long term list for doing generically amoungst virtual devices in LIO-Core v3.x. Many thanks for your most valuable of time, --nab > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Open/iSCSI + Logout Response Timeout + replacement_timeout firing
Hi Mike, On Mon, 2008-06-30 at 11:27 -0500, Mike Christie wrote: > Konrad Rzeszutek wrote: > >> Ah if your disk are using write back cache then you are going to hit > >> some problems. So if you see this in /var/log/messages when you loging: > >> > >> kernel: sd 9:0:0:1: [sdb] Write cache: enabled, > >> > >> then later when you run iscsiadm to log out you see: > >> > >> kernel: sd 9:0:0:1: [sdb] Synchronizing SCSI cache > >> > >> Then you are going to hit problems due to the scsi sysfs interface > >> changing on us. iscsiadm is going to hang. IO is going to hang. You > >> basically have to reboot the box by hand. > > > > Mike, > > > > Are you sure about this? When the sysfs entries are deleted (during the > > iscsiadm logout phase), the SCSI ml finishes all of the I/Os and the last > > operation is sending the SCSI Cache command. Wouldn't that quiesce I/O ? > > Granted > > See below. You are right if everything goes ok. > > > this means doing these steps which are outside the normal iscsiadm: > > 1). flush dirty pages (call 'sync') > > 2). delete the sysfs entries (echo 1 > /sys/block/sdX/device/delete) > > 3). wait until /sys/class/scsi_host/hostZ/host_busy reaches zero > > 4). iscsiadm -m logout > > > > > The problem that I described occurs when we run the iscsiadm logout > command and we used the sysfs delete file. When iscsiadm wrote to that > attr in 2.6.18 it would return when the cache sync was sent and the > device was fully deleted in the kernel. In 2.6.21 and above it returned > right away. So iscsiadm's logout code would write to that attr and think > the devices were cleaned up, then it would call the iscsi shutdown code > which would send a logout and cleanup the kernel session, connection and > host structs, thinking that the devices were properly flushed but IO > could still be waiting to get written so all types of fun things can > happen like > Yikes.. > We could get to the scsi host remove call and all the scsi device sysfs > delete calls would still be starting up, so the host remove call and > those delete calls would race (so this is we would have bypassed the > host_busy check in the connection deletion function in the kernel). When > this happens if the sysfs delete device got the scan mutex first, but > the iscsi shutdown code had blocked the devices, while we were trying to > remove the host then the iscsiadm logout command will hang, because the > delete device would wait forever to try and send the command (it is not > yet in the host so the command timer is not running and the device is > blocked), and the remove host call is waiting on the scan mutex which > the device has. > > If you have multiple devices then the remove host command can also end > up failing IO, because we will have sent the logout and later set the > session internal state to terminiate and incoming IO on the other > devices that was queued will be failed when the remove devices functions > flush the IO. > > If you do not have a write back cache we have other problems, where IO > can be failed when it should not have for the reason above where the > logout is sent, the terminate bit is set, and the remove host runs > before the devices were properly removed and that causes IO to be failed. > > And actually in some kernels you can hang (the app would hang not > iscsiadm in this case) when a cache sync was not needed, because if a > cache sync was not needed when we would remove the host and it would > delete the device but IO would be stuck in the queues and no one did a > unplug of the queue when the scsi device was removed. We added a > iscsi_unblock_session in the iscsi_session_teardown to flush everything > so at least apps would not hang there (but that resulted in IO getting > failed like above). > Ok, so with Open/iSCSI 2.0-869.2 running on 2.6.25.9 w/ LIO-Core IBLOCK + DRBD with write cache disabled, we are no longer seeing hanging umounts during the rsc_stop() section of ISCSI_MOUNT VHACS cluster resource. Thanks for the helpful info mnc!! :-) Things have started to look up with the latest kernel + open/iscsi userspace code. The only item left that I have been running into on Etch are strange sshd segfaults (requring a service restart) during ISCSI_MOUNT:rsc_stop. They seem to be somehow be Open/iSCSI related (not exactly sure how yet) and happens on multiple machines, from both myself and Jerome's clusters. We have been trying to track this issue down with little luck, but with the new code it definately seems to be happening alot less frequently on my setup (only once so far with the latest Open/iSCSI). Pretty vague I know.. :) Oh yeah, jerome also mentioned something to me this morning about left-over iscsid processes after ISCSI_MOUNT:rsc_stop() as well. What was the problem case on this one again..? --nab > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the
VHACS-VM x86_64 Alpha Preview on 45nm multicore
Greetings all,
I am pleased to announce the Alpha availibility of VHACS-VM Demo for
Hardware Virtualized x86_64 running (initially) on VMWare Workstation 6.
The VHACS cluster stack also obviously runs on 'bare-metal' hardware,
but today VHACS-VM is the easiest method to get your own open storage
cloud up and running.
What is VHACS..?
VHACS is a Cloud Storage implementation running on Linux v2.6. VHACS is
an acrynom for Virtualization, High Availibility, and Cluster Storage.
VHACS is a combination of at least eight (8) long term OSS/Linux based
projects, along with a CLI management interface for controlling VHACS
nodes, clouds, and vservers within the VHACS cluster. Here is a quick
rundown of the projects (so far) that make up the VHACS cluster stack:
CLUSTER:
* ) Pacemaker The scalable High-Availability cluster resource
manager formerly part of Heartbeat
* ) OpenAIS The OpenAIS Standards Based Cluster Framework is an
OSI Certified implementation of the Service Availability Forum
Application Interface Specification (AIS)
SERVER:
* ) LIO-Target w/ IBLOCK LIO-Core subsystem plugin for iSCSI
Target mode export
* ) DRBD Distributed Replicated Block device for Network RAID 1
Mirrors
* ) Linux Logical Volume Manager (LVM) ('vhacs' volume group for
creation of VHACS clouds)
CLIENT:
* ) Open/iSCSI iSCSI Initiator client
* ) Ext3 filesystem
* ) Linux-Vserver Linux Virtualization Layer
I am happy to report that VHACS-VM Alpha Demo x86_64 is running 0.8.15
VHACS code, along with the many other pieces of open source
infrastructure that the VHACS cloud builds upon, and providing
Active/Active H/A with Synchronous Data Replication iSCSI Target ERL=2 /
MC/2 export that makes up the SERVER side of the VHACS cloud. The
CLIENT side of the VHACS cloud is also running an iSCSI Initiator ext3
mounted filesystem with Open/iSCSI on top of Intel 45 nanometer
'Wolfdale' microprocessor silicon and DDR2 800 memory.
Futhermore, using commerically available x86 virtualization technology
on open/closed platforms, I am very happy to report that VHACS-VM 2x
node clusters are now up and running across both OPEN (Linux x86_64) and
CLOSED (win32) x86 platforms on top of the 45nm chips with the VMMU/VMX
parts enabled. The first OS independent storage clouds running on top
of v2.6.25.9-kdb are now a reality.
Here is a link to the code, screenshots, and all the goodies:
http://linux-iscsi.org/index.php/VHACS-VM
http://linux-iscsi.org/index.php/VHACS
The VHACS-VM x86_64 system images, which contain a Debian Etch system
iamges running the very latest compontents that make up the VHACS
cluster stack. Also included in the vhac64-west and vhacs64-east system
images is a complete development environment, including a KDB enabled
kernel that will allow the developer to pause the running image at any
time, examine memory, or setup a breakpoint to track down a problem.
The x86_64 system images currently require hardware x86 virtualization,
please see the wiki for more information about hardware requirements.
They are available from:
http://linux-iscsi.org/builds/VHACS-VM/x86_64/vmware6/
Also, there is great interest to see VHACS-VM running on KVM and QEMU
Virtualization platforms. This is something that should not be too
difficult as long as the virtualization platform supports execution of
x86_64 system images. i386 system images are also in the works to bring
the storage cloud to legacy environments without hardware x86_64
virtualization.
Enjoy,
--nab
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"open-iscsi" group.
To post to this group, send email to open-iscsi@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at http://groups.google.com/group/open-iscsi
-~--~~~~--~~--~--~---
[ANNOUNCE] - VHACS-VM x86_64 Alpha Preview - FOLLOWUP
Greetings all,
Just a quick couple of updates since the VHACS-VM announcement (which
happened to be a day after after v2.6.26. ;):
I) The two VHACS-VM x86_64 images, have been split into 2 compressed
~300 MB images (for the running demo), and a 800 MB BUILD .vmdk disk
image containing the source for VHACS v0.8.15. The three archives can
be found at the same location:
http://linux-iscsi.org/builds/VHACS-VM/x86_64/vmware6/
II) lio-core-2.6.git has been updated to v2.6.26 and loaded successfully
on 32-bit x86 VM. Only a very minor change to get things compiled on
v2.6.26. Please see
http://git.kernel.org/?p=linux/kernel/git/nab/lio-core-2.6.git;a=summary
for more information.
III) VHACS-VM has now successfully been able to handle node failures and
restart failed cluster resource allocations (RA) on the active node,
across multiple platforms using x86_64 hardware virtualization. :-)
Lots of things going on this week. More to come soon. Please keep
checking the http://Linux-iSCSI.org site for the latest information.
Many thanks for your most valuable of time,
--nab
On Mon, 2008-07-14 at 11:02 -0700, Nicholas A. Bellinger wrote:
> Greetings all,
>
> I am pleased to announce the Alpha availibility of VHACS-VM Demo for
> Hardware Virtualized x86_64 running (initially) on VMWare Workstation 6.
> The VHACS cluster stack also obviously runs on 'bare-metal' hardware,
> but today VHACS-VM is the easiest method to get your own open storage
> cloud up and running.
>
> What is VHACS..?
>
> VHACS is a Cloud Storage implementation running on Linux v2.6. VHACS is
> an acrynom for Virtualization, High Availibility, and Cluster Storage.
> VHACS is a combination of at least eight (8) long term OSS/Linux based
> projects, along with a CLI management interface for controlling VHACS
> nodes, clouds, and vservers within the VHACS cluster. Here is a quick
> rundown of the projects (so far) that make up the VHACS cluster stack:
>
> CLUSTER:
>
> * ) Pacemaker The scalable High-Availability cluster resource
> manager formerly part of Heartbeat
> * ) OpenAIS The OpenAIS Standards Based Cluster Framework is an
> OSI Certified implementation of the Service Availability Forum
> Application Interface Specification (AIS)
>
> SERVER:
>
> * ) LIO-Target w/ IBLOCK LIO-Core subsystem plugin for iSCSI
> Target mode export
> * ) DRBD Distributed Replicated Block device for Network RAID 1
> Mirrors
> * ) Linux Logical Volume Manager (LVM) ('vhacs' volume group for
> creation of VHACS clouds)
>
> CLIENT:
>
> * ) Open/iSCSI iSCSI Initiator client
> * ) Ext3 filesystem
> * ) Linux-Vserver Linux Virtualization Layer
>
> I am happy to report that VHACS-VM Alpha Demo x86_64 is running 0.8.15
> VHACS code, along with the many other pieces of open source
> infrastructure that the VHACS cloud builds upon, and providing
> Active/Active H/A with Synchronous Data Replication iSCSI Target ERL=2 /
> MC/2 export that makes up the SERVER side of the VHACS cloud. The
> CLIENT side of the VHACS cloud is also running an iSCSI Initiator ext3
> mounted filesystem with Open/iSCSI on top of Intel 45 nanometer
> 'Wolfdale' microprocessor silicon and DDR2 800 memory.
>
> Futhermore, using commerically available x86 virtualization technology
> on open/closed platforms, I am very happy to report that VHACS-VM 2x
> node clusters are now up and running across both OPEN (Linux x86_64) and
> CLOSED (win32) x86 platforms on top of the 45nm chips with the VMMU/VMX
> parts enabled. The first OS independent storage clouds running on top
> of v2.6.25.9-kdb are now a reality.
>
> Here is a link to the code, screenshots, and all the goodies:
>
> http://linux-iscsi.org/index.php/VHACS-VM
> http://linux-iscsi.org/index.php/VHACS
>
> The VHACS-VM x86_64 system images, which contain a Debian Etch system
> iamges running the very latest compontents that make up the VHACS
> cluster stack. Also included in the vhac64-west and vhacs64-east system
> images is a complete development environment, including a KDB enabled
> kernel that will allow the developer to pause the running image at any
> time, examine memory, or setup a breakpoint to track down a problem.
> The x86_64 system images currently require hardware x86 virtualization,
> please see the wiki for more information about hardware requirements.
> They are available from:
>
> http://linux-iscsi.org/builds/VHACS-VM/x86_64/vmware6/
>
> Also, there is great interest to see VHACS-VM running on KVM and QEMU
> Virtualization platforms. This is something that should not be too
> difficult as long as the virtualization platform supports execution of
> x86_
Re: "iSCSI PDU timed out" error trying to connect to LIO targets
On Wed, 2008-10-22 at 09:43 -0500, Mike Christie wrote: > Martin wrote: > > > > Target: iqn.2001-09.net.ketsds:keat.storage.fast1 > > Portal: 10.1.206.2:3260,1 > > Iface Name: default > > Target: iqn.2001-09.net.ketsds:keat.storage.fast2 > > Portal: 10.1.206.2:3260,1 > > Iface Name: default > > Target: iqn.2001-09.net.ketsds:keat.storage.fast3 > > Portal: 10.1.206.2:3260,1 > > Iface Name: default > > > > Ok it looks like we fixed the problem where we got duplicate records and > we got a already running error. > > For the pdu timed out problem, I think we need a ethereal/wrireshark > trace so we can see why the target is sending us the wrong task tag. Do > you know how to run this? In centos5 I believe you have to use ethereal > (I do not think we picked up the renamed wireshark). Just run that on > the initiator box, when you run the "iscsiadm -m node -l". When that > fails with timed out errors, stop ethereal and send the trace. > Greetings Mike and Martin, I just saw this thread this afternoon and decided to have a closer look at RFC-3720 wrt your "pdu timed out problem", which has to do with what fields of Login Response are considered "invalid" when a login fails (for whatever reason) and a Status-Class of non zero is returned to the iSCSI Initiator. From the RFC: 10.13.5. Status-Class and Status-Detail The status classes are as follows: 0 - Success - indicates that the iSCSI target successfully received, understood, and accepted the request. The numbering fields (StatSN, ExpCmdSN, MaxCmdSN) are only valid if Status-Class is 0. I had previously thought that Initiator Task Tag was also part of this list for *ONLY* being valid when Status-Class is 0 (which is why you are seeing it set to zero during login response failures), but as it is most definately *NOT* correct according to RFC-3720, I went ahead and made LIO-Target properly set Initiator Task Tag in Login Responses with non-zero Status-Class. Here is the commit that has been tested with v3.0: http://git.kernel.org/?p=linux/kernel/git/nab/lio-core-2.6.git;a=commit;h=23280e4252c447735c3dd810d0b936f34c6f06a7 It has also been backed ported to v2.9-STABLE as r402. Thanks! --nab Please make sure to include me on the thread if you run into any other strangeness. :-) > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
[Open/iSCSI] Memory leak in repetitive --login/--logout with v2.0-870.1
Greetings Mike, Hannes and Co, During some recent testing using the Open/iSCSI Initiator v2.0-870.1, against the LIO-Target v3.0 tree, I noticed that while running the following script: while [ 1 ]; do iscsiadm -m node -T $TARGETNAME -p $PORTAL --login iscsiadm -m node -T $TARGETNAME -p $PORTAL --logout done for an extended period of time that I started getting OOM failures on the VMs running Open/iSCSI. Upon closer examination, this is what I found: Linux ubuntu 2.6.27.10 #2 SMP Tue Jan 6 18:33:00 PST 2009 i686 GNU/Linux Using open-iscsi-2.0-870.1: [78196.520214] scsi7981 : iSCSI Initiator over TCP/IP [78284.175307] scsi7982 : iSCSI Initiator over TCP/IP [78338.568656] scsi7983 : iSCSI Initiator over TCP/IP [78405.22] scsi7984 : iSCSI Initiator over TCP/IP Output from slaptop: OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 1037001 1036598 99%0.03K 9177 113 36708K size-32 - Linux opensuse 2.6.22.5-31-default #1 SMP 2007/09/21 22:29:00 UTC i686 i686 i386 GNU/Linux scsi7046 : iSCSI Initiator over TCP/IP scsi7047 : iSCSI Initiator over TCP/IP scsi7048 : iSCSI Initiator over TCP/IP scsi7049 : iSCSI Initiator over TCP/IP Output from slabtop: OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 914057 913581 99%0.03K 8089 113 32356K size-32 - So it appears that memory is getting leaked in the size-32 range with each --login + --logout invocation. I also tried the same test with the shipping Open/iSCSI code in Debian v4 and OpenSuse 10.3 and these also suffer from the same issue. Also of interest is that running the following script for Discovery SendTargets *DOES NOT* reproduce the leak. while [ 1 ]; do iscsiadm -m discovery -t sendtargets -p $PORTAL done Please let me know if there is anything else I can do to help diagnose the issue. Many thanks for your most valuable of time, --nab --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: [Open/iSCSI] Memory leak in repetitive --login/--logout with v2.0-870.1
On Sun, 2009-01-11 at 20:02 -0600, Mike Christie wrote: > Nicholas A. Bellinger wrote: > > Greetings Mike, Hannes and Co, > > > > During some recent testing using the Open/iSCSI Initiator v2.0-870.1, > > against the LIO-Target v3.0 tree, I noticed that while running the > > following script: > > > > while [ 1 ]; do > > iscsiadm -m node -T $TARGETNAME -p $PORTAL --login > > iscsiadm -m node -T $TARGETNAME -p $PORTAL --logout > > done > > > > for an extended period of time that I started getting OOM failures on > > the VMs running Open/iSCSI. Upon closer examination, this is what I > > found: > > > > > > > > Linux ubuntu 2.6.27.10 #2 SMP Tue Jan 6 18:33:00 PST 2009 i686 GNU/Linux > > > > Using open-iscsi-2.0-870.1: > > > > [78196.520214] scsi7981 : iSCSI Initiator over TCP/IP > > [78284.175307] scsi7982 : iSCSI Initiator over TCP/IP > > [78338.568656] scsi7983 : iSCSI Initiator over TCP/IP > > [78405.22] scsi7984 : iSCSI Initiator over TCP/IP > > > > > Hey, so are there any devices on the target? I do not see the normal > type/size info we see when scsi disks are found. Just checking. That > rules a lot of places out. > > If there are disks, but they just are not gettting logged could you > remove them from the target so we can take some structs out of the mix? > I tried the same test both WITH and WITHOUT iSCSI LUNs getting registered on the Open/iSCSI side. The leak in size-32 is still present in both cases. > > > Output from slaptop: > > > > OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME > > > > 1037001 1036598 99%0.03K 9177 113 36708K size-32 > > > > - > > > > > > > > Linux opensuse 2.6.22.5-31-default #1 SMP 2007/09/21 22:29:00 UTC i686 i686 > > i386 GNU/Linux > > > > scsi7046 : iSCSI Initiator over TCP/IP > > scsi7047 : iSCSI Initiator over TCP/IP > > scsi7048 : iSCSI Initiator over TCP/IP > > scsi7049 : iSCSI Initiator over TCP/IP > > > > Output from slabtop: > > > > OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME > > > > 914057 913581 99%0.03K 8089 113 32356K size-32 > > > > - > > > > So it appears that memory is getting leaked in the size-32 range with > > each --login + --logout invocation. I also tried the same test with the > > shipping Open/iSCSI code in Debian v4 and OpenSuse 10.3 and these also > > suffer from the same issue. > > > > Also of interest is that running the following script for Discovery > > SendTargets *DOES NOT* reproduce the leak. > > > > while [ 1 ]; do > > iscsiadm -m discovery -t sendtargets -p $PORTAL > > done > > > The leak in the size-32 slab would be a kernel object right? if so the > sendtargets test not leaking means that this is a problem in the > session/connection kernel struct setup/destruction. The sendtargets code > is all in userspace so it would not leak in those objects. > > > I was out of the office sick last week, so let me catch up on work stuff > then I will try to send a patch. If you want you could try to stick > printks in iscsi driver model object release functions to make sure they > are getting fired, but that gets nasty. > > Ok, I still have my hands full with LIO v3.0 code atm, so I am not sure how soon I could get to this.. --nab > > > > Please let me know if there is anything else I can do to help diagnose > > the issue. > > > > Many thanks for your most valuable of time, > > > > --nab > > > > > > > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-scsi" in > > the body of a message to majord...@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: [Open/iSCSI] Memory leak in repetitive --login/--logout with v2.0-870.1
On Mon, 2009-01-12 at 20:41 -0600, Mike Christie wrote: > Mike Christie wrote: > > Nicholas A. Bellinger wrote: > > > >> Ok, I still have my hands full with LIO v3.0 code atm, so I am not sure > >> how soon I could get to this.. > >> > > > > Don't worry about it. I can replicate it here. > > > > Crazy, I guess it is a leak we always had (at least back to 2.6.22 (did > not look firther)), and no one noticed. Here is a patch. I will send it > to James with some other patches I had. > > Thanks for the report. > Thanks Mike!!! --nab > > > plain text document attachment (libiscsi-fix-pool-queue-leak.patch) > diff --git a/drivers/scsi/libiscsi.c b/drivers/scsi/libiscsi.c > index 7225b6e..257c241 100644 > --- a/drivers/scsi/libiscsi.c > +++ b/drivers/scsi/libiscsi.c > @@ -1981,6 +1981,7 @@ void iscsi_pool_free(struct iscsi_pool *q) > kfree(q->pool[i]); > if (q->pool) > kfree(q->pool); > + kfree(q->queue); > } > EXPORT_SYMBOL_GPL(iscsi_pool_free); > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
open-iscsi@googlegroups.com
On Fri, 2009-02-20 at 02:50 -0800, vaibhav nipunage wrote: > Hi All, > > I have question about how the iscsi-target & any initator negotiate > the block size. i.e. how initiator will understand that the iscsi- > target is configured for another size, say 4096. How will iscsi-target > tell the initiators that I am configured for 4096 as block size > instead of 512? > This is either done via READ_CAPACITY or READ_CAPACITY_16 CDBs during SCSI device scan on the initiator/client side using the "Logical Block Length in Bytes" or using Block Descriptor pages with MODE_SENSE* and MODE_SELECT* to determine the "Logical Block Length" according to sbc-3 documents. The latter is AFAICT is the preferred method to do detection of a changed sector/block size in an OS independent manner. Linux for example currently does this for TYPE_DISK using drivers/scsi/sd.c:sd_revalidate_disk() to do this in a OS dependent manner to redetermine what the total sector/block count and sector/block size are for a given SCSI target endpoint. --nab > Please help me. > > Thanks in advance. > > Vaibhav Nipunage > > > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: [Fwd: [PATCH] [Target_Core_Mod/Persistent_Reservations]: Add support for PREEMPT_AND_ABORT SA]
Hi Mike, Hope you are well.. :-) Any chance that I can bribe you at LSF next week to to take a look at this..? Many thanks for your most valuable of time, --nab On Fri, 2009-03-27 at 02:02 -0700, Nicholas A. Bellinger wrote: > Greetings Mike, Hannes and co: > > Here is the OOPs that I am seeing with upstream Open-iSCSI on kernel.org > v2.6.27.10 with the TASK_ABORTED status getting returned to outstanding > struct scsi_cmnd from lio-core.2.6.git provided fabric. Here is the > code from drivers/scsi/iscsi_tcp.c:iscsi_tcp_task_init(): > > BUG_ON(__kfifo_len(tcp_task->r2tqueue)); > > Is this something that has been fixed in recent Open-iSCSI code..? Any > ideas..? > > Many thanks for your most valuable of time, > > --nab > > > > > > > email message attachment, "Forwarded message - [PATCH] > [Target_Core_Mod/Persistent_Reservations]: Add support for > PREEMPT_AND_ABORT SA" > > Forwarded Message > > From: Nicholas A. Bellinger > > To: linux-scsi , LKML > > > > Cc: James Bottomley , Mike > > Christie , FUJITA Tomonori > > , Hannes Reinecke , > > Martin K. Petersen , Alan Stern > > , Douglas Gilbert > > , Alasdair G Kergon , Philipp > > Reisner , Ming Zhang > > > > Subject: [PATCH] [Target_Core_Mod/Persistent_Reservations]: Add > > support for PREEMPT_AND_ABORT SA > > Date: Fri, 27 Mar 2009 01:46:36 -0700 > > > > Greetings all, > > > > This patch adds support for the PROUT PREEMPT_AND_ABORT service action to > > core_scsi3_emulate_pro_preempt() and core_tmr_lun_reset() using existing TAS > > (TASK_ABORTED status) emulation. The logic assigns the initiator port > > provided pre-registered reservation key to allocated SCSI Task and Task > > Management > > through logical unit ACLs in target_core_mod code. > > > > This patch uses struct list_head preempt_and_abort_list inside of > > core_scsi3_emulate_pro_preempt() to track PR registrations/reservation > > data structure t10_pr_registration_t once it has been removed from the > > se_device_t list, and before they get released back into struct kmem_cache > > t10_pr_reg_cache in core_scsi3_release_preempt_and_abort(). > > > > These patches are made against lio-core-2.6.git/master and tested on > > v2.6.29 x86 32-bit HVM using sg_persist and sg_request from sg3_utils. > > The lio-core-2.6.git tree can be found at: > > > > http://git.kernel.org/?p=linux/kernel/git/nab/lio-core-2.6.git;a=summary > > > > So far, I have been primarily testing the following scenario, which is what > > linux-cluster userspace in RHEL/CentOS (and everyone else) does with > > fence_scsi > > today for their WRITE Exclusive, Registrants Only persistent > > reservation setup. Here is test setup I am using: > > > > # The same /sys/kernel/config/target/core/$HBA/$DEV Linux LVM object mapped > > across > > # two different LIO-Target iSCSI target ports > > initiator# lsscsi --transport > > [3:0:0:0]disk > > iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd,t,0x1 /dev/sde > > [4:0:0:0]disk > > iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0,t,0x1 /dev/sdf > > > > *) Initiator side using Open/iSCSI: > > > > [ 185.682972] scsi3 : iSCSI Initiator over TCP/IP > > [ 185.998817] scsi 3:0:0:0: Direct-Access LIO-ORG IBLOCK > > 3.0 PQ: 0 ANSI: 5 > > [ 186.009172] sd 3:0:0:0: [sde] 128000 4096-byte hardware sectors (524 MB) > > [ 186.013274] sd 3:0:0:0: [sde] Write Protect is off > > [ 186.013286] sd 3:0:0:0: [sde] Mode Sense: 2f 00 00 00 > > [ 186.034372] sd 3:0:0:0: [sde] Write cache: disabled, read cache: > > enabled, doesn't support DPO or FUA > > [ 186.045854] sd 3:0:0:0: [sde] 128000 4096-byte hardware sectors (524 MB) > > [ 186.054955] sd 3:0:0:0: [sde] Write Protect is off > > [ 186.054967] sd 3:0:0:0: [sde] Mode Sense: 2f 00 00 00 > > [ 186.072648] sd 3:0:0:0: [sde] Write cache: disabled, read cache: > > enabled, doesn't support DPO or FUA > > [ 186.073401] sde: unknown partition table > > [ 186.084108] sd 3:0:0:0: [sde] Attached SCSI disk > > [ 186.085052] sd 3:0:0:0: Attached scsi generic sg4 type 0 > > [ 186.316470] alua: device handler registered > > [ 186.321957] sd 3:0:0:0: alua: supports implicit TPGS > > [ 186.326881] sd 3:0:0:0: alua: port group 00 rel port 02 > > [ 186.331445] sd 3:0:0:0: alua: port group 00 state A supports tousNA > > [ 186.372696] sd 3:0:0:0: alua: port group 00 state A supports tousNA > > [ 188.204
Re: [Fwd: [PATCH] [Target_Core_Mod/Persistent_Reservations]: Add support for PREEMPT_AND_ABORT SA]
On Mon, 2009-03-30 at 19:34 -0500, Mike Christie wrote: > Nicholas A. Bellinger wrote: > > Greetings Mike, Hannes and co: > > > > Here is the OOPs that I am seeing with upstream Open-iSCSI on kernel.org > > v2.6.27.10 with the TASK_ABORTED status getting returned to outstanding > > It looks like you are returning TASK_ABORTED when a R2T has been sent to > the initiator and it was responding or do we have InitialR2T=No and are > doing that first part of the write. > > If so it might be the following: > > Novell's target sent a check condition when a R2T was being processed > and at that time, our reading of "10.4.2. Status": > > If a SCSI device error is detected while data from the initiator is > still expected (the command PDU did not contain all the data and the > target has not received a Data PDU with the final bit Set), the > target MUST wait until it receives a Data PDU with the F bit set in > the last expected sequence before sending the Response PDU. > > > We took this to mean that if we were sending data in response to a r2t > or with the command pdu as part of InitialR2T=No handling, then the > target would not send a scsi command response pdu on us. If it did then > that r2t handling is basically dangling around and when the task is > reallocated we see it did not get handled in that check. > Good eye, I completely agree with your interpretation of 10.4.2. > If we read the spec right then the fix for us is simple. Fix your target :) > Ok, I will fix LIO-Target to follow 10.4.2.. > If I goofed then the fix for us might be simple. See the attached patch. > I would have to look at the other drivers to make sure it works for them. > Of course, not running into this BUG on the initiator side (even when targets are being naughty) would be the ideal solution. :-) I will give our test a run and see what happens. Many thanks for your most valuable of time, --nab > > > plain text document attachment (force-cleanup.patch) > --- linux-2.6.27.20/drivers/scsi/libiscsi.c.orig 2009-03-30 > 19:32:26.0 -0500 > +++ linux-2.6.27.20/drivers/scsi/libiscsi.c 2009-03-30 19:33:25.0 > -0500 > @@ -332,6 +332,9 @@ static void iscsi_complete_command(struc > struct iscsi_session *session = conn->session; > struct scsi_cmnd *sc = task->sc; > > + if (task->state != ISCSI_TASK_PENDING) > + conn->session->tt->cleanup_task(conn, task); > + > list_del_init(&task->running); > task->state = ISCSI_TASK_COMPLETED; > task->sc = NULL; > @@ -402,8 +405,6 @@ static void fail_command(struct iscsi_co >* the cmd in the sequencing >*/ > conn->session->queued_cmdsn--; > - else > - conn->session->tt->cleanup_task(conn, task); > /* >* Check if cleanup_task dropped the lock and the command completed, >*/ --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet
On Thu, 2010-01-28 at 20:45 +0200, Pasi Kärkkäinen wrote: > On Thu, Jan 28, 2010 at 07:38:28PM +0100, Bart Van Assche wrote: > > On Thu, Jan 28, 2010 at 4:01 PM, Joe Landman > > wrote: > > > Pasi Kärkkäinen wrote: > > >> > > >> Please check these news items: > > >> > > >> http://blog.fosketts.net/2010/01/14/microsoft-intel-push-million-iscsi-iops/ > > >> > > >> http://communities.intel.com/community/openportit/server/blog/2010/01/19/100-iops-with-iscsi--thats-not-a-typo > > >> > > >> http://www.infostor.com/index/blogs_new/dave_simpson_storage/blogs/infostor/dave_simpon_storage/post987_37501094375591341.html > > >> > > >> "1,030,000 IOPS over a single 10 Gb Ethernet link" > > > > > > This is less than 1us per IOP. Interesting. Their hardware may not > > > actually support this. 10GbE typically is 7-10us, though ConnectX and > > > some > > > others get down to 2ish. > > > > Which I/O depth has been used in the test ? Latency matters most with > > an I/O depth of one and is almost irrelevant for high I/O depth > > values. > > > > iirc outstanding I/Os was 20 in that benchmark. Also of interest, according to the following link http://gestaltit.com/featured/top/stephen/wirespeed-10-gb-iscsi/ is that I/Os are being multiplexed across multiple TCP connections using RFC-3720 defined Multiple Connection per Session (MC/S) logic between the MSFT Initiator Nehalem machine and Netapp Target array: "The configuration tested (on the initiator side) was an IBM x3550 with dual 2 GHz CPUs, 4 GB of RAM, and an Intel 82598 adapter. This is not a special server – in fact, it’s pretty low-end! The connection was tuned with RSS, NetDMA, LRO, LSO, and jumbo frames and maxed out over 4 MCS connections per second. I’m not sure what kind of access they were doing (I’ll ask Suzanne), but it’s pretty impressive that the NetApp Filer could push 1,174 megabytes per second!' It just goes to show that software iSCSI MC/S can really scale to some very impressive results with enough x86_64 horsepower behind it.. --nab > > -- Pasi > -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: mc/s - not yet in open-iscsi?
On Thu, 2010-06-10 at 13:35 -0700, Nicholas A. Bellinger wrote: > On Thu, 2010-06-10 at 13:36 +0400, Vladislav Bolkhovitin wrote: > > Christopher Barry, on 06/10/2010 03:09 AM wrote: > > > Greetings everyone, > > > > > > Had a question about implementing mc/s using open-iscsi today. Wasn't > > > really sure exactly what it was. From googling about, I can't find any > > > references of people doing it with open-iscsi, although I see a few > > > references to people asking about it. Anyone know the status on that? > > > > http://scst.sourceforge.net/mc_s.html. In short, there's no point in it > > worth implementation and maintenance effort. > > Heh, this URL is a bunch of b*llshit handwaving because the iscsi-scst > target does not support the complete set of features defined by > RFC-3720, namely MC/S and ErrorRecoveryLevel=2, let alone asymmeteric > logical unit access (ALUA) MPIO. Vlad, if you are so sure that MC/S is > so awful, why don't you put your money where your mouth is and start > asking these questions on the IETF IPS list and see what Julian Satran > (the RFC editor) has to say about them..? > > H...? > Btw, just for those following along, here is what MC/S and ERL=2 when used in combination (yes, they are complementary) really do: http://linux-iscsi.org/builds/user/nab/Inter.vs.OuterNexus.Multiplexing.pdf Also, I should mention in all fairness that my team was the first to implement both a Target and Initiator capable of MC/S and ErrorRecoveryLevel=2 running on Linux, and the first target capable of running MC/S from multiple initiator implementations. Unfortuately Vlad has never implemented any of these features in either a target or initiator, so really he is not in a position to say what is 'good' or what is 'bad' about MC/S. Best, --nab -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: mc/s - not yet in open-iscsi?
On Thu, 2010-06-10 at 13:36 +0400, Vladislav Bolkhovitin wrote: > Christopher Barry, on 06/10/2010 03:09 AM wrote: > > Greetings everyone, > > > > Had a question about implementing mc/s using open-iscsi today. Wasn't > > really sure exactly what it was. From googling about, I can't find any > > references of people doing it with open-iscsi, although I see a few > > references to people asking about it. Anyone know the status on that? > > http://scst.sourceforge.net/mc_s.html. In short, there's no point in it > worth implementation and maintenance effort. Heh, this URL is a bunch of b*llshit handwaving because the iscsi-scst target does not support the complete set of features defined by RFC-3720, namely MC/S and ErrorRecoveryLevel=2, let alone asymmeteric logical unit access (ALUA) MPIO. Vlad, if you are so sure that MC/S is so awful, why don't you put your money where your mouth is and start asking these questions on the IETF IPS list and see what Julian Satran (the RFC editor) has to say about them..? H...? --nab -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: mc/s - not yet in open-iscsi?
On Fri, 2010-06-11 at 23:17 +, Raj wrote: > Nicholas A. Bellinger writes: > > > Btw, just for those following along, here is what MC/S and ERL=2 when > > used in combination (yes, they are complementary) really do: > > > > http://linux-iscsi.org/builds/user/nab/Inter.vs.OuterNexus.Multiplexing.pdf > > > > Also, I should mention in all fairness that my team was the first to > > implement both a Target and Initiator capable of MC/S and > > ErrorRecoveryLevel=2 running on Linux, and the first target capable of > > running MC/S from multiple initiator implementations. > > > > But the end result is what? open-iSCSI still doesn't have the MC/S even > though > it is useful? So without going into a multi-year history lesson as to why MC/S is not currently supported in Open-iSCSI, what it boils down to is this: MC/S (or InterNexus multiplexing) is very useful for scaling iSCSI sessions across multiple cores and network ports with minimal overhead compared with traditional link layer bonding, not to mention that individual iSCSI TCP connections can be independently going across multiple different subnets and backbone providers. It also provides less overhead and fewer kernel threads context switches than host OS dependent MPIO and allows for the ability to scale the number of LUNs and bandwith of individual LUNs across groups of TCP TX/RX kernel threads (for a single iSCSI session (I_T Nexus). MC/S is complementary to legacy symmetric and new SPC-4 ALUA MPIO, and can be used together following with RFC-3720. MC/S functions independently of the negotiated ErrorRecoveryLevel, but if one connection fails in ERL=0 then all of the connections must be restarted with session reinstatement (restart I_T Nexus). The Core-iSCSI MC/S Initiator has existed out of tree for Linux v2.6 for about 5 years now, and has been ported to a number of different embedded devices and server class architectures.MC/S support is included in every install of MSFT including their Hyper-V offering (and is used for the 1 million IOPs windows iSCSI initiator). MC/S is also now becoming a hard requirement for hypervisor level iSCSI initiators (either KVM or based on Linux) in order to scale bandwith across a number of CPUs cores on small pipes (4x 1 Gb/sec) and on bigger pipes (10 Gb/sec) with traditional iSCSI. MC/S is available from select iSCSI target implementations, including all versions of the open source LIO-Target stack and TCM fabric module (v2.x, v3.x and v4.x-rc) and from certain Netapp hardware and probably other big array vendors who compete with Netapp at this point. In terms of implementing MC/S support into Open-iSCSI, the initiator side fast path for MC/S is straight forward enough, eg: once the CmdSN window has opened, assign the incoming struct scsi_cmnd alligence to one active iSCSI connection, and queue up to that connections TX thread. I will say the user <-> kernel netlink design for the control path does make this a little more interesting, becase also with MC/S iSCSI connections can be brought up and down on the fly at any time, which also adds a certain level of complexity in the shutdown path for an iSCSI Initiator. Anyways, this is something that I have been considering for a while, and I think it will eventually happen (either by myself or someone else) who is doing bringup against existing stable LIO-Target MC/S logic. Best, --nab -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: mc/s - not yet in open-iscsi?
On Mon, 2010-06-14 at 23:44 +0400, Vladislav Bolkhovitin wrote: > Raj, on 06/12/2010 03:17 AM wrote: > > Nicholas A. Bellinger writes: > > > >> Btw, just for those following along, here is what MC/S and ERL=2 when > >> used in combination (yes, they are complementary) really do: > >> > >> http://linux-iscsi.org/builds/user/nab/Inter.vs.OuterNexus.Multiplexing.pdf > >> > >> Also, I should mention in all fairness that my team was the first to > >> implement both a Target and Initiator capable of MC/S and > >> ErrorRecoveryLevel=2 running on Linux, and the first target capable of > >> running MC/S from multiple initiator implementations. > >> > > > > But the end result is what? open-iSCSI still doesn't have the MC/S even > > though > > it is useful? > > The end result is that any driver level multipath, including MC/S, is > forbidden in Linux to encourage developers and vendors to improve MPIO > and not to try to workaround problems in it by their homebrewed > multipath solutions [1]. This is such pre multi-core pre FSB year 2005 agruement.. > As the result of this very smart policy, Linux > MPIO is in a very good shape now. Particularly, it scales with more > links quite well. In contrast, according to Microsoft's data linked in > this list recently, Windows MPIO scales quite badly, but Linux MPIO > scales similarly as Windows MC/S does [2]. (BTW, this is a good evidence > that MC/S doesn't have any inherent performance advantage over MPIO.) > Then why can't you produce any numbers for Linux or MSFT, hmmm..? Just as a matter of record, back in 2005 it was proved that Linux/iSCSI running with both MC/S *and* MPIO where complementary and improved throughput by ~1 Gb/sec using the 1st generation (single core) AMD Operton x86_64 chips on PCI-X 133 Mhz 10 Gb/sec with stateless TCP offload: http://www.mail-archive.com/linux-s...@vger.kernel.org/msg02225.html > But we are on the Linux list, so we don't care about Windows' problems. > Everybody are encouraged to use MPIO and, if have any problem with it, > report it in the appropriate mailing lists. > You are completely missing the point and ignoring the 'bigger picture' of what is going on in the iSCSI ecosystem. Also, if you think that arguing against the transparency of the upstream Linux/iSCSI fabric for complete RFC-3720 support with some unproven conjecture is going to win you something here, you are complete wrong. Just because you have not done the work yourself to implement the interesting RFC-3720 features does not mean you get to dictate (or dictate to others on this list) what the future of Linux/iSCSI will be. > Vlad > > [1] Yes, MC/S is just a workaround apparently introduced by IETF > committee to eliminate multipath problems they see in SCSI inside their > _own_ protocol instead of to push T10 committee to make the necessary > changes in SAM. Or, because a lack of acceptance of those problem from > T10 committee. But I'm not familiar with so deep history, so can only > speculate about it. Complete and utterly wrong. You might want to check your copy of SAM, because a SCSI fabric is allowed to have multipath communication paths and ports as long as the ordering is enforced for the I_T Nexus at the Target port. Again, you can speculate however you want without any MC/S implementation experience, but if you are ready to get serious with your unproven conjecture, please take it to the IETF IPS list and we will see what the fathers of iSCSI have to say about it. > > [2] The Windows MPIO limitations can well explain why Microsoft is the > only OS vendor pushing MC/S: for them it's simpler to implement MC/S > than to fix those MPIO scalability problems. Additionally, it could have > a future marketing value for them: the improved MPIO scalability would > be a big point to push customers to migrate to the new Windows version. > But this is again just a vague speculation ground. We will see in the > future. > Yeah sure, Intel, MSFT and Netapp claiming 1.25 million IOPs with MC/S on 5500 series chipsets with 5600 32nm chips and demonstrating it pubically all over the place for the last quarter is *really* a vague speculation. --nab -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: [PATCH 01/12] libiscsi: Convert to host_lock less w/ interrupts disabled internally
On Sun, 2010-12-19 at 19:07 -0700, Matthew Wilcox wrote: > On Sun, Dec 19, 2010 at 05:22:06PM -0800, Nicholas A. Bellinger wrote: > > Actually sorry, Mike Christie did already make a clarification on this > > subject here: > > > > http://marc.info/?l=linux-scsi&m=129010439421506&w=2 > > > > I had originally thought the same that session->lock should be using > > some flavour of spin_lock_irq*() as well, but apparently this is not the > > case for libiscsi. > > Right, so it seems. "the session lock is just locked in softirqs/timers" > means that it does need to be the _bh() version of spin_lock though. > > I'm actually not sure ... is it safe to use the _bh versions in BH > context? I think it is because the preempt count is nested, unlike the > _irq variants of spinlocks. > Hmmm, fair point. Merging the following incremental patch to convert session->lock to use spin_lock_bh() in iscsi_queuecommand() into lock_less-LLDs-for-38-v3: commit 744f1c119b3fb0c1a6b3c67a7b490e234d1a7e75 Author: Nicholas Bellinger Date: Mon Dec 20 09:17:19 2010 + libiscsi: Convert iscsi_queuecommand to use spin_lock_bh This patch converts iscsi_queuecommand() code to obtain struct iscsi_session->lock using spin_lock_bh() to properly handle bottom-half context operation. Reported-by: Matthew Wilcox Signed-off-by: Nicholas A. Bellinger After a quick audit of iscsi_session->lock usage, and I see that iscsi_complete_pdu(), iscsi_tmf_timedout(), iscsi_eh_cmd_timed_out(), iscsi_check_transport_timeouts() are using spin_lock(), and iscsi_session_failure() and iscsi_conn_failure() are using spin_lock_irqsave(). Mike and Hannes, would you guys mind commenting on this..? From what I can determine these should all be converted to use spin_lock_bh(), yes..? --nab -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: [PATCH 01/12] libiscsi: Convert to host_lock less w/ interrupts disabled internally
On Mon, 2010-12-20 at 18:36 -0600, Mike Christie wrote: > On 12/20/2010 03:30 AM, Nicholas A. Bellinger wrote: > > After a quick audit of iscsi_session->lock usage, and I see that > > iscsi_complete_pdu(), iscsi_tmf_timedout(), iscsi_eh_cmd_timed_out(), > > iscsi_check_transport_timeouts() are using spin_lock(), and > > iscsi_session_failure() and iscsi_conn_failure() are using > > spin_lock_irqsave(). > > > > Mike and Hannes, would you guys mind commenting on this..? From what I > > can determine these should all be converted to use spin_lock_bh(), > > yes..? > > > > Yeah, they can use _bh locking. I was going to use them for qla4xxx eh, > which does iscsi processing in its irq, but we never did. Thanks for the clarification. I would be happy to submit a patch next week for this seperately from the host_lock-less conversion pieces, or would you prefer handle this yourself..? Best Regards, --nab -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: [RFC-v2 01/12] iscsi: Resolve iscsi_proto.h naming conflicts with drivers/target/iscsi
On Mon, 2011-03-14 at 15:59 +0100, Christoph Hellwig wrote: > On Mon, Mar 14, 2011 at 04:56:58AM -0700, Nicholas A. Bellinger wrote: > > From: Nicholas Bellinger > > > > This patch renames the following iscsi_proto.h structures to avoid > > namespace issues with drivers/target/iscsi/iscsi_target_core.h: > > > > *) struct iscsi_cmd -> struct iscsi_scsi_cmd > > This name is not overly descriptive. Using _req/_rsp uniformly would make a > lot more sense. > Mmmm, fair point here.. I don't have a strong preference so I will defer to Mike and Hannes on this (open-iscsi CC'ed). Guys is this OK with you..? Thanks, --nab -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: [RFC-v2 01/12] iscsi: Resolve iscsi_proto.h naming conflicts with drivers/target/iscsi
On Mon, 2011-03-14 at 17:13 -0500, Mike Christie wrote: > On 03/14/2011 04:40 PM, Nicholas A. Bellinger wrote: > > On Mon, 2011-03-14 at 15:59 +0100, Christoph Hellwig wrote: > >> On Mon, Mar 14, 2011 at 04:56:58AM -0700, Nicholas A. Bellinger wrote: > >>> From: Nicholas Bellinger > >>> > >>> This patch renames the following iscsi_proto.h structures to avoid > >>> namespace issues with drivers/target/iscsi/iscsi_target_core.h: > >>> > >>> *) struct iscsi_cmd -> struct iscsi_scsi_cmd > >> > >> This name is not overly descriptive. Using _req/_rsp uniformly would make > >> a > >> lot more sense. > >> > > > > Mmmm, fair point here.. I don't have a strong preference so I will > > defer to Mike and Hannes on this (open-iscsi CC'ed). Guys is this OK > > with you..? > > > > Ok with me. OK, I will go ahead and convert these libiscsi PDU definitions to iscsi_scsi_req and iscsi_scsi_rsp and update patch #1 of the series for RFC-v3. Thanks Mike! --nab -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: [PATCH] iscsi: Simplify serial number comparisons
On Thu, 2011-04-07 at 14:26 -0700, Mark Rustad wrote:
> Unsigned serial number comparison is very simple if you simply put the
> difference into a signed integer of the same size and then compare that
> value with zero. All the complexity and confusion fall away.
>
> Signed-off-by: Mark Rustad
> ---
> include/scsi/iscsi_proto.h | 21 -
> 1 files changed, 12 insertions(+), 9 deletions(-)
>
> diff --git a/include/scsi/iscsi_proto.h b/include/scsi/iscsi_proto.h
> index 0c6c1d6..98ebc3d 100644
> --- a/include/scsi/iscsi_proto.h
> +++ b/include/scsi/iscsi_proto.h
> @@ -35,30 +35,33 @@
> /*
> * Serial Number Arithmetic, 32 bits, RFC1982
> */
> -#define SNA32_CHECK 2147483648UL
>
> static inline int iscsi_sna_lt(u32 n1, u32 n2)
> {
> - return n1 != n2 && ((n1 < n2 && (n2 - n1 < SNA32_CHECK)) ||
> - (n1 > n2 && (n2 - n1 < SNA32_CHECK)));
> + s32 diff = n1 - n2;
> +
> + return diff < 0;
> }
>
> static inline int iscsi_sna_lte(u32 n1, u32 n2)
> {
> - return n1 == n2 || ((n1 < n2 && (n2 - n1 < SNA32_CHECK)) ||
> - (n1 > n2 && (n2 - n1 < SNA32_CHECK)));
> + s32 diff = n1 - n2;
> +
> + return diff <= 0;
> }
>
> static inline int iscsi_sna_gt(u32 n1, u32 n2)
> {
> - return n1 != n2 && (((n1 < n2) && ((n2 - n1) > SNA32_CHECK)) ||
> - ((n1 > n2) && ((n1 - n2) < SNA32_CHECK)));
> + s32 diff = n1 - n2;
> +
> + return diff > 0;
> }
>
> static inline int iscsi_sna_gte(u32 n1, u32 n2)
> {
> - return n1 == n2 || (((n1 < n2) && ((n2 - n1) > SNA32_CHECK)) ||
> - ((n1 > n2) && ((n1 - n2) < SNA32_CHECK)));
> + s32 diff = n1 - n2;
> +
> + return diff >= 0;
> }
Hi Mark,
My apolgies for the delayed response on this patch.
I think this simplification to use s32 for iSCSI Command Serial Number
arithmetic looks perfectly reasonable, and am including this into
lio-core-2.6.git/lio-4.1. Thank you for your contribution!!
Mike, Hannes and Co, do you have any objections here for mainline
Open-iSCSI and iSCSI-Target code for .40 code..?
Best Regards,
--nab
--
You received this message because you are subscribed to the Google Groups
"open-iscsi" group.
To post to this group, send email to open-iscsi@googlegroups.com.
To unsubscribe from this group, send email to
open-iscsi+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/open-iscsi?hl=en.
Re: do discovery through SW transports too
On Tue, 2013-06-18 at 18:35 +0300, Or Gerlitz wrote:
> On 15/06/2013 11:25, Nicholas A. Bellinger wrote:
> > Hi Or & Mike,
> >
> > On Tue, 2013-06-11 at 08:04 +0300, Or Gerlitz wrote:
> >> On 06/06/2013 18:01, Mike Christie wrote:
> >>> However, above I am not talking about that or doing discovery over a
> >>> normal session in general. I was just trying to get clarification for
> >>> what you wanted. I was not sure if there was some new iser spec stuff
> >>> that I missed and you wanted to implement.
> >> Hi Mike,
> >>
> >> Its not new spec stuff, its just environments where plain TCP services
> >> might be out of hand for either of the sides and we don't want that to
> >> be an obstacle to deploy iser.
> > No luck yet generating discovery ib_iser session traffic with mnc's
> > initial patch + latest open-iscsi.git userspace.
> >
> > iscsiadm is currently defaulting to tcp for all sendtargets discovery:
> >
> > root@barret:# /sbin/iscsiadm -m discovery -t sendtargets --portal
> > 10.100.0.1:3261 -I iser -d 4
> > Following the debug pointer, CAP_TEXT_NEGO appears to be missing from
> > session->t->caps during iscsi_create_session() setup usage:
> >
> > iscsiadm: iscsi_create_session discovery ep connect iser, caps: 0x0009
> >
> > Any reason why userspace would not matching iscsi_iser_transport->caps
> > w/ CAP_TEXT_NEGO set (0x0089)..?
>
> Hi Nic, Mike
>
> in my environment (user space running latest open-iscsi tools, kernel in
> both initiator and target based on your "queue" branch (3.10-rc5) +
> Nic'sLIO patches to enable TEXT / Sendtargets support by iser + the
> initiator patch that advertizes CAP_TEXT_NEGO --the user space
> initiator tools do see the text cap and attempt to issue the discovery
> session through iser, but this somehow fails with LIO, with the
> following print which I have already reported earlier:
>
> xena017-3 kernel: i_buf: iqn.1994-05.com.redhat:308a2565a30, s_buf:
> Discovery, t_buf: (null)
> xena017-3 kernel: Unable to accept text parameter length: 16greater than
> MaxXmitDataSegmentLength 0.
>
>
So I'm pretty sure this is due to iscsi_target_parameters.c:
iscsi_set_keys_irrelevant_for_discovery() currently clearing
INITIATORRECVDATASEGMENTLENGTH and TARGETRECVDATASEGMENTLENGTH for
all discovery scenarios..
As I'm still not yet able to force iser discovery to occur on the
initiator side, can you give the following patch a shot on your setup..?
--nab
diff --git a/drivers/target/iscsi/iscsi_target_login.c
b/drivers/target/iscsi/iscsi_target_login.c
index bb5d5c5..f42eb84 100644
--- a/drivers/target/iscsi/iscsi_target_login.c
+++ b/drivers/target/iscsi/iscsi_target_login.c
@@ -369,7 +369,7 @@ static int iscsi_login_zero_tsih_s2(
if (sess->sess_ops->SessionType)
return iscsi_set_keys_irrelevant_for_discovery(
- conn->param_list);
+ conn->param_list, iser);
na = iscsit_tpg_get_node_attrib(sess);
diff --git a/drivers/target/iscsi/iscsi_target_parameters.c
b/drivers/target/iscsi/iscsi_target_para
index e382221..0b8c819 100644
--- a/drivers/target/iscsi/iscsi_target_parameters.c
+++ b/drivers/target/iscsi/iscsi_target_parameters.c
@@ -545,7 +545,8 @@ int iscsi_set_keys_to_negotiate(
}
int iscsi_set_keys_irrelevant_for_discovery(
- struct iscsi_param_list *param_list)
+ struct iscsi_param_list *param_list,
+ bool iser)
{
struct iscsi_param *param;
@@ -582,10 +583,15 @@ int iscsi_set_keys_irrelevant_for_discovery(
param->state &= ~PSTATE_NEGOTIATE;
else if (!strcmp(param->name, RDMAEXTENSIONS))
param->state &= ~PSTATE_NEGOTIATE;
- else if (!strcmp(param->name, INITIATORRECVDATASEGMENTLENGTH))
+ else if (!strcmp(param->name, INITIATORRECVDATASEGMENTLENGTH)) {
+ if (!iser)
+ continue;
param->state &= ~PSTATE_NEGOTIATE;
- else if (!strcmp(param->name, TARGETRECVDATASEGMENTLENGTH))
+ } else if (!strcmp(param->name, TARGETRECVDATASEGMENTLENGTH)) {
+ if (!iser)
+ continue;
param->state &= ~PSTATE_NEGOTIATE;
+ }
}
return 0;
diff --git a/drivers/target/iscsi/iscsi_target_parameters.h
b/drivers/target/iscsi/iscsi_target_para
index a47046a..76d2fdf 100644
--- a/drivers/target/iscsi/iscsi_target_parameters.h
+++ b/drivers/target/iscsi/iscsi_target_parameters.h
@@ -30,7
Re: do discovery through SW transports too
On Thu, 2013-06-20 at 19:01 +0300, Or Gerlitz wrote: > On 19/06/2013 04:32, Nicholas A. Bellinger wrote: > > So I'm pretty sure this is due to iscsi_target_parameters.c: > > iscsi_set_keys_irrelevant_for_discovery() currently clearing > > INITIATORRECVDATASEGMENTLENGTH and TARGETRECVDATASEGMENTLENGTH for > > all discovery scenarios.. > > > > As I'm still not yet able to force iser discovery to occur on the > > initiator side, can you give the following patch a shot on your setup..? > > > > --nab > > Nic, > > I see that this patch exists in the queue branch of your target-pending > tree, so I tested with that branch and still faced problems. So what > happens now is that the login of the discovery session works fine! , thanks for verifying that particular bit.. > But, the sending over the IB connection of the TEXT PDU containing > Sendtargets=Allwhich is 16 byte long > > iscsiadm: in kstart_conn > iscsiadm: in __kipc_call > iscsiadm: in kwritev > iscsiadm: in nlpayload_read > iscsiadm: in nlpayload_read > iscsiadm: sending text pdu with CmdSN 1, exp_statsn 1 > iscsiadm: >SendTargets=All > iscsiadm: in ksend_pdu_begin > iscsiadm: send PDU began for hdr 48 bytes and data 16 bytes > iscsiadm: in kwritev > iscsiadm: wrote 48 bytes of PDU header > iscsiadm: in kwritev > iscsiadm: wrote 16 bytes of PDU data > iscsiadm: in ksend_pdu_end > iscsiadm: in __kipc_call > iscsiadm: in kwritev > iscsiadm: in nlpayload_read > iscsiadm: in nlpayload_read > iscsiadm: send PDU finished for conn 28:0 > > > > fails, that is the kernel iser driver gets TX completion with error > IB_WC_REM_OP_ERR > > iser: iser_drain_tx_cq:tx id 88006facac98 status 11 vend_err 89 > > > Is there a chance you don't post RX buffer which is large enough to hold > these 48 + 16 bytes in case this is discovery and not normal session? > > AFAICT, the isert_put_login_tx() -> isert_alloc_rx_descriptors() -> isert_post_recv() is happening for both types of sessions, before the last login response is posted and login state moves to full feature phase. Can you post a target side dump with dynamic debugging enabled with the following..? echo 'module iscsi_target_mod +p' > /debug/dynamic_debug/control echo 'module ib_isert +p' > /debug/dynamic_debug/control --nab -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/groups/opt_out.
Re: do discovery through SW transports too
On Sun, 2013-06-23 at 17:47 +0300, Or Gerlitz wrote:
> On 23/06/2013 17:40, Or Gerlitz wrote:
> >
> >
> > there you go, here's the output
> >
> > isert_cma_handler: event 4 status 0 conn 88011a55d600 id
> > 8801085c5400
> > RDMA_CM_EVENT_CONNECT_REQUEST: >>>
> > Entering isert_connect_request cma_id: 8801085c5400, context:
> > 88011a55d600
> > Using responder_resources: 1 initiator_depth: 4
> > Set login_buf: 880116a38000 login_req_buf: 880116a38000
> > login_rsp_buf: 880116a3a000
> > Using 3 CQs, device mlx4_0 supports 3 vectors
> [...]
>
> and here's the initiator output (if you set debug_level=3 for ib_iser)
>
> > iser: iser_rcv_completion:op 0x3f itt 0x dlen 0
> > iser: iscsi_iser_recv:wrong datalen 48 (hdr), 0 (IB)
> >
>
> This means that LIO sent ISCSI_OP_REJECT (0x3f) and that there's a bug
> for reject since the iSCSI header
> says there are 48 bytes beyond the BHS but on the wire [1] there where none
>
Hi Or & Co,
Ok, sendtargets discovery is now functioning over iser. The updated
target patches + v2 changelogs are in for-next commits here:
http://git.kernel.org/cgit/linux/kernel/git/nab/target-pending.git/commit/?h=for-next&id=889c8a68b8483a8b3482ac440af3ad7368c58555
http://git.kernel.org/cgit/linux/kernel/git/nab/target-pending.git/commit/?h=for-next&id=4ebc2e95f49aca3114acb13b15c3e6f769ee6300
http://git.kernel.org/cgit/linux/kernel/git/nab/target-pending.git/commit/?h=for-next&id=58203ef7ae06f17f7ee238491cd7301abe3dfc19
The running initiator + target discovery output is below. Please go
ahead and give it a shot on your setup.
One thing to note, this currently *only* works with open-iscsi when the
TEXT_RSP payload posted by the target is <= 128 bytes (eg: equal to or
less than the initiator posted ISER_RECV_DATA_SEG_LEN), otherwise a
IB_WC_LOC_LEN_ERR error will be generated by the initiator..
So that said, I think we need to go ahead and bump the open-iscsi side
ISER_RECV_DATA_SEG_LEN constant to 8192 in order to match what is
actually advertised by the initiator's MaxRecvDataSegmentLength during
discovery negotiation.
Or and Mike, here is a quick patch on the initiator side.. Ideally this
would follow the negotiated MRDSL, but it's certainly better than
hitting IB_WC_LOC_LEN_ERR errors during (most) typical sendtargets
cases. WDYT..?
diff --git a/drivers/infiniband/ulp/iser/iscsi_iser.h
b/drivers/infiniband/ulp/iser/iscsi_iser.h
index 4f069c0..2fab32c 100644
--- a/drivers/infiniband/ulp/iser/iscsi_iser.h
+++ b/drivers/infiniband/ulp/iser/iscsi_iser.h
@@ -153,7 +153,7 @@ struct iser_cm_hdr {
/* Constant PDU lengths calculations */
#define ISER_HEADERS_LEN (sizeof(struct iser_hdr) + sizeof(struct iscsi_hdr))
-#define ISER_RECV_DATA_SEG_LEN 128
+#define ISER_RECV_DATA_SEG_LEN 8192
#define ISER_RX_PAYLOAD_SIZE (ISER_HEADERS_LEN + ISER_RECV_DATA_SEG_LEN)
#define ISER_RX_LOGIN_SIZE (ISER_HEADERS_LEN + ISCSI_DEF_MAX_RECV_SEG_LEN)
Also, as you reported above there is still a problem somewhere with the
posting/handling of REJECTs PDUs. I'm currently tracking this down, and
is a bug separate from the v3.11 changes.
More on that soon..
--nab
Initiator side:
[ 251.926481] iscsi: registered transport (iser)
[ 251.926705] caps for iscsi transport: 0x0089
[ 296.946755] iser: iser_connect:connecting to: 10.100.0.1, port 0xbd0c
[ 296.951793] iser: iscsi_iser_ep_poll:ib conn 8802354699b8 rc = 0
[ 296.952319] iser: iser_cma_handler:event 0 status 0 conn 8802354699b8 id
880234325c00
[ 296.952737] iser: iser_create_device_ib_res:using 4 CQs, device mlx4_0
supports 19 vectors
[ 296.968061] iser: iser_cma_handler:event 2 status 0 conn 8802354699b8 id
880234325c00
[ 296.978296] iser: iser_create_ib_conn_res:cq index 0 used for ib_conn
8802354699b8
[ 296.979763] iser: iser_create_ib_conn_res:setting conn 8802354699b8
cma_id 880234325c00: fmr_pool 880236a23640 qp 8802346c6e00
[ 297.011751] iser: iser_cma_handler:event 9 status 0 conn 8802354699b8 id
880234325c00
[ 297.954147] iser: iscsi_iser_ep_poll:ib conn 8802354699b8 rc = 1
[ 297.954614] scsi6 : iSCSI Initiator over iSER, v.0.1
[ 297.955350] iser: iscsi_iser_conn_bind:binding iscsi/iser conn
88022b7fadf0 88022b7fafd0 to ib_conn 8802354699b8
[ 297.955807] iser: iscsi_iser_mtask_xmit:mtask xmit [cid 0 itt 0x0]
[ 297.955810] iser: iser_post_rx_bufs:req op 43 flags 87
[ 297.955811] iser: iser_post_rx_bufs:Initially post: 32
[ 297.966140] iser: iser_rcv_completion:op 0x23 itt 0x0 dlen 176
[ 297.966143] iser: iscsi_iser_recv:aligned datalen (173) hdr, 176 (IB)
[ 297.966152] iser: iser_cq_tasklet_fn:got 1 rx 1 tx completions
[ 297.966576] iser: iscsi_iser_mtask_xmit:mtask xmit [cid 0 itt 0x0]
[ 297.966581] iser: iser_post_rx_bufs:req op 4 flags 80
[ 297.982123] iser: iser_rcv_completion:op 0x24 itt 0x0 dlen 84
[ 297.982126] iser: iscsi_iser_recv:aligned datalen (83) hdr, 84 (IB)
[ 297.9
Re: iscsiadm login failed
Hi Saeed, On Fri, 2013-08-30 at 23:19 -0700, Saeed Mirzamohamadi wrote: > Hi > I'm trying to connect my open-iscsi initiator to target, but failed > to login with the command "sudo iscsiadm -m node --login". Here is my > error: > > > Logging in to [iface: default, target: > iqn.2003-01.org.linux-iscsi.ubuntu.i686:sn.1fe6a7cbc123, portal: > 192.168.157.161,3260] > Logging in to [iface: default, target: > iqn.2003-01.org.linux-iscsi.ubuntu.i686:sn.d0a998675819, portal: > 192.168.157.160,3260] > iscsiadm: Could not login to [iface: default, target: > iqn.2003-01.org.linux-iscsi.ubuntu.i686:sn.1fe6a7cbc123, portal: > 192.168.157.161,3260]: > iscsiadm: initiator reported error (19 - encountered non-retryable > iSCSI login failure) > iscsiadm: Could not login to [iface: default, target: > iqn.2003-01.org.linux-iscsi.ubuntu.i686:sn.d0a998675819, portal: > 192.168.157.160,3260]: > iscsiadm: initiator reported error (8 - connection timed out) > > I'm guessing that you've not setup CHAP authentication properly for the initiator NodeACL listed below, or disabled authentication enforcement via: set attribute authentication=0 from within TPG context. You might want to have a look at: http://www.linux-iscsi.org/wiki/ISCSI#Define_access_rights --nab > I'm using instructions in the following link: > https://help.ubuntu.com/lts/serverguide/iscsi-initiator.html > > > > iscsiadm discovery is OK too. > > > and my target is LIO. It's fine and my targetcli shows this: > o- / . > [...] > o- > backstores .. > [...] > | o- fileio ... [0 > Storage Object] > | o- iblock ... [0 > Storage Object] > | o- pscsi [0 > Storage Object] > | o- rd_dr [0 > Storage Object] > | o- rd_mcp ... [1 > Storage Object] > | o- saeedRAM .. [ramdisk > activated] > o- > iscsi .. [1 > Target] > | o- > iqn.2003-01.org.linux-iscsi.ubuntu.i686:sn.1fe6a7cbc123 . [1 > TPG] > | o- tpgt1 ... > [enabled] > | o- > acls [1 ACL] > | | o- iqn.1993-08.org.debian:01:3db9b7d39af .. [1 > Mapped LUN] > | | o- > mapped_lun0 . [lun0 (rw)] > | o- > luns [1 LUN] > | | o- lun0 .. [rd_mcp/saeedRAM > (ramdisk)] > | o- portals .. > [1 Portal] > | o- > 192.168.157.161:3260 . [OK] > o- loopback ... > [0 Target] > o- tcm_fc . > [0 Target] > > > What might be the problem? I have searched a lot but didn't find > anything beneficial. > > > Additionally I'm using both initiator and target in same computer. > My system: Linux ubuntu 12.04 LTS. > > > Thanks in advance! > > > -- > You received this message because you are subscribed to the Google > Groups "open-iscsi" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to open-iscsi+unsubscr...@googlegroups.com. > To post to this group, send email to open-iscsi@googlegroups.com. > Visit this group at http://groups.google.com/group/open-iscsi. > For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/groups/opt_out.
scsi-mq + open-iscsi support patches..?
Hi Mike, Just curious as to the status of the scsi-mq conversion patches for open-iscsi that you where working on a while back..? I'm asking because recently I've been spending alot of time with scsi-mq + virtio-scsi, and would really to start banging on iser-target ports using an scsi-mq enabled iser-initiator as well.. So that said, are there any blockers holding things up towards getting a working prototype..? --nab -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/groups/opt_out.
Re: scsi-mq + open-iscsi support patches..?
On Fri, 2013-11-01 at 09:46 -0500, Mike Christie wrote: > On 10/28/13 5:03 PM, Nicholas A. Bellinger wrote: > > Hi Mike, > > > > Just curious as to the status of the scsi-mq conversion patches for > > open-iscsi that you where working on a while back..? > > > > I'm asking because recently I've been spending alot of time with scsi-mq > > + virtio-scsi, and would really to start banging on iser-target ports > > using an scsi-mq enabled iser-initiator as well.. > > > > So that said, are there any blockers holding things up towards getting a > > working prototype..? > > > > Just time. Just finished a release at work and just got back from > vacation so hoping to work on this some more in the next weeks. > Btw, I'm happy to help move things along here too, and have a few extra cycles to spare.. > On a related note, some iscsi vendor has been hitting a crash with your > tree. I got an email from Jayamohan recently, but the OOPs did not appear to be scsi-mq related.. > Have you rebased it it to take in Christoph's changes and fixes > that are in Jens's tree? The code here > > https://git.kernel.org/cgit/linux/kernel/git/nab/target-pending.git/log/?h=scsi-mq > > looks ok. That is the current code right? > It's current as of a few weeks ago, but does not include the very latest blk-mq code that's now in next from Jens. I'll be rebasing once blk-mq hits mainline over the next weeks.. ;) --nab -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/groups/opt_out.
Re: some iscsi vendor has been hitting a crash
On Fri, 2013-11-01 at 22:28 +0100, Thomas Glanzmann wrote: > Hello Mike, > > * Mike Christie [2013-11-01 15:56]: > > On a related note, some iscsi vendor has been hitting a crash with > > your tree. > > I hit an 'IO stall' with the tree, but was not able to reproduce this. > Has this vendor information on it (dmesg, OOPS)? I reverted back to > RTSOS for production classes that I do not give myself, but will run the > Nab's tree as soon as I give a VMware class myself in order to reproduce > the issue and than track it down. > Hey Thomas, Mike's talking about the WIP scsi-mq -> blk-mq initiator stack here, and not specifically about anything related to target code. --nab -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/groups/opt_out.
Re: scsi-mq + open-iscsi support patches..?
On Sat, 2013-11-02 at 16:10 +, Jayamohan Kallickal wrote: > >> On a related note, some iscsi vendor has been hitting a crash with > >> your tree. > > >I got an email from Jayamohan recently, but the OOPs did not appear to be > >scsi-mq related.. > > I do see the crash on my Ubuntu VM running on VirtualBox but don't see the > crash on a Standalone system. > Attaching the screenshot > Hi Jayamohan, Care to enable the virtual serial port for this VM in order to grab the full dmesg output..? Also, an idea of what SATA emulation is being used here would be helpful as well. (eg: lspci -vvv for the ATA device on the running system) --nab -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/groups/opt_out.
Re: [LSF/MM TOPIC] iSCSI MQ adoption via MCS discussion
On Thu, 2015-01-08 at 14:29 -0800, James Bottomley wrote: > On Thu, 2015-01-08 at 14:16 -0800, Nicholas A. Bellinger wrote: > > On Thu, 2015-01-08 at 08:50 +0100, Bart Van Assche wrote: > > > On 01/07/15 22:39, Mike Christie wrote: > > > > On 01/07/2015 10:57 AM, Hannes Reinecke wrote: > > > >> On 01/07/2015 05:25 PM, Sagi Grimberg wrote: > > > >>> Hi everyone, > > > >>> > > > >>> Now that scsi-mq is fully included, we need an iSCSI initiator that > > > >>> would use it to achieve scalable performance. The need is even greater > > > >>> for iSCSI offload devices and transports that support multiple HW > > > >>> queues. As iSER maintainer I'd like to discuss the way we would choose > > > >>> to implement that in iSCSI. > > > >>> > > > >>> My measurements show that iSER initiator can scale up to ~2.1M IOPs > > > >>> with multiple sessions but only ~630K IOPs with a single session where > > > >>> the most significant bottleneck the (single) core processing > > > >>> completions. > > > >>> > > > >>> In the existing single connection per session model, given that > > > >>> command > > > >>> ordering must be preserved session-wide, we end up in a serial command > > > >>> execution over a single connection which is basically a single queue > > > >>> model. The best fit seems to be plugging iSCSI MCS as a multi-queued > > > >>> scsi LLDD. In this model, a hardware context will have a 1x1 mapping > > > >>> with an iSCSI connection (TCP socket or a HW queue). > > > >>> > > > >>> iSCSI MCS and it's role in the presence of dm-multipath layer was > > > >>> discussed several times in the past decade(s). The basic need for MCS > > > >>> is > > > >>> implementing a multi-queue data path, so perhaps we may want to avoid > > > >>> doing any type link aggregation or load balancing to not overlap > > > >>> dm-multipath. For example we can implement ERL=0 (which is basically > > > >>> the > > > >>> scsi-mq ERL) and/or restrict a session to a single portal. > > > >>> > > > >>> As I see it, the todo's are: > > > >>> 1. Getting MCS to work (kernel + user-space) with ERL=0 and a > > > >>> round-robin connection selection (per scsi command execution). > > > >>> 2. Plug into scsi-mq - exposing num_connections as nr_hw_queues and > > > >>> using blk-mq based queue (conn) selection. > > > >>> 3. Rework iSCSI core locking scheme to avoid session-wide locking > > > >>> as much as possible. > > > >>> 4. Use blk-mq pre-allocation and tagging facilities. > > > >>> > > > >>> I've recently started looking into this. I would like the community to > > > >>> agree (or debate) on this scheme and also talk about implementation > > > >>> with anyone who is also interested in this. > > > >>> > > > >> Yes, that's a really good topic. > > > >> > > > >> I've pondered implementing MC/S for iscsi/TCP but then I've figured my > > > >> network implementation knowledge doesn't spread that far. > > > >> So yeah, a discussion here would be good. > > > >> > > > >> Mike? Any comments? > > > > > > > > I have been working under the assumption that people would be ok with > > > > MCS upstream if we are only using it to handle the issue where we want > > > > to do something like have a tcp/iscsi connection per CPU then map the > > > > connection to a blk_mq_hw_ctx. In this more limited MCS implementation > > > > there would be no iscsi layer code to do something like load balance > > > > across ports or transport paths like how dm-multipath does, so there > > > > would be no feature/code duplication. For balancing across hctxs, then > > > > the iscsi layer would also leave that up to whatever we end up with in > > > > upper layers, so again no feature/code duplication with upper layers. > > > > > > > > So pretty non controversial I hope :) > > > > > > > > If people want to add something like round robin connection selection in > &g
Re: [PATCH 24/29] drivers: convert iblock_req.pending from atomic_t to refcount_t
Hi Elena, On Mon, 2017-03-06 at 16:21 +0200, Elena Reshetova wrote: > refcount_t type and corresponding API should be > used instead of atomic_t when the variable is used as > a reference counter. This allows to avoid accidental > refcounter overflows that might lead to use-after-free > situations. > > Signed-off-by: Elena Reshetova > Signed-off-by: Hans Liljestrand > Signed-off-by: Kees Cook > Signed-off-by: David Windsor > --- > drivers/target/target_core_iblock.c | 12 ++-- > drivers/target/target_core_iblock.h | 3 ++- > 2 files changed, 8 insertions(+), 7 deletions(-) For the target_core_iblock part: Acked-by: Nicholas Bellinger -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at https://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
Re: [PATCH 24/29] drivers: convert iblock_req.pending from atomic_t to refcount_t
Hi Elena, On Mon, 2017-03-06 at 16:21 +0200, Elena Reshetova wrote: > refcount_t type and corresponding API should be > used instead of atomic_t when the variable is used as > a reference counter. This allows to avoid accidental > refcounter overflows that might lead to use-after-free > situations. > > Signed-off-by: Elena Reshetova > Signed-off-by: Hans Liljestrand > Signed-off-by: Kees Cook > Signed-off-by: David Windsor > --- > drivers/target/target_core_iblock.c | 12 ++-- > drivers/target/target_core_iblock.h | 3 ++- > 2 files changed, 8 insertions(+), 7 deletions(-) After reading up on this thread, it looks like various subsystem maintainers are now picking these atomic_t -> refcount_t conversions.. That said, applied to target-pending/for-next and will plan to include for v4.12-rc1 merge window. Thanks! -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at https://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
Re: [kernel-hardening] Re: [PATCH v4 06/13] iscsi: ensure RNG is seeded before use
Hi Stephan, Lee & Jason, (Adding target-devel CC') Apologies for coming late to the discussion. Comments below. On Sun, 2017-06-18 at 10:04 +0200, Stephan Müller wrote: > Am Samstag, 17. Juni 2017, 05:45:57 CEST schrieb Lee Duncan: > > Hi Lee, > > > In your testing, how long might a process have to wait? Are we talking > > seconds? Longer? What about timeouts? > > > > In current kernels (starting with 4.8) this timeout should clear within a few > seconds after boot. > > In older kernels (pre 4.8), my KVM takes up to 90 seconds to reach that > seeding point. I have heard that on IBM System Z this trigger point requires > minutes to be reached. > I share the same concern as Lee wrt to introducing latency into the existing iscsi-target login sequence. Namely in the use-cases where a single node is supporting ~1K unique iscsi-target IQNs, and fail-over + re-balancing of IQNs where 100s of login attempts are expected to occur in parallel. If environments exist that require non trivial amounts of time for RNG seeding to be ready for iscsi-target usage, then enforcing this requirement at iscsi login time can open up problems, especially when iscsi host environments may be sensitive to login timeouts, I/O timeouts, et al. That said, I'd prefer to simply wait for RNG to be seeded at modprobe iscsi_target_mod time, instead of trying to enforce randomness during login. This way, those of use who have distributed storage platforms can know to avoid putting a node into a state to accept iscsi-target IQN export migration, before modprobe iscsi_target_mod has successfully loaded and RNG seeding has been confirmed. WDYT..? -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at https://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
Re: [kernel-hardening] Re: [PATCH v4 06/13] iscsi: ensure RNG is seeded before use
On Mon, 2017-06-26 at 19:38 +0200, Stephan Müller wrote: > Am Montag, 26. Juni 2017, 03:23:09 CEST schrieb Nicholas A. Bellinger: > > Hi Nicholas, > > > Hi Stephan, Lee & Jason, > > > > (Adding target-devel CC') > > > > Apologies for coming late to the discussion. Comments below. > > > > On Sun, 2017-06-18 at 10:04 +0200, Stephan Müller wrote: > > > Am Samstag, 17. Juni 2017, 05:45:57 CEST schrieb Lee Duncan: > > > > > > Hi Lee, > > > > > > > In your testing, how long might a process have to wait? Are we talking > > > > seconds? Longer? What about timeouts? > > > > > > In current kernels (starting with 4.8) this timeout should clear within a > > > few seconds after boot. > > > > > > In older kernels (pre 4.8), my KVM takes up to 90 seconds to reach that > > > seeding point. I have heard that on IBM System Z this trigger point > > > requires minutes to be reached. > > > > I share the same concern as Lee wrt to introducing latency into the > > existing iscsi-target login sequence. > > > > Namely in the use-cases where a single node is supporting ~1K unique > > iscsi-target IQNs, and fail-over + re-balancing of IQNs where 100s of > > login attempts are expected to occur in parallel. > > > > If environments exist that require non trivial amounts of time for RNG > > seeding to be ready for iscsi-target usage, then enforcing this > > requirement at iscsi login time can open up problems, especially when > > iscsi host environments may be sensitive to login timeouts, I/O > > timeouts, et al. > > > > That said, I'd prefer to simply wait for RNG to be seeded at modprobe > > iscsi_target_mod time, instead of trying to enforce randomness during > > login. > > > > This way, those of use who have distributed storage platforms can know > > to avoid putting a node into a state to accept iscsi-target IQN export > > migration, before modprobe iscsi_target_mod has successfully loaded and > > RNG seeding has been confirmed. > > > > WDYT..? > > We may have a chicken and egg problem when the wait is at the modprobe time. > Assume the modprobe happens during initramfs time to get access to the root > file system. In this case, you entire boot process will lock up for an > indefinite amount of time. The reason is that in order to obtain events > detected by the RNG, devices need to be initialized and working. Such devices > commonly start working after the the root partition is mounted as it contains > all drivers, all configuration information etc. > > Note, during the development of my /dev/random implementation, I added the > getrandom-like blocking behavior to /dev/urandom (which is the equivalent to > Jason's patch except that it applies to user space). The boot process locked > up since systemd wanted data from /dev/urandom while it processed the > initramfs. As it did not get any, the boot process did not commence that > could > deliver new events to be picked up by the RNG. > > As I do not have such an iscsi system, I cannot test Jason's patch. But maybe > the mentioned chicken-and-egg problem I mentioned above is already visible > with the current patch as it will lead to a blocking of the mounting of the > root partition in case the root partition is on an iscsi target. AFAIK, there are no distro initramfs dependencies for iscsi_target_mod, and every environment I've ever seen loads iscsi_target_mod after switching to the real rootfs. For an iscsi initiator that might not been the case, especially if the rootfs is running atop a iscsi LUN. But at least for iscsi-target mode, any blocking during modprobe waiting for RNG seeding would happen outside of initramfs. -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at https://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
