open-iscsi configuration for wasabi storage builder
Hello, I'm interested to see if anybody has a working configuration for open-iscsi initiator with a wasabi storage builder target. We have a few Wasabi Storage Builder 1.6 targets, which we've used for years with linux-iscsi and core-iscsi initiators. Recently we decided to repurpose them, and use a newer OS for the initiator role. Unfortunately, we can't seem to get it going properly. The targets are detected correctly, we're able to see the target LUNs, but any attempt to write more than a few megabytes of data fails. Initially the target would show us: --- thread main:target.c:kq_conn:1610: ***ERROR*** Detected packet with illegal packet size 16777215 (max is 8192) from iqn.2008-06.draugluin,i,00023d06 to iqn.2000-05.com.wasabisystems.storagebuilder:iscsi-6-0,t,0032 cid 0 PDU was number 77 on the connection, and started at byte 12352 --- Setting this option helped a little bit, but now we get kernel bug errors: node.conn[0].iscsi.MaxRecvDataSegmentLength = 8192 Here's our current config: --- node.name = iqn.2000-05.com.wasabisystems.storagebuilder:iscsi-6-0 node.tpgt = 1 node.startup = automatic iface.hwaddress = default iface.iscsi_ifacename = default iface.net_ifacename = default iface.transport_name = tcp node.discovery_address = 10.0.0.16 node.discovery_port = 3260 node.discovery_type = send_targets node.session.initial_cmdsn = 0 node.session.initial_login_retry_max = 4 node.session.cmds_max = 128 node.session.queue_depth = 32 node.session.auth.authmethod = None node.session.timeo.replacement_timeout = 120 node.session.err_timeo.abort_timeout = 10 node.session.err_timeo.reset_timeout = 30 node.session.iscsi.InitialR2T = No node.session.iscsi.ImmediateData = Yes node.session.iscsi.FirstBurstLength = 262144 node.session.iscsi.MaxBurstLength = 16776192 node.session.iscsi.DefaultTime2Retain = 0 node.session.iscsi.DefaultTime2Wait = 0 node.session.iscsi.MaxConnections = 1 node.session.iscsi.MaxOutstandingR2T = 1 node.session.iscsi.ERL = 0 node.conn[0].address = 10.0.0.16 node.conn[0].port = 3260 node.conn[0].startup = manual node.conn[0].tcp.window_size = 524288 node.conn[0].tcp.type_of_service = 0 node.conn[0].timeo.logout_timeout = 15 node.conn[0].timeo.login_timeout = 15 node.conn[0].timeo.auth_timeout = 45 node.conn[0].timeo.active_timeout = 5 node.conn[0].timeo.idle_timeout = 60 node.conn[0].timeo.ping_timeout = 5 node.conn[0].timeo.noop_out_interval = 10 node.conn[0].timeo.noop_out_timeout = 15 node.conn[0].iscsi.MaxRecvDataSegmentLength = 8192 node.conn[0].iscsi.HeaderDigest = None,CRC32C node.conn[0].iscsi.DataDigest = None node.conn[0].iscsi.IFMarker = No node.conn[0].iscsi.OFMarker = No --- Logs on the initiator side logs showed us initially: --- Jun 26 11:06:09 draugluin iscsid: Nop-out timedout after 15 seconds on connection 2:0 state (3). Dropping session. Jun 26 11:06:09 draugluin iscsid: Nop-out timedout after 15 seconds on connection 3:0 state (3). Dropping session. Jun 26 11:06:09 draugluin iscsid: Nop-out timedout after 15 seconds on connection 1:0 state (3). Dropping session. Jun 26 11:06:10 draugluin iscsid: connection3:0 is operational after recovery (1 attempts) Jun 26 11:06:10 draugluin iscsid: connection2:0 is operational after recovery (1 attempts) Jun 26 11:06:10 draugluin iscsid: connection1:0 is operational after recovery (1 attempts) Jun 26 11:06:18 draugluin iscsid: Nop-out timedout after 15 seconds on connection 6:0 state (3). Dropping session. Jun 26 11:06:19 draugluin iscsid: connection6:0 is operational after recovery (1 attempts) Jun 26 11:06:25 draugluin iscsid: Nop-out timedout after 15 seconds on connection 5:0 state (3). Dropping session. Jun 26 11:06:25 draugluin iscsid: connection5:0 is operational after recovery (1 attempts) Jun 26 11:06:45 draugluin iscsid: Nop-out timedout after 15 seconds on connection 6:0 state (3). Dropping session. Jun 26 11:06:45 draugluin iscsid: connection6:0 is operational after recovery (1 attempts) --- Right now writing any larger amount of data [more than a few kilobytes] results in: --- Jun 26 14:55:33 draugluin kernel: === Jun 26 14:55:33 draugluin kernel: BUG: workqueue leaked lock or atomic: scsi_wq_5/0x0001/5316 Jun 26 14:55:33 draugluin kernel: last function: iscsi_xmitworker+0x0/0x584 [libiscsi] Jun 26 14:55:33 draugluin kernel: [] run_workqueue+0xcf/0x114 Jun 26 14:55:33 draugluin kernel: [] worker_thread+0x0/0xca Jun 26 14:55:33 draugluin kernel: [] worker_thread+0xbc/0xca Jun 26 14:55:33 draugluin kernel: [] autoremove_wake_function+0x0/0x33 Jun 26 14:55:33 draugluin kernel: [] worker_thread+0x0/0xca Jun 26 14:55:33 draugluin kernel: [] kthread+0x38/0x5e Jun 26 14:55:33 draugluin kernel: [] kthread+0x0/0x5e Jun 26 14:55:33 draugluin kernel: [] kernel_thread_helper+0x7/0x10 Jun 26 14:55:33 draugluin kernel: === --- We're using OpenSUSE 10.3 with open-iscsi-2.0.866-15.2 and 2.6.22.18-0.2-default SMP i686 kernel. [OpenSUSE
Re: open-iscsi configuration for wasabi storage builder
Dominik L. Borkowski wrote: > > We're using OpenSUSE 10.3 with open-iscsi-2.0.866-15.2 and > 2.6.22.18-0.2-default SMP i686 kernel. [OpenSUSE 11 with > open-iscsi-2.0.869-8.1 had another set of issues, we couldn't even login > properly]. > Weird. We had this guy that tested the heck with wasabi, and had not seen any bug reports about it for a long time. Now in this last week we have two or three bug reports on it. Has wasabi done a firmware update or are you using a older firmware? I am ccing another users that reported issues with wasabi the other day. Ken, what firmware are you using? There definately seems to be an issue with the initiator sending nops. For Ken's problem we turned them off and that fixed the flood of these: Jun 26 11:06:18 draugluin iscsid: Nop-out timedout after 15 seconds on connection 6:0 state (3). Dropping session. Jun 26 11:06:19 draugluin iscsid: connection6:0 is operational after recovery and the slow down that results. But Ken still has a slown down in IO. Dominik, to turn off nops set node.conn[0].timeo.noop_out_interval = 0 node.conn[0].timeo.noop_out_timeout = 0 You can set that in iscsi.conf then redo discovery to pick them up. But I would like to get the bottom of the problem. Could you guys try http://www.open-iscsi.org/bits/open-iscsi-2.0-869.2.tar.gz With nops back on (set those noop values to 10 for testing). Then could you send me a wireshark/ethereal trace? I do not need a lot of data just when you see the first Jun 26 11:06:18 draugluin iscsid: Nop-out timedout after 15 seconds on connection 6:0 state (3). Dropping session. Jun 26 11:06:19 draugluin iscsid: connection6:0 is operational after recovery in the log, stop the trace and send it. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: open-iscsi configuration for wasabi storage builder
On Friday 27 June 2008 11:44:08 Mike Christie wrote: > Dominik L. Borkowski wrote: > > We're using OpenSUSE 10.3 with open-iscsi-2.0.866-15.2 and > > 2.6.22.18-0.2-default SMP i686 kernel. [OpenSUSE 11 with > > open-iscsi-2.0.869-8.1 had another set of issues, we couldn't even login > > properly]. > > Weird. We had this guy that tested the heck with wasabi, and had not > seen any bug reports about it for a long time. Now in this last week we > have two or three bug reports on it. > > Has wasabi done a firmware update or are you using a older firmware? We use the following firmware: Current WSB iSCSI SAN v1.6.2.1 5565245 Wed Apr 5 01:04:03 UTC 2006 Hasn't been updated in awhile, and I'm not sure what the latest firmware is. I'm contacting the vendor to see whether there is anything newer available. > Could you guys try > http://www.open-iscsi.org/bits/open-iscsi-2.0-869.2.tar.gz I'm not sure how to debug this further (I had to use instructions from http://groups.google.com/group/open-iscsi/browse_thread/thread/a6e5ddf832ed87eb to get past the initial errors), but I can't get it compiled: make[1]: Entering directory `/root/iscsi/open/open-iscsi-2.0-869.2/kernel' make -C /usr/src/linux M=`pwd` KBUILD_OUTPUT=/usr/src/linux-obj/i586/default/ V=0 modules make[2]: Entering directory `/usr/src/linux-2.6.22.18-0.2' CC [M] /root/iscsi/open/open-iscsi-2.0-869.2/kernel/scsi_transport_iscsi.o In file included from /root/iscsi/open/open-iscsi-2.0-869.2/kernel/scsi_transport_iscsi.c:32: /root/iscsi/open/open-iscsi-2.0-869.2/kernel/iscsi_2.6.22_compat.h:75: error: redefinition of ‘scsi_set_resid’ /usr/src/linux-2.6.22.18-0.2/include/scsi/scsi_cmnd.h:148: error: previous definition of ‘scsi_set_resid’ was here /root/iscsi/open/open-iscsi-2.0-869.2/kernel/iscsi_2.6.22_compat.h:80: error: redefinition of ‘scsi_get_resid’ /usr/src/linux-2.6.22.18-0.2/include/scsi/scsi_cmnd.h:153: error: previous definition of ‘scsi_get_resid’ was here make[4]: *** [/root/iscsi/open/open-iscsi-2.0-869.2/kernel/scsi_transport_iscsi.o] Error 1 make[3]: *** [_module_/root/iscsi/open/open-iscsi-2.0-869.2/kernel] Error 2 make[2]: *** [modules] Error 2 make[2]: Leaving directory `/usr/src/linux-2.6.22.18-0.2' make[1]: *** [all] Error 2 make[1]: Leaving directory `/root/iscsi/open/open-iscsi-2.0-869.2/kernel' make: *** [all] Error 2 Would it be worth getting sniffer dump from the existing 2.0-866 initiator? -- Dominik L. Borkowski - Senior Systems Administrator Virginia Bioinformatics Institute - www.vbi.vt.edu --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: open-iscsi configuration for wasabi storage builder
Mike Christie wrote: > Dominik L. Borkowski wrote: >> >> We're using OpenSUSE 10.3 with open-iscsi-2.0.866-15.2 and >> 2.6.22.18-0.2-default SMP i686 kernel. [OpenSUSE 11 with >> open-iscsi-2.0.869-8.1 had another set of issues, we couldn't even >> login properly]. > > Weird. We had this guy that tested the heck with wasabi, and had not > seen any bug reports about it for a long time. Now in this last week we > have two or three bug reports on it. > > Has wasabi done a firmware update or are you using a older firmware? > > I am ccing another users that reported issues with wasabi the other day. > Ken, what firmware are you using? WSB iSCSI SAN v2.3.0.1 7272641 Wed Apr 25 21:58:22 UTC 2007 I'll send a debug log or trace or both shortly. The nops do slow things down, but the big issue seems to be the initiator system lockup when connecting to the Wasabi. fwiw, I tried connecting to an IET target and I don't experience the system freeze/lockup problem. Ken > > There definately seems to be an issue with the initiator sending nops. > For Ken's problem we turned them off and that fixed the flood of these: > > Jun 26 11:06:18 draugluin iscsid: Nop-out timedout after 15 seconds on > connection 6:0 state (3). Dropping session. > Jun 26 11:06:19 draugluin iscsid: connection6:0 is operational after > recovery > > and the slow down that results. > > But Ken still has a slown down in IO. > > Dominik, to turn off nops set > node.conn[0].timeo.noop_out_interval = 0 > node.conn[0].timeo.noop_out_timeout = 0 > > You can set that in iscsi.conf then redo discovery to pick them up. > > But I would like to get the bottom of the problem. > > Could you guys try > http://www.open-iscsi.org/bits/open-iscsi-2.0-869.2.tar.gz > With nops back on (set those noop values to 10 for testing). Then could > you send me a wireshark/ethereal trace? I do not need a lot of data just > when you see the first > > Jun 26 11:06:18 draugluin iscsid: Nop-out timedout after 15 seconds on > connection 6:0 state (3). Dropping session. > Jun 26 11:06:19 draugluin iscsid: connection6:0 is operational after > recovery > > in the log, stop the trace and send it. > -- Ken Anderson Pacific.Net --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: open-iscsi configuration for wasabi storage builder
Dominik L. Borkowski wrote: > On Friday 27 June 2008 11:44:08 Mike Christie wrote: >> Dominik L. Borkowski wrote: >>> We're using OpenSUSE 10.3 with open-iscsi-2.0.866-15.2 and >>> 2.6.22.18-0.2-default SMP i686 kernel. [OpenSUSE 11 with >>> open-iscsi-2.0.869-8.1 had another set of issues, we couldn't even login >>> properly]. >> Weird. We had this guy that tested the heck with wasabi, and had not >> seen any bug reports about it for a long time. Now in this last week we >> have two or three bug reports on it. >> >> Has wasabi done a firmware update or are you using a older firmware? > > We use the following firmware: > > Current WSB iSCSI SAN v1.6.2.1 5565245 Wed Apr 5 01:04:03 UTC > 2006 > > Hasn't been updated in awhile, and I'm not sure what the latest firmware is. > I'm contacting the vendor to see whether there is anything newer available. > >> Could you guys try >> http://www.open-iscsi.org/bits/open-iscsi-2.0-869.2.tar.gz > > I'm not sure how to debug this further (I had to use instructions from > http://groups.google.com/group/open-iscsi/browse_thread/thread/a6e5ddf832ed87eb > > to get past the initial errors), but I can't get it compiled: > Ah yeah, forgot that you were using the SUSE kernel which has some stuff backported. If you cannot try a kernel.org kernel let me try some stuff here. > make[1]: Entering directory `/root/iscsi/open/open-iscsi-2.0-869.2/kernel' > make -C /usr/src/linux M=`pwd` KBUILD_OUTPUT=/usr/src/linux-obj/i586/default/ > > V=0 modules > make[2]: Entering directory `/usr/src/linux-2.6.22.18-0.2' > CC [M] /root/iscsi/open/open-iscsi-2.0-869.2/kernel/scsi_transport_iscsi.o > In file included > from /root/iscsi/open/open-iscsi-2.0-869.2/kernel/scsi_transport_iscsi.c:32: > /root/iscsi/open/open-iscsi-2.0-869.2/kernel/iscsi_2.6.22_compat.h:75: error: > redefinition of ‘scsi_set_resid’ > /usr/src/linux-2.6.22.18-0.2/include/scsi/scsi_cmnd.h:148: error: previous > definition of ‘scsi_set_resid’ was here > /root/iscsi/open/open-iscsi-2.0-869.2/kernel/iscsi_2.6.22_compat.h:80: error: > redefinition of ‘scsi_get_resid’ > /usr/src/linux-2.6.22.18-0.2/include/scsi/scsi_cmnd.h:153: error: previous > definition of ‘scsi_get_resid’ was here > make[4]: *** > [/root/iscsi/open/open-iscsi-2.0-869.2/kernel/scsi_transport_iscsi.o] Error 1 > make[3]: *** [_module_/root/iscsi/open/open-iscsi-2.0-869.2/kernel] Error 2 > make[2]: *** [modules] Error 2 > make[2]: Leaving directory `/usr/src/linux-2.6.22.18-0.2' > make[1]: *** [all] Error 2 > make[1]: Leaving directory `/root/iscsi/open/open-iscsi-2.0-869.2/kernel' > make: *** [all] Error 2 > > Would it be worth getting sniffer dump from the existing 2.0-866 initiator? > Yeah, if you can do it, send it. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: open-iscsi configuration for wasabi storage builder
On Friday 27 June 2008 15:48:13 Mike Christie wrote: > > Would it be worth getting sniffer dump from the existing 2.0-866 > > initiator? I placed the dump at: http://staff.vbi.vt.edu/dom/debug/debug.tar.bz2 In that archive I included tcpdump, sample script session of what commands were issued, initiator configs and the kernel logs. Not sure if joining this thread with Ken's would be a good idea. Somehow I didn't notice his, when I was searching the archive. However, a lot of his symptoms fit my problem, including the os completely freezing. Thank you, dom --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: open-iscsi configuration for wasabi storage builder
Dominik L. Borkowski wrote: > On Friday 27 June 2008 15:48:13 Mike Christie wrote: >>> Would it be worth getting sniffer dump from the existing 2.0-866 >>> initiator? > > > I placed the dump at: > http://staff.vbi.vt.edu/dom/debug/debug.tar.bz2 > > In that archive I included tcpdump, sample script session of what commands > were issued, initiator configs and the kernel logs. > > Not sure if joining this thread with Ken's would be a good idea. Somehow I > didn't notice his, when I was searching the archive. However, a lot of his > symptoms fit my problem, including the os completely freezing. > Jon France at Wasabi looked into the issue, and he thinks this is fixed in newer firmware. He asked you guys to contact Wasabi to get a firmware update. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: open-iscsi configuration for wasabi storage builder
On Tuesday 01 July 2008 12:20:32 Mike Christie wrote: > Jon France at Wasabi looked into the issue, and he thinks this is fixed > in newer firmware. He asked you guys to contact Wasabi to get a firmware > update. Yep, we've been in touch, just waiting to work out some basic details. We'll see how the latest firmware 4.0.2 will behave with openiscsi. Thank you, dom --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
