Re: [ofw] What is an IOC device ID and what is DriverStore
I am also seeing the following Windows events for all eight drives:
Log Name: System
Source:Microsoft-Windows-UserPnp
Date: 8/30/2010 11:58:58 AM
Event ID: 20001
Task Category: (7005)
Level: Information
Keywords:
User: SYSTEM
Computer: WIN-6EB0ETQ8M85
Description:
Driver Management concluded the process to install driver
FileRepository\disk.inf_amd64_neutral_10ce25bbc5a9cc43\disk.inf for
Device Instance ID SCSI\DISKVEN_SCST_BIOPROD_FIO-41001\07 with
the following status: 0x0.
Event Xml:
Event xmlns=http://schemas.microsoft.com/win/2004/08/events/event;
System
Provider Name=Microsoft-Windows-UserPnp
Guid={96F4A050-7E31-453C-88BE-9634F4E02139} /
EventID20001/EventID
Version0/Version
Level4/Level
Task7005/Task
Opcode0/Opcode
Keywords0x8000/Keywords
TimeCreated SystemTime=2010-08-30T17:58:58.86760Z /
EventRecordID6846/EventRecordID
Correlation /
Execution ProcessID=3228 ThreadID=3260 /
ChannelSystem/Channel
ComputerWIN-6EB0ETQ8M85/Computer
Security UserID=S-1-5-18 /
/System
UserData
InstallDeviceID
xmlns:auto-ns2=http://schemas.microsoft.com/win/2004/08/events;
xmlns=http://manifests.microsoft.com/win/2004/08/windows/userpnp;
DriverNameFileRepository\disk.inf_amd64_neutral_10ce25bbc5a9cc43\disk.inf/DriverName
DriverVersion6.1.7600.16385/DriverVersion
DriverProviderMicrosoft/DriverProvider
DeviceInstanceIDSCSI\DISKamp;VEN_SCST_BIOamp;PROD_FIO-41001\07/DeviceInstanceID
SetupClass{4D36E967-E325-11CE-BFC1-08002BE10318}/SetupClass
RebootOptionfalse/RebootOption
UpgradeDevicefalse/UpgradeDevice
IsDriverOEMfalse/IsDriverOEM
InstallStatus0x0/InstallStatus
DriverDescriptionDisk drive/DriverDescription
/InstallDeviceID
/UserData
/Event
Log Name: System
Source:Disk
Date: 8/30/2010 11:58:58 AM
Event ID: 11
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: WIN-6EB0ETQ8M85
Description:
The driver detected a controller error on \Device\Harddisk8\DR8.
Event Xml:
Event xmlns=http://schemas.microsoft.com/win/2004/08/events/event;
System
Provider Name=Disk /
EventID Qualifiers=4915611/EventID
Level2/Level
Task0/Task
Keywords0x80/Keywords
TimeCreated SystemTime=2010-08-30T17:58:58.78960Z /
EventRecordID6845/EventRecordID
ChannelSystem/Channel
ComputerWIN-6EB0ETQ8M85/Computer
Security /
/System
EventData
Data\Device\Harddisk8\DR8/Data
Binary0F0081000B0004C00301822B060058030207FE200A124A0128003C0050B39D0280F8B84B962D80FA70C8992D80FA2500/Binary
/EventData
/Event
Chris
On Fri, Aug 27, 2010 at 6:10 PM, Chris Worley worl...@gmail.com wrote:
On Fri, Aug 27, 2010 at 6:01 PM, Sufficool, Stanley
ssuffic...@rov.sbcounty.gov wrote:
What does your scst.conf look like? I assume you have LUNS in the default
group.
A Linux initiator has no trouble seeing these:
# cat /proc/scsi_tgt/vdisk/vdisk
Name Size(MB) Block size Options File name
T10 device id
fio-71957 76774 512 NV BIO /dev/sda
fio-71957 8d58d2cc
fio-71962 76774 512 NV BIO /dev/sdb
fio-71962 4ab477ef
fio-71965 76774 512 NV BIO /dev/sdc
fio-71965 28ba5c58
fio-71964 76774 512 NV BIO /dev/sdd
fio-71964 f2c69cf6
fio-41000 76774 512 NV BIO /dev/sde
fio-41000 2492daab
fio-41002 76774 512 NV BIO /dev/sdf
fio-41002 dd797f32
fio-40948 76774 512 NV BIO /dev/sdg
fio-40948 a3ef7aa3
fio-41001 76774 512 NV BIO /dev/sdh
fio-41001 543b6d8e
[r...@fusion-io-live .bak]# cat /proc/scsi_tgt/groups/Default/devices
Device (host:ch:id:lun or name) LUN
Options
fio-71957 0
fio-71962 1
fio-71965 2
fio-71964 3
fio-41000 4
fio-41002 5
fio-40948 6
fio-41001
Re: [ofw] What is an IOC device ID and what is DriverStore
Hint:
SCST rev 1901 ends compatibility w/ Windows; r1900 works fine. As
there were a lot of diffs between these two revs, I won't include it
here.
svn diff -r 1900:1901 https://scst.svn.sourceforge.net/svnroot/scst/trunk
Thanks,
Chris
On Mon, Aug 30, 2010 at 12:37 PM, Chris Worley worl...@gmail.com wrote:
I am also seeing the following Windows events for all eight drives:
Log Name: System
Source: Microsoft-Windows-UserPnp
Date: 8/30/2010 11:58:58 AM
Event ID: 20001
Task Category: (7005)
Level: Information
Keywords:
User: SYSTEM
Computer: WIN-6EB0ETQ8M85
Description:
Driver Management concluded the process to install driver
FileRepository\disk.inf_amd64_neutral_10ce25bbc5a9cc43\disk.inf for
Device Instance ID SCSI\DISKVEN_SCST_BIOPROD_FIO-41001\07 with
the following status: 0x0.
Event Xml:
Event xmlns=http://schemas.microsoft.com/win/2004/08/events/event;
System
Provider Name=Microsoft-Windows-UserPnp
Guid={96F4A050-7E31-453C-88BE-9634F4E02139} /
EventID20001/EventID
Version0/Version
Level4/Level
Task7005/Task
Opcode0/Opcode
Keywords0x8000/Keywords
TimeCreated SystemTime=2010-08-30T17:58:58.86760Z /
EventRecordID6846/EventRecordID
Correlation /
Execution ProcessID=3228 ThreadID=3260 /
ChannelSystem/Channel
ComputerWIN-6EB0ETQ8M85/Computer
Security UserID=S-1-5-18 /
/System
UserData
InstallDeviceID
xmlns:auto-ns2=http://schemas.microsoft.com/win/2004/08/events;
xmlns=http://manifests.microsoft.com/win/2004/08/windows/userpnp;
DriverNameFileRepository\disk.inf_amd64_neutral_10ce25bbc5a9cc43\disk.inf/DriverName
DriverVersion6.1.7600.16385/DriverVersion
DriverProviderMicrosoft/DriverProvider
DeviceInstanceIDSCSI\DISKamp;VEN_SCST_BIOamp;PROD_FIO-41001\07/DeviceInstanceID
SetupClass{4D36E967-E325-11CE-BFC1-08002BE10318}/SetupClass
RebootOptionfalse/RebootOption
UpgradeDevicefalse/UpgradeDevice
IsDriverOEMfalse/IsDriverOEM
InstallStatus0x0/InstallStatus
DriverDescriptionDisk drive/DriverDescription
/InstallDeviceID
/UserData
/Event
Log Name: System
Source: Disk
Date: 8/30/2010 11:58:58 AM
Event ID: 11
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: WIN-6EB0ETQ8M85
Description:
The driver detected a controller error on \Device\Harddisk8\DR8.
Event Xml:
Event xmlns=http://schemas.microsoft.com/win/2004/08/events/event;
System
Provider Name=Disk /
EventID Qualifiers=4915611/EventID
Level2/Level
Task0/Task
Keywords0x80/Keywords
TimeCreated SystemTime=2010-08-30T17:58:58.78960Z /
EventRecordID6845/EventRecordID
ChannelSystem/Channel
ComputerWIN-6EB0ETQ8M85/Computer
Security /
/System
EventData
Data\Device\Harddisk8\DR8/Data
Binary0F0081000B0004C00301822B060058030207FE200A124A0128003C0050B39D0280F8B84B962D80FA70C8992D80FA2500/Binary
/EventData
/Event
Chris
On Fri, Aug 27, 2010 at 6:10 PM, Chris Worley worl...@gmail.com wrote:
On Fri, Aug 27, 2010 at 6:01 PM, Sufficool, Stanley
ssuffic...@rov.sbcounty.gov wrote:
What does your scst.conf look like? I assume you have LUNS in the default
group.
A Linux initiator has no trouble seeing these:
# cat /proc/scsi_tgt/vdisk/vdisk
Name Size(MB) Block size Options File name
T10 device id
fio-71957 76774 512 NV BIO /dev/sda
fio-71957 8d58d2cc
fio-71962 76774 512 NV BIO /dev/sdb
fio-71962 4ab477ef
fio-71965 76774 512 NV BIO /dev/sdc
fio-71965 28ba5c58
fio-71964 76774 512 NV BIO /dev/sdd
fio-71964 f2c69cf6
fio-41000 76774 512 NV BIO /dev/sde
fio-41000 2492daab
fio-41002 76774 512 NV BIO /dev/sdf
fio-41002 dd797f32
fio-40948 76774 512 NV BIO /dev/sdg
fio-40948 a3ef7aa3
fio-41001 76774 512 NV BIO /dev/sdh
fio-41001 543b6d8e
[r...@fusion-io-live .bak]# cat /proc/scsi_tgt/groups/Default/devices
Device (host:ch:id:lun or name) LUN
Options
fio-71957 0
fio-71962
Re: [ofw] What is an IOC device ID and what is DriverStore
On Mon, Aug 30, 2010 at 5:46 PM, Chris Worley worl...@gmail.com wrote:
Hint:
SCST rev 1901 ends compatibility w/ Windows; r1900 works fine.
srpt_autodetect_cred_req was added in r1901, with default Y.
Setting it to N... or in later revs, 1, allows Windows to work
again.
Thanks,
Chris
As
there were a lot of diffs between these two revs, I won't include it
here.
svn diff -r 1900:1901 https://scst.svn.sourceforge.net/svnroot/scst/trunk
Thanks,
Chris
On Mon, Aug 30, 2010 at 12:37 PM, Chris Worley worl...@gmail.com wrote:
I am also seeing the following Windows events for all eight drives:
Log Name: System
Source: Microsoft-Windows-UserPnp
Date: 8/30/2010 11:58:58 AM
Event ID: 20001
Task Category: (7005)
Level: Information
Keywords:
User: SYSTEM
Computer: WIN-6EB0ETQ8M85
Description:
Driver Management concluded the process to install driver
FileRepository\disk.inf_amd64_neutral_10ce25bbc5a9cc43\disk.inf for
Device Instance ID SCSI\DISKVEN_SCST_BIOPROD_FIO-41001\07 with
the following status: 0x0.
Event Xml:
Event xmlns=http://schemas.microsoft.com/win/2004/08/events/event;
System
Provider Name=Microsoft-Windows-UserPnp
Guid={96F4A050-7E31-453C-88BE-9634F4E02139} /
EventID20001/EventID
Version0/Version
Level4/Level
Task7005/Task
Opcode0/Opcode
Keywords0x8000/Keywords
TimeCreated SystemTime=2010-08-30T17:58:58.86760Z /
EventRecordID6846/EventRecordID
Correlation /
Execution ProcessID=3228 ThreadID=3260 /
ChannelSystem/Channel
ComputerWIN-6EB0ETQ8M85/Computer
Security UserID=S-1-5-18 /
/System
UserData
InstallDeviceID
xmlns:auto-ns2=http://schemas.microsoft.com/win/2004/08/events;
xmlns=http://manifests.microsoft.com/win/2004/08/windows/userpnp;
DriverNameFileRepository\disk.inf_amd64_neutral_10ce25bbc5a9cc43\disk.inf/DriverName
DriverVersion6.1.7600.16385/DriverVersion
DriverProviderMicrosoft/DriverProvider
DeviceInstanceIDSCSI\DISKamp;VEN_SCST_BIOamp;PROD_FIO-41001\07/DeviceInstanceID
SetupClass{4D36E967-E325-11CE-BFC1-08002BE10318}/SetupClass
RebootOptionfalse/RebootOption
UpgradeDevicefalse/UpgradeDevice
IsDriverOEMfalse/IsDriverOEM
InstallStatus0x0/InstallStatus
DriverDescriptionDisk drive/DriverDescription
/InstallDeviceID
/UserData
/Event
Log Name: System
Source: Disk
Date: 8/30/2010 11:58:58 AM
Event ID: 11
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: WIN-6EB0ETQ8M85
Description:
The driver detected a controller error on \Device\Harddisk8\DR8.
Event Xml:
Event xmlns=http://schemas.microsoft.com/win/2004/08/events/event;
System
Provider Name=Disk /
EventID Qualifiers=4915611/EventID
Level2/Level
Task0/Task
Keywords0x80/Keywords
TimeCreated SystemTime=2010-08-30T17:58:58.78960Z /
EventRecordID6845/EventRecordID
ChannelSystem/Channel
ComputerWIN-6EB0ETQ8M85/Computer
Security /
/System
EventData
Data\Device\Harddisk8\DR8/Data
Binary0F0081000B0004C00301822B060058030207FE200A124A0128003C0050B39D0280F8B84B962D80FA70C8992D80FA2500/Binary
/EventData
/Event
Chris
On Fri, Aug 27, 2010 at 6:10 PM, Chris Worley worl...@gmail.com wrote:
On Fri, Aug 27, 2010 at 6:01 PM, Sufficool, Stanley
ssuffic...@rov.sbcounty.gov wrote:
What does your scst.conf look like? I assume you have LUNS in the default
group.
A Linux initiator has no trouble seeing these:
# cat /proc/scsi_tgt/vdisk/vdisk
Name Size(MB) Block size Options File name
T10 device id
fio-71957 76774 512 NV BIO /dev/sda
fio-71957 8d58d2cc
fio-71962 76774 512 NV BIO /dev/sdb
fio-71962 4ab477ef
fio-71965 76774 512 NV BIO /dev/sdc
fio-71965 28ba5c58
fio-71964 76774 512 NV BIO /dev/sdd
fio-71964 f2c69cf6
fio-41000 76774 512 NV BIO /dev/sde
fio-41000 2492daab
fio-41002 76774 512 NV BIO /dev/sdf
fio-41002 dd797f32
fio-40948 76774 512 NV BIO /dev/sdg
fio-40948 a3ef7aa3
fio-41001 76774 512 NV BIO /dev/sdh
fio-41001 543b6d8e
[r...@fusion-io-live .bak]# cat /proc
What is an IOC device ID and what is DriverStore
In scouring the many manuals with the Window's IB drivers I found a clue as to why I can't get SRP working in WinOF: Windows PNP (Plug-n-Play) will match the IOC device ID with DriverStore installed drives and then will load the IBiou.sys driver How do I set the IOC device ID and what is DriverStore? I can make the Windows SRP miniport devices appear and disappear by removing/reloading ib_srpt on the target (at least with the 2.1.1 driver from Mellanox; the latest at openfabrics.org panics the WS2008R2 system when ib_srpt is rmmod'ed). Beyond that, Windows never sees the drives... Linux initiators don't have an issue. Someone throw me a bone! The SRP initiator seems to have been broken for many months on WinDoh's. Thanks, Chris -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [ofw] What is an IOC device ID and what is DriverStore
On Fri, Aug 27, 2010 at 4:45 PM, Sufficool, Stanley ssuffic...@rov.sbcounty.gov wrote: -Original Message- From: ofw-boun...@lists.openfabrics.org [mailto:ofw-boun...@lists.openfabrics.org] On Behalf Of Chris Worley Sent: Friday, August 27, 2010 1:52 PM To: o...@lists.openfabrics.org; OFED mailing list Subject: [ofw] What is an IOC device ID and what is DriverStore In scouring the many manuals with the Window's IB drivers I found a clue as to why I can't get SRP working in WinOF: Windows PNP (Plug-n-Play) will match the IOC device ID with DriverStore installed drives and then will load the IBiou.sys driver How do I set the IOC device ID and what is DriverStore? Not sure what this is, but I have never had to set any values on the OFED-W initiator. I can make the Windows SRP miniport devices appear and disappear by removing/reloading ib_srpt on the target (at least with the 2.1.1 driver from Mellanox; the latest at openfabrics.org panics the WS2008R2 system when ib_srpt is rmmod'ed). Beyond that, Windows never sees the drives... Linux initiators don't have an issue. What messages do you get on the target? It it even receiving the request? Yes: ib_srpt: ***ERROR***: received unrecognized IB CM event 10 ib_srpt: Received DREQ and sent DREP for session 0x0024710002c10024710003d8. ib_srpt: Received SRP_LOGIN_REQ with i_port_id 0x24710002c1:0x24710003d8, t_port_id 0x24710002c1:0x24710002c1 and it_iu_len 4148 on port 1 (guid=0xfe80:0x24710002bf) ib_srpt: disconnected session 0x0024710002c10024710003d8 because a new SRP_LOGIN_REQ has been received. ib_srpt: Session : kernel thread ib_srpt_compl (PID 31542) started scst: Using security group Default for initiator 0x0024710002c10024710003d8 Does windows disk management show a broken LUN? No. Nothing (except local drives). Someone throw me a bone! The SRP initiator seems to have been broken for many months on WinDoh's. I have had no problems with the SRP initiator as long as I start the target _after_ the windows systems boot. I've tried five different revs of the Windows driver, and no drives are seen. Thanks, Chris Thanks, Chris ___ ofw mailing list o...@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [ofw] What is an IOC device ID and what is DriverStore
I'm resending this as plain text, as the linux-rdma@vger.kernel.org mailing list did not like my attempt to make the messages readable... On Fri, Aug 27, 2010 at 5:43 PM, Chris Worley worl...@gmail.com wrote: On Fri, Aug 27, 2010 at 5:02 PM, Chris Worley worl...@gmail.com wrote: On Fri, Aug 27, 2010 at 4:45 PM, Sufficool, Stanley ssuffic...@rov.sbcounty.gov wrote: -Original Message- From: ofw-boun...@lists.openfabrics.org [mailto:ofw-boun...@lists.openfabrics.org] On Behalf Of Chris Worley Sent: Friday, August 27, 2010 1:52 PM To: o...@lists.openfabrics.org; OFED mailing list Subject: [ofw] What is an IOC device ID and what is DriverStore In scouring the many manuals with the Window's IB drivers I found a clue as to why I can't get SRP working in WinOF: Windows PNP (Plug-n-Play) will match the IOC device ID with DriverStore installed drives and then will load the IBiou.sys driver How do I set the IOC device ID and what is DriverStore? Not sure what this is, but I have never had to set any values on the OFED-W initiator. I can make the Windows SRP miniport devices appear and disappear by removing/reloading ib_srpt on the target (at least with the 2.1.1 driver from Mellanox; the latest at openfabrics.org panics the WS2008R2 system when ib_srpt is rmmod'ed). Beyond that, Windows never sees the drives... Linux initiators don't have an issue. What messages do you get on the target? It it even receiving the request? Yes: ib_srpt: ***ERROR***: received unrecognized IB CM event 10 ib_srpt: Received DREQ and sent DREP for session 0x0024710002c10024710003d8. ib_srpt: Received SRP_LOGIN_REQ with i_port_id 0x24710002c1:0x24710003d8, t_port_id 0x24710002c1:0x24710002c1 and it_iu_len 4148 on port 1 (guid=0xfe80:0x24710002bf) ib_srpt: disconnected session 0x0024710002c10024710003d8 because a new SRP_LOGIN_REQ has been received. ib_srpt: Session : kernel thread ib_srpt_compl (PID 31542) started scst: Using security group Default for initiator 0x0024710002c10024710003d8 There is more to it than that. I'm going to do this as HTML so I can fix the font: ib_srpt: Received SRP_LOGIN_REQ with i_port_id 0x24710002c1:0x24710003d8, t_port_id 0x24710002c1:0x24710002c1 and it_iu_len 4148 on port 2 (guid=0xfe80:0x24710002c3) ib_srpt: Session : kernel thread ib_srpt_compl (PID 1109) started scst: Using security group Default for initiator 0x0024710002c10024710003d8 scst: Processing thread fio-719570_0 (PID 1110) started scst: Processing thread fio-719570_1 (PID ) started scst: Processing thread fio-719620_0 (PID 1112) started scst: Processing thread fio-719620_1 (PID 1113) started scst: Processing thread fio-719650_0 (PID 1114) started scst: Processing thread fio-719650_1 (PID 1115) started scst: Processing thread fio-719640_0 (PID 1116) started scst: Processing thread fio-719640_1 (PID 1117) started scst: Processing thread fio-41_0 (PID 1118) started scst: Processing thread fio-41_1 (PID 1119) started scst: Processing thread fio-410020_0 (PID 1120) started scst: Processing thread fio-410020_1 (PID 1121) started scst: Processing thread fio-409480_0 (PID 1122) started scst: Processing thread fio-409480_1 (PID 1123) started scst: Processing thread fio-410010_0 (PID 1124) started scst: Processing thread fio-410010_1 (PID 1125) started ib_srpt: ***ERROR***: received unrecognized IB CM event 10 ib_srpt: Received DREQ and sent DREP for session 0x0024710002c10024710003d8. ib_srpt: Received InfiniBand TimeWait exit for cm_id 81030b7c6c00. ib_srpt: Session 0x0024710002c10024710003d8: kernel thread ib_srpt_compl (PID 1109) stopped scst: Processing thread fio-719570_0 (PID 1110) finished scst: Processing thread fio-719570_1 (PID ) finished scst: Processing thread fio-719620_0 (PID 1112) finished scst: Processing thread fio-719620_1 (PID 1113) finished scst: Processing thread fio-719650_0 (PID 1114) finished scst: Processing thread fio-719650_1 (PID 1115) finished scst: Processing thread fio-719640_0 (PID 1116) finished scst: Processing thread fio-719640_1 (PID 1117) finished scst: Processing thread fio-41_0 (PID 1118) finished scst: Processing thread fio-41_1 (PID 1119) finished scst: Processing thread fio-410020_0 (PID 1120) finished scst: Processing thread fio-410020_1 (PID 1121) finished scst: Processing thread fio-409480_0 (PID 1122) finished scst: Processing thread fio-409480_1 (PID 1123) finished scst: Processing thread fio-410010_0 (PID 1124) finished scst: Processing thread fio-410010_1 (PID 1125) finished ib_srpt: Received SRP_LOGIN_REQ with i_port_id 0x24710002c1:0x2471000390, t_port_id 0x24710002c1:0x24710002c1 and it_iu_len 4148 on port 2 (guid=0xfe80:0x24710002c0) ib_srpt: Session : kernel thread
Re: [ofw] What is an IOC device ID and what is DriverStore
On Fri, Aug 27, 2010 at 6:01 PM, Sufficool, Stanley ssuffic...@rov.sbcounty.gov wrote: What does your scst.conf look like? I assume you have LUNS in the default group. A Linux initiator has no trouble seeing these: # cat /proc/scsi_tgt/vdisk/vdisk Name Size(MB)Block size Options File name T10 device id fio-71957 76774 512 NV BIO /dev/sda fio-71957 8d58d2cc fio-71962 76774 512 NV BIO /dev/sdb fio-71962 4ab477ef fio-71965 76774 512 NV BIO /dev/sdc fio-71965 28ba5c58 fio-71964 76774 512 NV BIO /dev/sdd fio-71964 f2c69cf6 fio-41000 76774 512 NV BIO /dev/sde fio-41000 2492daab fio-41002 76774 512 NV BIO /dev/sdf fio-41002 dd797f32 fio-40948 76774 512 NV BIO /dev/sdg fio-40948 a3ef7aa3 fio-41001 76774 512 NV BIO /dev/sdh fio-41001 543b6d8e [r...@fusion-io-live .bak]# cat /proc/scsi_tgt/groups/Default/devices Device (host:ch:id:lun or name) LUN Options fio-71957 0 fio-71962 1 fio-71965 2 fio-71964 3 fio-41000 4 fio-41002 5 fio-40948 6 fio-41001 7 (Adding scst-devel, as it looks like Windows logs in then immediately logs out or is forced out.) Thanks, Chris -Original Message- From: ofw-boun...@lists.openfabrics.org [mailto:ofw-boun...@lists.openfabrics.org] On Behalf Of Chris Worley Sent: Friday, August 27, 2010 4:46 PM To: OFED mailing list Cc: o...@lists.openfabrics.org Subject: Re: [ofw] What is an IOC device ID and what is DriverStore I'm resending this as plain text, as the linux-rdma@vger.kernel.org mailing list did not like my attempt to make the messages readable... On Fri, Aug 27, 2010 at 5:43 PM, Chris Worley worl...@gmail.com wrote: On Fri, Aug 27, 2010 at 5:02 PM, Chris Worley worl...@gmail.com wrote: On Fri, Aug 27, 2010 at 4:45 PM, Sufficool, Stanley ssuffic...@rov.sbcounty.gov wrote: -Original Message- From: ofw-boun...@lists.openfabrics.org [mailto:ofw-boun...@lists.openfabrics.org] On Behalf Of Chris Worley Sent: Friday, August 27, 2010 1:52 PM To: o...@lists.openfabrics.org; OFED mailing list Subject: [ofw] What is an IOC device ID and what is DriverStore In scouring the many manuals with the Window's IB drivers I found a clue as to why I can't get SRP working in WinOF: Windows PNP (Plug-n-Play) will match the IOC device ID with DriverStore installed drives and then will load the IBiou.sys driver How do I set the IOC device ID and what is DriverStore? Not sure what this is, but I have never had to set any values on the OFED-W initiator. I can make the Windows SRP miniport devices appear and disappear by removing/reloading ib_srpt on the target (at least with the 2.1.1 driver from Mellanox; the latest at openfabrics.org panics the WS2008R2 system when ib_srpt is rmmod'ed). Beyond that, Windows never sees the drives... Linux initiators don't have an issue. What messages do you get on the target? It it even receiving the request? Yes: ib_srpt: ***ERROR***: received unrecognized IB CM event 10 ib_srpt: Received DREQ and sent DREP for session 0x0024710002c10024710003d8. ib_srpt: Received SRP_LOGIN_REQ with i_port_id 0x24710002c1:0x24710003d8, t_port_id 0x24710002c1:0x24710002c1 and it_iu_len 4148 on port 1 (guid=0xfe80:0x24710002bf) ib_srpt: disconnected session 0x0024710002c10024710003d8 because a new SRP_LOGIN_REQ has been received. ib_srpt: Session : kernel thread ib_srpt_compl (PID 31542) started scst: Using security group Default for initiator 0x0024710002c10024710003d8 There is more to it than that. I'm going to do this as HTML so I can fix the font: ib_srpt: Received SRP_LOGIN_REQ with i_port_id 0x24710002c1:0x24710003d8, t_port_id 0x24710002c1:0x24710002c1 and it_iu_len 4148 on port 2 (guid=0xfe80:0x24710002c3) ib_srpt: Session : kernel thread ib_srpt_compl (PID 1109) started scst: Using security group Default for initiator 0x0024710002c10024710003d8 scst: Processing thread fio-719570_0 (PID
Re: How do I get scst_vdisk/IB_SRP(T) to properly handle drives w/ 4KB sectors?
On Wed, Jun 23, 2010 at 11:08 AM, Vladislav Bolkhovitin v...@vlnb.net wrote: Chris Worley, on 06/22/2010 08:06 PM wrote: When given an LBA w/ a bad boundary, the drive returns an error and the target side says: dev_vdisk: ***ERROR***: cmd 810196f58b70 returned error -22 ... and the initiator: sd 8:0:0:0: SCSI error: return code = 0x0802 sdc: Current: sense key: Medium Error Add. Sense: Unrecovered read error Is there a way to tell scst that this drive requires 4KB block sizes, and pass that upstream? I'm not sure what you mean here under tell and pass upstream. Generally, such problems are outside of SCST scope and responsibilities. With vdisk kernel I/O stack should make sure you use correct alignment accessing your backend drive and you can always choose your own 512b block size for all vdisk devices. DOH! Thanks, Chris Vlad -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
How do I get scst_vdisk/IB_SRP(T) to properly handle drives w/ 4KB sectors?
When given an LBA w/ a bad boundary, the drive returns an error and the target side says: dev_vdisk: ***ERROR***: cmd 810196f58b70 returned error -22 ... and the initiator: sd 8:0:0:0: SCSI error: return code = 0x0802 sdc: Current: sense key: Medium Error Add. Sense: Unrecovered read error Is there a way to tell scst that this drive requires 4KB block sizes, and pass that upstream? Thanks, Chris -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: SRP Q's: 1) When is asynchronous I/O complete, 2) Is sequential I/O coalesced, and 3) why is iSCSI faster than SRP in some instances
On Wed, Jan 6, 2010 at 6:57 PM, David Dillow d...@thedillows.org wrote: On Wed, 2010-01-06 at 17:16 -0700, Chris Worley wrote: 1) I'm seeing small block random writes (32KB and smaller) get better performance over SRP than they do as a local drive. I'm guessing this is async behavior: once the written data is on the wire, it's deemed complete, and setting a sync flag would disable this. Is this correct? No, from the initiator point of view, the request is not complete until the target has responded to the command. If not, any ideas why SRP random writes would be faster than the same writes locally? I would guess deeper queue depths and more cache available on the target, especially if you are using a Linux-based SRP target. I do set the ib_srp initiator srp_sg_tablesize to its maximum of 58. On the Target, I set the srp_max_rdma_size to 128KB (but that won't effect small blocks). I also set thread=1, to work around another problem. But it would only be a guess without knowing more about your setup. 2) I'm seeing very poor sequential vs. random I/O performance (both read and write) at small block sizes (random performs well, sequential performance is poor). I'm using direct I/O and the noop scheduler on the initiator, so there should be no coalescing. Coalescing on these drives is not a good thing to do, as they are ultra low latency, and much faster if the OS doesn't try to coalesce. Could anything in the IB/SRP/SCST stack be trying to coalesce sequential data? Yes, if you have more requests outstanding than available queue depth -- ie queue backpressure/congestion -- even noop will merge sequential requests in the queue. You could avoid this by setting max_sectors_kb to the maximum IO size you wish the drive to see. I thought if the device was opened with the O_DIRECT flag, then the scheduler should have nothing to coalesce. Though, I'd be surprised if there was no benefit at all to the OS coalescing under congestion. For sequential I/O benchmarking, I need to see the real results for that size packet. Direct I/O works for me everywhere except SRP. The problem turns out to be more curious: sequential reads and writes are being coalesced. I'm getting my IOPS from the diskstats, and therefore it was very low because the block size given to the device driver is very high (i.e. 32KB is delivered to the device driver, while 512 byte blocks were being sent). So, had I been looking at the bandwidth, I would have seen it inordinately/artificially high. What's more curious is the write performance excels when coalesced (w.r.t. the block size you think you're benchmarking), but the read performance does not. 3) In my iSCSI (tgt) results using the HCA as a 10G interface (not IPoIB, but mlnx4_en), comparing this to the results of using the same HCA as IB under SRP, I get much better results with SRP when benchmarking the raw device, as you'd expect. This is w/ a drive that does under 1GB/s. When I use MD to mirror that SRP or iSCSI device w/ an identical local device, and benchmark the raw MD device, iSCSI gets superior write performance and about equal read performance. Does iSCSI/TGT have some special hook into MD devices that IB/SRP isn't privy to? Are trying to achieve high IOPS or high bandwidth? I'm guessing IOPS from your other comments, but device-mapper (and I suspect MD as well) used to suffer from an internal limit on the max_sectors_kb -- you could have it set to 8 MB on the raw devices, but MD would end up restricting it to 512 KB. This is unlikely the problem if you are going for IOPS, I'm doing the MD on the initiator side. I'll try playing with this. but can play a factor in bandwidth. Then again, since the setup seems to be identical, I'm not sure it is your problem here either. :( Have you tried using the function tracer or perf tools found in recent kernels to follow the data path and find the hotspots? I have not. I parse the data from diskstats. A pointer to these tools would be appreciated. Chris Dave -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: SRP Q's: 1) When is asynchronous I/O complete, 2) Is sequential I/O coalesced, and 3) why is iSCSI faster than SRP in some instances
On Fri, Jan 8, 2010 at 3:17 PM, David Dillow d...@thedillows.org wrote: On Fri, 2010-01-08 at 14:40 -0700, Chris Worley wrote: On Wed, Jan 6, 2010 at 6:57 PM, David Dillow d...@thedillows.org wrote: On Wed, 2010-01-06 at 17:16 -0700, Chris Worley wrote: 1) I'm seeing small block random writes (32KB and smaller) get better performance over SRP than they do as a local drive. I'm guessing this is async behavior: once the written data is on the wire, it's deemed complete, and setting a sync flag would disable this. Is this correct? If not, any ideas why SRP random writes would be faster than the same writes locally? I would guess deeper queue depths and more cache available on the target, especially if you are using a Linux-based SRP target. I do set the ib_srp initiator srp_sg_tablesize to its maximum of 58. The max is 255, which will guarantee you can send up to a 1020 KB I/O without breaking it into two SCSI commands. In practice, you're likely to be able to send larger requests, as you will often have some contiguous runs in the data pages. I've tried a larger max... 58 is all I can get. Maybe getting more is dependent on some other setting. This is probably not hurting you at smaller request sizes. 2) I'm seeing very poor sequential vs. random I/O performance (both read and write) at small block sizes (random performs well, sequential performance is poor). I'm using direct I/O and the noop scheduler on the initiator, so there should be no coalescing. Coalescing on these drives is not a good thing to do, as they are ultra low latency, and much faster if the OS doesn't try to coalesce. Could anything in the IB/SRP/SCST stack be trying to coalesce sequential data? Yes, if you have more requests outstanding than available queue depth -- ie queue backpressure/congestion -- even noop will merge sequential requests in the queue. You could avoid this by setting max_sectors_kb to the maximum IO size you wish the drive to see. I thought if the device was opened with the O_DIRECT flag, then the scheduler should have nothing to coalesce. Depends on how many I/Os your application has in flight at once, assuming it is using AIO or threads. If you have more requests in flight than can be queued, the block layer will coalesce if possible. I do use AIO, always 64 threads, each w/ 64 outstanding I/O's. Local or iSER initiator based, I never see any coalescing. Only w/ SRP. Though, I'd be surprised if there was no benefit at all to the OS coalescing under congestion. Benefit isn't the issue. It needs to be benchmarked w/o artificial aids that cloud the results. I'm not really fond of sequential I/O, as it seldom really exists in real applications (except for logging apps), but if I'm going to test it, I need valid numbers. I could do like the SAN/FC vendors do, and just take the throughput for 1MB blocks and divide the TPS by 2M and call that the 512 byte block IOPS ;) For sequential I/O benchmarking, I need to see the real results for that size packet. Direct I/O works for me everywhere except SRP. Hmm, that seems a bit odd, but there is nothing in the SRP initiator that would cause the behavior you are seeing -- it just hands over the requests the SCSI and block layers give it. Are you observing this via diskstats at the initiator or the target side of the SRP connection? Diskstats on the initiator side. There is the scst_vdisk Direct I/O option that's been commented out of the code, as it's not supposed to work... maybe direct I/O doesn't work... but that would be the target side. You could also try using sgp_dd from lustre-iokit, but I've seen some oddities from it -- it couldn't drive the hardware I was testing at full speed, where XDD and some custom tools I wrote did. You may have mentioned this, but are you using the raw device, or a filesystem over top of it? It depends: this #2 issue, sequential vs random: it's atop the raw block device. The third issue was atop MD. As some of this thread has been snipped, I'm not completely sure which issue we're discussing. Also, I've seen some interesting things like device mapper reporting a 4 KB read as 8 512 byte sectors, even though it was handed to DM as a 4KB request, so there could be gremlins there as well. I don't know how the MD device driver reports this. What does the output of 'cd /sys/block/sda/queue head *' look like, where sda should be replaced with the SRP disk. It would also be interesting to see that for iSCSI, and in /sys/class/scsi_disk/0:0:0:0/device for both connection types to see if there is a difference. Initiator or target? The target side isn't a SCSI device, it's a block device. I guess I could use scst_local to make it look scsi-ish. Have you tried using the function tracer or perf tools found in recent kernels to follow the data path and find the hotspots? I have not. I parse the data from diskstats. A pointer
SRP Q's: 1) When is asynchronous I/O complete, 2) Is sequential I/O coalesced, and 3) why is iSCSI faster than SRP in some instances
In shifting through a great deal of benchmark data collected from two identical machines (including the attached drive array), I see the following SRP anomalies: 1) I'm seeing small block random writes (32KB and smaller) get better performance over SRP than they do as a local drive. I'm guessing this is async behavior: once the written data is on the wire, it's deemed complete, and setting a sync flag would disable this. Is this correct? If not, any ideas why SRP random writes would be faster than the same writes locally? 2) I'm seeing very poor sequential vs. random I/O performance (both read and write) at small block sizes (random performs well, sequential performance is poor). I'm using direct I/O and the noop scheduler on the initiator, so there should be no coalescing. Coalescing on these drives is not a good thing to do, as they are ultra low latency, and much faster if the OS doesn't try to coalesce. Could anything in the IB/SRP/SCST stack be trying to coalesce sequential data? If not, any other ideas on why I might see this? 3) In my iSCSI (tgt) results using the HCA as a 10G interface (not IPoIB, but mlnx4_en), comparing this to the results of using the same HCA as IB under SRP, I get much better results with SRP when benchmarking the raw device, as you'd expect. This is w/ a drive that does under 1GB/s. When I use MD to mirror that SRP or iSCSI device w/ an identical local device, and benchmark the raw MD device, iSCSI gets superior write performance and about equal read performance. Does iSCSI/TGT have some special hook into MD devices that IB/SRP isn't privy to? Any ideas or clues would be helpful. Thanks, Chris -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: SRPT and SCST
On Mon, Nov 9, 2009 at 1:26 PM, Vladislav Bolkhovitin v...@vlnb.net wrote: Bart Van Assche, on 11/08/2009 12:49 PM wrote: On Fri, Nov 6, 2009 at 6:28 PM, Arend Dittmer aditt...@penguincomputing.com wrote: Please find attached the gzip'ed /var/log/messages. This log clearly show the login and logout actions from the different initiators. I couldn't find anything unusual in the posted log file however. Around which time did the initiator start complaining about aborted SCSI commands ? Does this issue also happen when using the SRP initiator included in a vanilla (non-OFED) Linux kernel ? It looks painfully similar to what Chris Worley experienced some time ago and somehow fixed/workarounded. Chris, can you comment on this? The thread=1 fixed the problem mostly, but I am working with another group that says they still get an abort, but haven't gotten around to providing me with the info I need to look at it. Chris Bart. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Adjusting minimum packet size or wait to merge requests in SRP
It appears that SRP tries to coalesce and fragment initiator I/O requests into 64KB packets, as that looks to be the size requested to/from the device on the target side (and the I/O scheduler is disabled on the target). Is there a way to control this, where no coalescing occurs when latency is an issue and requests are small, and no fragmentation occurs when requests are large? Or, am I totally wrong in my assumption that SRP is coalescing/fragmenting data? Thanks, Chris -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Adjusting minimum packet size or wait to merge requests in SRP
On Wed, Oct 28, 2009 at 1:14 PM, Bart Van Assche bart.vanass...@gmail.com wrote: On Wed, Oct 28, 2009 at 7:47 PM, Chris Worley worl...@gmail.com wrote: It appears that SRP tries to coalesce and fragment initiator I/O requests into 64KB packets, as that looks to be the size requested to/from the device on the target side (and the I/O scheduler is disabled on the target). Is there a way to control this, where no coalescing occurs when latency is an issue and requests are small, and no fragmentation occurs when requests are large? Or, am I totally wrong in my assumption that SRP is coalescing/fragmenting data? Regarding avoiding coalescing of I/O requests: which I/O scheduler is being used on the initiator system and how has it been configured via sysfs ? There is no scheduler running on either target or initiator on the drives in question (sorry I worded that incorrectly initially), or so I've been told (this information is second-hand). I did see iostat output from the initiator in his case, where there were long waits and service times that I'm guessing was due to some coalescing/merging. There was also a hint in the iostat output that a scheduler was enabled, as there were non-zero values (occasionally) under the [rw]qm/s columns, which, if I understand iostat correctly, means there is a scheduler merging results. So you're saying there is no hold-off for merging on the initiator side of the IB/SRP stack? Adjusting the constant MAX_RDMA_SIZE in scst/srpt/src/ib_srpt.h might help to avoid fragmentation of large requests by the SRP protocol. Please post a follow-up message to the mailing list with your findings such that MAX_RDMA_SIZE can be converted from a compile-time constant to a sysfs variable if this would be useful. Will do. Thanks, Chris Bart. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Scst-devel] [ofa-general] WinOF_2_0_5/SRP initiator: slow reads and eventually hangs
On Wed, Sep 16, 2009 at 1:03 AM, Bart Van Assche bart.vanass...@gmail.com wrote: On Tue, Sep 15, 2009 at 10:51 PM, Chris Worley worl...@gmail.com wrote: In lots of testing today, I've seen this panic twice on the Ubuntu 8.10 targets: [ 330.155992] ib_srpt: disconnected session 0x0024714600247146 because a new SRP_LOGIN_REQ has been received. The above message means that an initiator logged in, did not log out and logged in again. Has one of the initiator systems e.g. been power cycled while an SRP connection was active ? Yes. Once ib_srp is hung on one device, I re-login to get the device and test again. I can't log out of the previous, as it's hung... this hanging is the issue I'm having. When I re-login, I get a new device... i.e. I hung /dev/sdc and re-login to get /dev/sdd... then test that until it hangs. Using the ramdisks makes the problem much easier to trigger, and occasionally causes the panic, especially using: fio --rw=randrw --bs=1k --numjobs=64 --iodepth=64 --sync=0 \ --direct=1 --randrepeat=0 --group_reporting --ioengine=libaio \ --filename=/dev/sdp --name=test --loops=1 --runtime=1600 \ --rwmixread=100 ...on the initiator to cause it. Chris [ 357.207046] ib_srpt: srpt_xmit_response: tag= 17 channel in bad state 2 This means that an attempt was made by SRPT to transmit a response for a channel in the state 2 (disconnecting). This must be analyzed further, just like the bug report triggered from srpt_abort_scst_cmd(). [ ... ] [ 411.636699] WARNING: at /root/scst/srpt/src/ib_srpt.c:924 srpt_abort_scst_cmd+0xac/0x160 [ib_srpt]() ... This message has been triggered by the statement WARN_ON(unexpected cmd state). It must be analyzed whether this is a consequence of what went wrong before or whether this is a separate issue. [ ... ] This may have been due to low memory, as I was using most target memory for the ramdisk. The kernel warning message in ib_srpt.c at line 924 should never be triggered, not even under low memory circumstances. I'll have a look at this anyway. Thanks for the detailed report. Bart. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Scst-devel] [ofa-general] WinOF_2_0_5/SRP initiator: slow reads and eventually hangs
On Tue, Sep 15, 2009 at 12:10 AM, Bart Van Assche bart.vanass...@gmail.com wrote: On Tue, Sep 15, 2009 at 1:03 AM, Chris Worley worl...@gmail.com wrote: On Mon, Sep 14, 2009 at 12:51 PM, Vladislav Bolkhovitin v...@vlnb.net wrote: Chris Worley, on 09/11/2009 11:50 PM wrote: I've definitely removed the switch/firmware from being the cause. I'm thinking the reason you can't repeat the test may be latency related. We get ~50usecs average latency (on small block sizes), which can't be achieved using regular SSD's (and rotating drives are nowhere close). Maybe a ramdisk would help repeat the issue. I think you should try to reproduce the problem with ramdisk or nullio. By so you will eliminate possible influence of the SSD backend. W/ 12GB RAM in the target, I created a 7GB ramdisk: mount -t ramfs -o size=7g ramfs /mnt/ dd if=/dev/zero of=/mnt/foo bs=1024k count=7000 echo open ramdisk /mnt/foo /proc/scsi_tgt/vdisk/vdisk echo add ramdisk 2 /proc/scsi_tgt/groups/Default/devices Then, on the initiator, I tested it... and it hung during sequential 8KB block reads: fio --rw=read --bs=8k --numjobs=64 --iodepth=64 --sync=0 --direct=1 --randrepeat=0 \ --group_reporting --ioengine=libaio --filename=/dev/sde --name=test --loops=1 --runtime=600 Note that I was running the SM on the target this time too. Which Linux distro was installed on the inititiator and on the target ? And if applicable, which OFED version ? Which kernel messages were logged by SRPT around the time the issue occurred (after having enabled SRPT logging first) ? As logging hadn't helped this issue previously, I've not been enabling it. That plus the kernel hacks needed to invoke logging, it's not worth enabling. This was with Ubuntu 8.10, built-in IB on the 2.6.27-14-server kernel. I couldn't get ramdisks working w/ SCST in RHEL5.2. When running: echo open ramdisk /mnt/foo /proc/scsi_tgt/vdisk/vdisk I get the error: dev_vdisk: ***ERROR***: Wrong f_op or FS doesn't have required capabilities ... which doesn't occur in the Ubuntu kernel, so I've been unable to test RHEL kernels w/ ramdisks. In general, this problem occurs w/ 8KB and smaller blocks w/ the Ubuntu kernels, and 2KB and smaller blocks w/ RHEL kernels. Chris Bart. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Scst-devel] [ofa-general] WinOF_2_0_5/SRP initiator: slow reads and eventually hangs
On Tue, Sep 15, 2009 at 10:57 AM, Vladislav Bolkhovitin v...@vlnb.net wrote: Chris Worley, on 09/15/2009 08:53 PM wrote: On Tue, Sep 15, 2009 at 10:43 AM, Vladislav Bolkhovitin v...@vlnb.net wrote: Chris Worley, on 09/15/2009 07:50 PM wrote: On Tue, Sep 15, 2009 at 12:10 AM, Bart Van Assche bart.vanass...@gmail.com wrote: On Tue, Sep 15, 2009 at 1:03 AM, Chris Worley worl...@gmail.com wrote: On Mon, Sep 14, 2009 at 12:51 PM, Vladislav Bolkhovitin v...@vlnb.net wrote: Chris Worley, on 09/11/2009 11:50 PM wrote: I've definitely removed the switch/firmware from being the cause. I'm thinking the reason you can't repeat the test may be latency related. We get ~50usecs average latency (on small block sizes), which can't be achieved using regular SSD's (and rotating drives are nowhere close). Maybe a ramdisk would help repeat the issue. I think you should try to reproduce the problem with ramdisk or nullio. By so you will eliminate possible influence of the SSD backend. W/ 12GB RAM in the target, I created a 7GB ramdisk: mount -t ramfs -o size=7g ramfs /mnt/ dd if=/dev/zero of=/mnt/foo bs=1024k count=7000 echo open ramdisk /mnt/foo /proc/scsi_tgt/vdisk/vdisk echo add ramdisk 2 /proc/scsi_tgt/groups/Default/devices Then, on the initiator, I tested it... and it hung during sequential 8KB block reads: fio --rw=read --bs=8k --numjobs=64 --iodepth=64 --sync=0 --direct=1 --randrepeat=0 \ --group_reporting --ioengine=libaio --filename=/dev/sde --name=test --loops=1 --runtime=600 Note that I was running the SM on the target this time too. Which Linux distro was installed on the inititiator and on the target ? And if applicable, which OFED version ? Which kernel messages were logged by SRPT around the time the issue occurred (after having enabled SRPT logging first) ? As logging hadn't helped this issue previously, I've not been enabling it. That plus the kernel hacks needed to invoke logging, it's not worth enabling. This was with Ubuntu 8.10, built-in IB on the 2.6.27-14-server kernel. I couldn't get ramdisks working w/ SCST in RHEL5.2. When running: echo open ramdisk /mnt/foo /proc/scsi_tgt/vdisk/vdisk I get the error: dev_vdisk: ***ERROR***: Wrong f_op or FS doesn't have required capabilities ... which doesn't occur in the Ubuntu kernel, so I've been unable to test RHEL kernels w/ ramdisks. In general, this problem occurs w/ 8KB and smaller blocks w/ the Ubuntu kernels, and 2KB and smaller blocks w/ RHEL kernels. Use ramfs instead. Do you mean: mount -t ramfs -o size=7g ramfs /mnt/ You should then create a file on it and use it. That's what I'm doing, I believe. From above: mount -t ramfs -o size=7g ramfs /mnt/ dd if=/dev/zero of=/mnt/foo bs=1024k count=7000 echo open ramdisk /mnt/foo /proc/scsi_tgt/vdisk/vdisk echo add ramdisk 2 /proc/scsi_tgt/groups/Default/devices ... but the open, on RHEL5.2 kernel 2.6.18-92.el5, generates the following kernel messages: dev_vdisk: Registering virtual FILEIO device ramdisk scst: Processing thread started, PID 9629 scst: Processing thread started, PID 9630 scst: Processing thread started, PID 9631 scst: Processing thread started, PID 9632 scst: Processing thread started, PID 9633 dev_vdisk: ***ERROR***: Wrong f_op or FS doesn't have required capabilities scst: ***ERROR***: New device handler's vdisk attach() failed: -22 scst: Processing thread PID 9629 finished scst: Processing thread PID 9630 finished scst: Processing thread PID 9631 finished scst: Processing thread PID 9632 finished scst: Processing thread PID 9633 finished scst: Failed to attach to virtual device ramdisk Chris ? That's what I'm doing. Chris Chris Bart. -- Come build with us! The BlackBerryreg; Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9#45;12, 2009. Register now#33; http://p.sf.net/sfu/devconf ___ Scst-devel mailing list scst-de...@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/scst-devel ___ general mailing list gene...@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
