This is to announce Open-iSCSI project: High-Performance iSCSI Initiator for Linux.
MOTIVATION ==========
Our initial motivations for the project were: (1) implement the right user/kernel split, and (2) design iSCSI data path for performance. Recently we added (3): get accepted into the mainline kernel.
As far as user/kernel, the existing iSCSI initiators bloat the kernel with ever-growing control plane code, including but not limited to: iSCSI discovery, Login (Authentication and Operational), session and connection management, connection-level error processing, iSCSI Text, Nop-Out/In, Async Message, iSNS, SLP, Radius... Open-iSCSI puts the entire control plane in the user space. This control plane talks to the data plane via well defined interface over the netlink transport.
(Side note: prior to closing on the netlink we considered: sysfs, ioctl, and syscall. Because the entire control plane logic resides in the user space, we needed a real bi-directional transport that could support asynchronous API to transfer iSCSI control PDUs: Login, Logout, Nop-in, Nop-Out, Text, Async Message.
Performance. This is the major goal and motivation for this project. As it happens, iSCSI has to compete with Fibre Channel, which is a more entrenched technology in the storage space. In addition, the "soft" iSCSI implementation have to show good results in presence of specialized hardware offloads.
Our today's performance numbers are:
- 450MB/sec Read on a single connection (2-way 2.4Ghz Opteron, 64KB block size);
- 320MB/sec Write on a single connection (2-way 2.4Ghz Opteron, 64KB block size);
- 50,000 Read IOPS on a single connection (2-way 2.4Ghz Opteron, 4KB block size).
Prior to starting from-scratch the data path code we did evaluate the sfnet Initiator. And eventually decided against patching it. Instead, we reused its Discovery, Login, etc. control plane code. Technically, it was the shortest way to achieve the (1) and (2) goals stated above. We believe that it remains the easiest and the most practical thing on the larger scale of: iSCSI for Linux.
STATUS ======
There's a 100% working code that interoperates with all (count=5) iSCSI targets we could get our hands on.
The software was tested on AMD Opteron (TM) and Intel Xeon (TM).
Code is available online via either Subversion source control database or the latest development release (i.e., the tarball containing Open-iSCSI sources, including user space, that will build and run on kernels starting 2.6.10).
http://www.open-iscsi.org
Features:
- highly optimized and small-footprint data path; - multiple outstanding R2Ts; - thread-less receive; - sendpage() based transmit; - zero-copy header processing on receive; - no data path memory allocations at runtime; - persistent configuration database; - SendTargets discovery; - CHAP; - DataSequenceInOrder=No; - PDU header Digest; - multiple sessions; - MC/S (note: disabled in the patch); - SCSI-level recovery via Abort Task and session re-open.
TODO ====
The near term plan is: test, test, and test. We need to stabilize the existing code, after 5 months of development this seems to be the right thing to do.
Other short-term plans include:
a) process community feedback, implement comments and apply patches; b) cleanup user side of the iSCSI open interface; use API calls (instead of directly constructing events); c) eliminate runtime control path memory allocations (for Nop-In, Nop-Out, etc.); d) implement Write path optimizations (delayed because of the self-imposed submission deadline); e) oProfile the data path, use the reports for further optimization; f) complete the readme.
Comments, code reviews, patches - are greatly appreciated!
THANKS ======
Special thanks to our first reviewers: Christoph Hellwig and Mike Christie.
Special thanks to Ming Zhang for help in testing and for insightful questions.
Regards,
Alex Aizman & Dmitry Yusupov
=============================================
The following 6 patches alltogether represent the Open-iSCSI Initiator:
Patch 1: SCSI LLDD consists of 3 files: - iscsi_if.c (iSCSI open interface over netlink); - iscsi_tcp.[ch] (iSCSI transport over TCP/IP).
Patch 2: Common header files: - iscsi_if.h (iSCSI open interface over netlink); - iscsi_proto.h (RFC3720 #defines and types); - iscsi_ifev.h (user/kernel events).
Patch 3: drivers/scsi/Kconfig changes.
Patch 4: drivers/scsi/Makefile changes.
Patch 5: include/linux/netlink.h changes (added new protocol NETLINK_ISCSI)
Patch 6: Documentation/scsi/iscsi.txt
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
