Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet, 1.25 million IOPS update
Vladislav Bolkhovitin, on 06/22/2010 11:04 PM wrote: Pasi Kärkkäinen, on 06/18/2010 12:23 PM wrote: [1] None of the performance measurement tools for Linux I've seen so far, including disktest (although I've not looked at newer (1-1.5 years) versions) and fio satisfied me for various reasons. What's missing from ltp disktest? It can't do the I/O method needed in this test: async 512b sequential zero copy SG/BSG in FIFO order + I have some questions to how it calculates results and how effectively it implements multithreading, so it should be audited to clear those questions. I mean before I can trust it, I need to audit how it works to make sure it does everything correctly. Otherwise it would be hard to narrow down exact bottleneck of the test. Vlad -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet, 1.25 million IOPS update
Pasi Kärkkäinen, on 06/18/2010 12:23 PM wrote: [1] None of the performance measurement tools for Linux I've seen so far, including disktest (although I've not looked at newer (1-1.5 years) versions) and fio satisfied me for various reasons. What's missing from ltp disktest? It can't do the I/O method needed in this test: async 512b sequential zero copy SG/BSG in FIFO order + I have some questions to how it calculates results and how effectively it implements multithreading, so it should be audited to clear those questions. Vlad -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet, 1.25 million IOPS update
On Fri, Jun 18, 2010 at 04:48:57PM +0300, guy keren wrote: > Pasi Kärkkäinen wrote: >> On Fri, Jun 18, 2010 at 12:10:04PM +0300, guy keren wrote: >>> Pasi Kärkkäinen wrote: On Mon, Jun 14, 2010 at 11:39:47PM +0400, Vladislav Bolkhovitin wrote: > Pasi Kärkkäinen, on 06/11/2010 11:26 AM wrote: >> On Fri, Feb 05, 2010 at 02:10:32PM +0300, Vladislav Bolkhovitin wrote: >>> Pasi Kärkkäinen, on 01/28/2010 03:36 PM wrote: Hello list, Please check these news items: http://blog.fosketts.net/2010/01/14/microsoft-intel-push-million-iscsi-iops/ http://communities.intel.com/community/openportit/server/blog/2010/01/19/100-iops-with-iscsi--thats-not-a-typo http://www.infostor.com/index/blogs_new/dave_simpson_storage/blogs/infostor/dave_simpon_storage/post987_37501094375591341.html "1,030,000 IOPS over a single 10 Gb Ethernet link" "Specifically, Intel and Microsoft clocked 1,030,000 IOPS (with 512-byte blocks), and more than 2,250MBps with large block sizes (16KB to 256KB) using the Iometer benchmark" So.. who wants to beat that using Linux + open-iscsi? :) >>> I personally, don't like such tests and don't trust them at >>> all. They are pure marketing. The only goal of them is to >>> create impression that X (Microsoft and Windows in this >>> case) is a super-puper ahead of the world. I've seen on the >>> Web a good article about usual tricks used by vendors to >>> cheat benchmarks to get good marketing material, but, >>> unfortunately, can't find link on it at the moment. >>> >>> The problem is that you can't say from such tests if X will >>> also "ahead of the world" on real life usages, because such >>> tests always heavily optimized for particular used >>> benchmarks and such optimizations almost always hurt real >>> life cases. And you hardly find descriptions of those >>> optimizations as well as a scientific description of the >>> tests themself. The results published practically only in >>> marketing documents. >>> >>> Anyway, as far as I can see Linux supports all the used >>> hardware as well as all advance performance modes of it, so >>> if one repeats this test in the same setup, he/she should >>> get not worse results. >>> >>> For me personally it was funny to see how MS presents in the >>> WinHEC presentation >>> (http://download.microsoft.com/download/5/E/6/5E66B27B-988B-4F50-AF3A-C2FF1E62180F/COR-T586_WH08.pptx) >>> >>> that they have 1.1GB/s from 4 connections. In the beginning >>> of 2008 I saw a *single* dd pushing data on that rate over a >>> *single* connection from Linux initiator to iSCSI-SCST >>> target using regular Myricom hardware without any special >>> acceleration. I didn't know how proud I must have been for >>> Linux :). >>> >> It seems they've described the setup here: >> http://communities.intel.com/community/wired/blog/2010/04/20/1-million-iop-article-explained >> >> And today they seem to have a demo which produces 1.3 million IOPS! >> >> "1 Million IOPS? How about 1.25 Million!": >> http://communities.intel.com/community/wired/blog/2010/04/22/1-million-iops-how-about-125-million > I'm glad for them. The only thing surprises me that none of the > Linux vendors, including Intel itself, interested to repeat this > test for Linux and fix possible found problems, if any. Ten > years ago similar test about Linux TCP scalability limitations > comparing with Windows caused massive reaction and great TCP > improvements. > Yeah, I'd like to see this aswell. I don't think I have enough extra hardware myself.. atm. Does someone have enough boxes with 10 Gbit connections? :) > The way how to do the test is quite straightforward, starting > from making for Linux similarly effective test tool as IOMeter > on Windows [1]. Maybe, the lack of such tool scares the vendors > away? > I'm wondering how big effort it would be to fix IOmeter for linux.. iirc there were some patches to fix the AIO stuff. >>> the AIO stuff inside IOMeter won't necessarily help, since the AIO >>> implementation in linux kernels is not efficient enough* >>> >>> * note: i'm only updated to kernel 2.6.18 - but i didn't here there >>> was a strong effort to make this better in newer kernels. correct me >>> if i'm wrong. >>> >> >> So what's the actual problem? > > as far as i know - sometimes, AIO requests are handled in a synchronous > manner - so it's not AIO all the way. maybe this has become better in > later kernels - but i doubt that. > Ok. Mike: Do you see/know any scalability problems related to this kind of >1 million IOPS benchmark? I was thinking of ask
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet, 1.25 million IOPS update
Pasi Kärkkäinen wrote: On Fri, Jun 18, 2010 at 12:10:04PM +0300, guy keren wrote: Pasi Kärkkäinen wrote: On Mon, Jun 14, 2010 at 11:39:47PM +0400, Vladislav Bolkhovitin wrote: Pasi Kärkkäinen, on 06/11/2010 11:26 AM wrote: On Fri, Feb 05, 2010 at 02:10:32PM +0300, Vladislav Bolkhovitin wrote: Pasi Kärkkäinen, on 01/28/2010 03:36 PM wrote: Hello list, Please check these news items: http://blog.fosketts.net/2010/01/14/microsoft-intel-push-million-iscsi-iops/ http://communities.intel.com/community/openportit/server/blog/2010/01/19/100-iops-with-iscsi--thats-not-a-typo http://www.infostor.com/index/blogs_new/dave_simpson_storage/blogs/infostor/dave_simpon_storage/post987_37501094375591341.html "1,030,000 IOPS over a single 10 Gb Ethernet link" "Specifically, Intel and Microsoft clocked 1,030,000 IOPS (with 512-byte blocks), and more than 2,250MBps with large block sizes (16KB to 256KB) using the Iometer benchmark" So.. who wants to beat that using Linux + open-iscsi? :) I personally, don't like such tests and don't trust them at all. They are pure marketing. The only goal of them is to create impression that X (Microsoft and Windows in this case) is a super-puper ahead of the world. I've seen on the Web a good article about usual tricks used by vendors to cheat benchmarks to get good marketing material, but, unfortunately, can't find link on it at the moment. The problem is that you can't say from such tests if X will also "ahead of the world" on real life usages, because such tests always heavily optimized for particular used benchmarks and such optimizations almost always hurt real life cases. And you hardly find descriptions of those optimizations as well as a scientific description of the tests themself. The results published practically only in marketing documents. Anyway, as far as I can see Linux supports all the used hardware as well as all advance performance modes of it, so if one repeats this test in the same setup, he/she should get not worse results. For me personally it was funny to see how MS presents in the WinHEC presentation (http://download.microsoft.com/download/5/E/6/5E66B27B-988B-4F50-AF3A-C2FF1E62180F/COR-T586_WH08.pptx) that they have 1.1GB/s from 4 connections. In the beginning of 2008 I saw a *single* dd pushing data on that rate over a *single* connection from Linux initiator to iSCSI-SCST target using regular Myricom hardware without any special acceleration. I didn't know how proud I must have been for Linux :). It seems they've described the setup here: http://communities.intel.com/community/wired/blog/2010/04/20/1-million-iop-article-explained And today they seem to have a demo which produces 1.3 million IOPS! "1 Million IOPS? How about 1.25 Million!": http://communities.intel.com/community/wired/blog/2010/04/22/1-million-iops-how-about-125-million I'm glad for them. The only thing surprises me that none of the Linux vendors, including Intel itself, interested to repeat this test for Linux and fix possible found problems, if any. Ten years ago similar test about Linux TCP scalability limitations comparing with Windows caused massive reaction and great TCP improvements. Yeah, I'd like to see this aswell. I don't think I have enough extra hardware myself.. atm. Does someone have enough boxes with 10 Gbit connections? :) The way how to do the test is quite straightforward, starting from making for Linux similarly effective test tool as IOMeter on Windows [1]. Maybe, the lack of such tool scares the vendors away? I'm wondering how big effort it would be to fix IOmeter for linux.. iirc there were some patches to fix the AIO stuff. the AIO stuff inside IOMeter won't necessarily help, since the AIO implementation in linux kernels is not efficient enough* * note: i'm only updated to kernel 2.6.18 - but i didn't here there was a strong effort to make this better in newer kernels. correct me if i'm wrong. So what's the actual problem? as far as i know - sometimes, AIO requests are handled in a synchronous manner - so it's not AIO all the way. maybe this has become better in later kernels - but i doubt that. --guy -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet, 1.25 million IOPS update
On Fri, Jun 18, 2010 at 12:10:04PM +0300, guy keren wrote: > Pasi Kärkkäinen wrote: >> On Mon, Jun 14, 2010 at 11:39:47PM +0400, Vladislav Bolkhovitin wrote: >>> Pasi Kärkkäinen, on 06/11/2010 11:26 AM wrote: On Fri, Feb 05, 2010 at 02:10:32PM +0300, Vladislav Bolkhovitin wrote: > Pasi Kärkkäinen, on 01/28/2010 03:36 PM wrote: >> Hello list, >> >> Please check these news items: >> http://blog.fosketts.net/2010/01/14/microsoft-intel-push-million-iscsi-iops/ >> http://communities.intel.com/community/openportit/server/blog/2010/01/19/100-iops-with-iscsi--thats-not-a-typo >> http://www.infostor.com/index/blogs_new/dave_simpson_storage/blogs/infostor/dave_simpon_storage/post987_37501094375591341.html >> >> "1,030,000 IOPS over a single 10 Gb Ethernet link" >> >> "Specifically, Intel and Microsoft clocked 1,030,000 IOPS (with >> 512-byte blocks), and more than 2,250MBps with large block >> sizes (16KB to 256KB) using the Iometer benchmark" >> >> So.. who wants to beat that using Linux + open-iscsi? :) > I personally, don't like such tests and don't trust them at all. > They are pure marketing. The only goal of them is to create > impression that X (Microsoft and Windows in this case) is a > super-puper ahead of the world. I've seen on the Web a good > article about usual tricks used by vendors to cheat benchmarks > to get good marketing material, but, unfortunately, can't find > link on it at the moment. > > The problem is that you can't say from such tests if X will also > "ahead of the world" on real life usages, because such tests > always heavily optimized for particular used benchmarks and such > optimizations almost always hurt real life cases. And you hardly > find descriptions of those optimizations as well as a scientific > description of the tests themself. The results published > practically only in marketing documents. > > Anyway, as far as I can see Linux supports all the used hardware > as well as all advance performance modes of it, so if one > repeats this test in the same setup, he/she should get not worse > results. > > For me personally it was funny to see how MS presents in the > WinHEC presentation > (http://download.microsoft.com/download/5/E/6/5E66B27B-988B-4F50-AF3A-C2FF1E62180F/COR-T586_WH08.pptx) > > that they have 1.1GB/s from 4 connections. In the beginning of > 2008 I saw a *single* dd pushing data on that rate over a > *single* connection from Linux initiator to iSCSI-SCST target > using regular Myricom hardware without any special acceleration. > I didn't know how proud I must have been for Linux :). > It seems they've described the setup here: http://communities.intel.com/community/wired/blog/2010/04/20/1-million-iop-article-explained And today they seem to have a demo which produces 1.3 million IOPS! "1 Million IOPS? How about 1.25 Million!": http://communities.intel.com/community/wired/blog/2010/04/22/1-million-iops-how-about-125-million >>> I'm glad for them. The only thing surprises me that none of the Linux >>> vendors, including Intel itself, interested to repeat this test for >>> Linux and fix possible found problems, if any. Ten years ago similar >>> test about Linux TCP scalability limitations comparing with Windows >>> caused massive reaction and great TCP improvements. >>> >> >> Yeah, I'd like to see this aswell. >> I don't think I have enough extra hardware myself.. atm. >> >> Does someone have enough boxes with 10 Gbit connections? :) >> >>> The way how to do the test is quite straightforward, starting from >>> making for Linux similarly effective test tool as IOMeter on Windows >>> [1]. Maybe, the lack of such tool scares the vendors away? >>> >> >> I'm wondering how big effort it would be to fix IOmeter for linux.. >> iirc there were some patches to fix the AIO stuff. > > the AIO stuff inside IOMeter won't necessarily help, since the AIO > implementation in linux kernels is not efficient enough* > > * note: i'm only updated to kernel 2.6.18 - but i didn't here there was > a strong effort to make this better in newer kernels. correct me if i'm > wrong. > So what's the actual problem? -- Pasi -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet, 1.25 million IOPS update
Pasi Kärkkäinen wrote: On Mon, Jun 14, 2010 at 11:39:47PM +0400, Vladislav Bolkhovitin wrote: Pasi Kärkkäinen, on 06/11/2010 11:26 AM wrote: On Fri, Feb 05, 2010 at 02:10:32PM +0300, Vladislav Bolkhovitin wrote: Pasi Kärkkäinen, on 01/28/2010 03:36 PM wrote: Hello list, Please check these news items: http://blog.fosketts.net/2010/01/14/microsoft-intel-push-million-iscsi-iops/ http://communities.intel.com/community/openportit/server/blog/2010/01/19/100-iops-with-iscsi--thats-not-a-typo http://www.infostor.com/index/blogs_new/dave_simpson_storage/blogs/infostor/dave_simpon_storage/post987_37501094375591341.html "1,030,000 IOPS over a single 10 Gb Ethernet link" "Specifically, Intel and Microsoft clocked 1,030,000 IOPS (with 512-byte blocks), and more than 2,250MBps with large block sizes (16KB to 256KB) using the Iometer benchmark" So.. who wants to beat that using Linux + open-iscsi? :) I personally, don't like such tests and don't trust them at all. They are pure marketing. The only goal of them is to create impression that X (Microsoft and Windows in this case) is a super-puper ahead of the world. I've seen on the Web a good article about usual tricks used by vendors to cheat benchmarks to get good marketing material, but, unfortunately, can't find link on it at the moment. The problem is that you can't say from such tests if X will also "ahead of the world" on real life usages, because such tests always heavily optimized for particular used benchmarks and such optimizations almost always hurt real life cases. And you hardly find descriptions of those optimizations as well as a scientific description of the tests themself. The results published practically only in marketing documents. Anyway, as far as I can see Linux supports all the used hardware as well as all advance performance modes of it, so if one repeats this test in the same setup, he/she should get not worse results. For me personally it was funny to see how MS presents in the WinHEC presentation (http://download.microsoft.com/download/5/E/6/5E66B27B-988B-4F50-AF3A-C2FF1E62180F/COR-T586_WH08.pptx) that they have 1.1GB/s from 4 connections. In the beginning of 2008 I saw a *single* dd pushing data on that rate over a *single* connection from Linux initiator to iSCSI-SCST target using regular Myricom hardware without any special acceleration. I didn't know how proud I must have been for Linux :). It seems they've described the setup here: http://communities.intel.com/community/wired/blog/2010/04/20/1-million-iop-article-explained And today they seem to have a demo which produces 1.3 million IOPS! "1 Million IOPS? How about 1.25 Million!": http://communities.intel.com/community/wired/blog/2010/04/22/1-million-iops-how-about-125-million I'm glad for them. The only thing surprises me that none of the Linux vendors, including Intel itself, interested to repeat this test for Linux and fix possible found problems, if any. Ten years ago similar test about Linux TCP scalability limitations comparing with Windows caused massive reaction and great TCP improvements. Yeah, I'd like to see this aswell. I don't think I have enough extra hardware myself.. atm. Does someone have enough boxes with 10 Gbit connections? :) The way how to do the test is quite straightforward, starting from making for Linux similarly effective test tool as IOMeter on Windows [1]. Maybe, the lack of such tool scares the vendors away? I'm wondering how big effort it would be to fix IOmeter for linux.. iirc there were some patches to fix the AIO stuff. the AIO stuff inside IOMeter won't necessarily help, since the AIO implementation in linux kernels is not efficient enough* * note: i'm only updated to kernel 2.6.18 - but i didn't here there was a strong effort to make this better in newer kernels. correct me if i'm wrong. --guy -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet, 1.25 million IOPS update
On Mon, Jun 14, 2010 at 11:39:47PM +0400, Vladislav Bolkhovitin wrote: > Pasi Kärkkäinen, on 06/11/2010 11:26 AM wrote: >> On Fri, Feb 05, 2010 at 02:10:32PM +0300, Vladislav Bolkhovitin wrote: >>> Pasi Kärkkäinen, on 01/28/2010 03:36 PM wrote: Hello list, Please check these news items: http://blog.fosketts.net/2010/01/14/microsoft-intel-push-million-iscsi-iops/ http://communities.intel.com/community/openportit/server/blog/2010/01/19/100-iops-with-iscsi--thats-not-a-typo http://www.infostor.com/index/blogs_new/dave_simpson_storage/blogs/infostor/dave_simpon_storage/post987_37501094375591341.html "1,030,000 IOPS over a single 10 Gb Ethernet link" "Specifically, Intel and Microsoft clocked 1,030,000 IOPS (with 512-byte blocks), and more than 2,250MBps with large block sizes (16KB to 256KB) using the Iometer benchmark" So.. who wants to beat that using Linux + open-iscsi? :) >>> I personally, don't like such tests and don't trust them at all. They >>> are pure marketing. The only goal of them is to create impression >>> that X (Microsoft and Windows in this case) is a super-puper ahead >>> of the world. I've seen on the Web a good article about usual tricks >>> used by vendors to cheat benchmarks to get good marketing material, >>> but, unfortunately, can't find link on it at the moment. >>> >>> The problem is that you can't say from such tests if X will also >>> "ahead of the world" on real life usages, because such tests always >>> heavily optimized for particular used benchmarks and such >>> optimizations almost always hurt real life cases. And you hardly >>> find descriptions of those optimizations as well as a scientific >>> description of the tests themself. The results published practically >>> only in marketing documents. >>> >>> Anyway, as far as I can see Linux supports all the used hardware as >>> well as all advance performance modes of it, so if one repeats this >>> test in the same setup, he/she should get not worse results. >>> >>> For me personally it was funny to see how MS presents in the WinHEC >>> presentation >>> (http://download.microsoft.com/download/5/E/6/5E66B27B-988B-4F50-AF3A-C2FF1E62180F/COR-T586_WH08.pptx) >>> >>> that they have 1.1GB/s from 4 connections. In the beginning of 2008 I >>> saw a *single* dd pushing data on that rate over a *single* >>> connection from Linux initiator to iSCSI-SCST target using regular >>> Myricom hardware without any special acceleration. I didn't know how >>> proud I must have been for Linux :). >>> >> >> It seems they've described the setup here: >> http://communities.intel.com/community/wired/blog/2010/04/20/1-million-iop-article-explained >> >> And today they seem to have a demo which produces 1.3 million IOPS! >> >> "1 Million IOPS? How about 1.25 Million!": >> http://communities.intel.com/community/wired/blog/2010/04/22/1-million-iops-how-about-125-million > > I'm glad for them. The only thing surprises me that none of the Linux > vendors, including Intel itself, interested to repeat this test for > Linux and fix possible found problems, if any. Ten years ago similar > test about Linux TCP scalability limitations comparing with Windows > caused massive reaction and great TCP improvements. > Yeah, I'd like to see this aswell. I don't think I have enough extra hardware myself.. atm. Does someone have enough boxes with 10 Gbit connections? :) > The way how to do the test is quite straightforward, starting from > making for Linux similarly effective test tool as IOMeter on Windows > [1]. Maybe, the lack of such tool scares the vendors away? > I'm wondering how big effort it would be to fix IOmeter for linux.. iirc there were some patches to fix the AIO stuff. > Vlad > > [1] None of the performance measurement tools for Linux I've seen so > far, including disktest (although I've not looked at newer (1-1.5 years) > versions) and fio satisfied me for various reasons. > What's missing from ltp disktest? -- Pasi -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet, 1.25 million IOPS update
guy keren, on 06/15/2010 01:46 AM wrote: Vladislav Bolkhovitin wrote: Pasi Kärkkäinen, on 06/11/2010 11:26 AM wrote: On Fri, Feb 05, 2010 at 02:10:32PM +0300, Vladislav Bolkhovitin wrote: Pasi Kärkkäinen, on 01/28/2010 03:36 PM wrote: Hello list, Please check these news items: http://blog.fosketts.net/2010/01/14/microsoft-intel-push-million-iscsi-iops/ http://communities.intel.com/community/openportit/server/blog/2010/01/19/100-iops-with-iscsi--thats-not-a-typo http://www.infostor.com/index/blogs_new/dave_simpson_storage/blogs/infostor/dave_simpon_storage/post987_37501094375591341.html "1,030,000 IOPS over a single 10 Gb Ethernet link" "Specifically, Intel and Microsoft clocked 1,030,000 IOPS (with 512-byte blocks), and more than 2,250MBps with large block sizes (16KB to 256KB) using the Iometer benchmark" So.. who wants to beat that using Linux + open-iscsi? :) I personally, don't like such tests and don't trust them at all. They are pure marketing. The only goal of them is to create impression that X (Microsoft and Windows in this case) is a super-puper ahead of the world. I've seen on the Web a good article about usual tricks used by vendors to cheat benchmarks to get good marketing material, but, unfortunately, can't find link on it at the moment. The problem is that you can't say from such tests if X will also "ahead of the world" on real life usages, because such tests always heavily optimized for particular used benchmarks and such optimizations almost always hurt real life cases. And you hardly find descriptions of those optimizations as well as a scientific description of the tests themself. The results published practically only in marketing documents. Anyway, as far as I can see Linux supports all the used hardware as well as all advance performance modes of it, so if one repeats this test in the same setup, he/she should get not worse results. For me personally it was funny to see how MS presents in the WinHEC presentation (http://download.microsoft.com/download/5/E/6/5E66B27B-988B-4F50-AF3A-C2FF1E62180F/COR-T586_WH08.pptx) that they have 1.1GB/s from 4 connections. In the beginning of 2008 I saw a *single* dd pushing data on that rate over a *single* connection from Linux initiator to iSCSI-SCST target using regular Myricom hardware without any special acceleration. I didn't know how proud I must have been for Linux :). It seems they've described the setup here: http://communities.intel.com/community/wired/blog/2010/04/20/1-million-iop-article-explained And today they seem to have a demo which produces 1.3 million IOPS! "1 Million IOPS? How about 1.25 Million!": http://communities.intel.com/community/wired/blog/2010/04/22/1-million-iops-how-about-125-million I'm glad for them. The only thing surprises me that none of the Linux vendors, including Intel itself, interested to repeat this test for Linux and fix possible found problems, if any. Ten years ago similar test about Linux TCP scalability limitations comparing with Windows caused massive reaction and great TCP improvements. The way how to do the test is quite straightforward, starting from making for Linux similarly effective test tool as IOMeter on Windows [1]. Maybe, the lack of such tool scares the vendors away? Vlad [1] None of the performance measurement tools for Linux I've seen so far, including disktest (although I've not looked at newer (1-1.5 years) versions) and fio satisfied me for various reasons. there's iometer agent (dynamo) for linux (but the official version has some one fundamental flow, which should be fixed - it doesn't use AIO properly) - you just need a windows desktop to launch the test, and run the dynamo agent on a linux machine. there is also vdbench from sun. vdbench is Java-based, so not suitable for this particular case, where low CPU overhead is a key. --guy -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet, 1.25 million IOPS update
Vladislav Bolkhovitin wrote: Pasi Kärkkäinen, on 06/11/2010 11:26 AM wrote: On Fri, Feb 05, 2010 at 02:10:32PM +0300, Vladislav Bolkhovitin wrote: Pasi Kärkkäinen, on 01/28/2010 03:36 PM wrote: Hello list, Please check these news items: http://blog.fosketts.net/2010/01/14/microsoft-intel-push-million-iscsi-iops/ http://communities.intel.com/community/openportit/server/blog/2010/01/19/100-iops-with-iscsi--thats-not-a-typo http://www.infostor.com/index/blogs_new/dave_simpson_storage/blogs/infostor/dave_simpon_storage/post987_37501094375591341.html "1,030,000 IOPS over a single 10 Gb Ethernet link" "Specifically, Intel and Microsoft clocked 1,030,000 IOPS (with 512-byte blocks), and more than 2,250MBps with large block sizes (16KB to 256KB) using the Iometer benchmark" So.. who wants to beat that using Linux + open-iscsi? :) I personally, don't like such tests and don't trust them at all. They are pure marketing. The only goal of them is to create impression that X (Microsoft and Windows in this case) is a super-puper ahead of the world. I've seen on the Web a good article about usual tricks used by vendors to cheat benchmarks to get good marketing material, but, unfortunately, can't find link on it at the moment. The problem is that you can't say from such tests if X will also "ahead of the world" on real life usages, because such tests always heavily optimized for particular used benchmarks and such optimizations almost always hurt real life cases. And you hardly find descriptions of those optimizations as well as a scientific description of the tests themself. The results published practically only in marketing documents. Anyway, as far as I can see Linux supports all the used hardware as well as all advance performance modes of it, so if one repeats this test in the same setup, he/she should get not worse results. For me personally it was funny to see how MS presents in the WinHEC presentation (http://download.microsoft.com/download/5/E/6/5E66B27B-988B-4F50-AF3A-C2FF1E62180F/COR-T586_WH08.pptx) that they have 1.1GB/s from 4 connections. In the beginning of 2008 I saw a *single* dd pushing data on that rate over a *single* connection from Linux initiator to iSCSI-SCST target using regular Myricom hardware without any special acceleration. I didn't know how proud I must have been for Linux :). It seems they've described the setup here: http://communities.intel.com/community/wired/blog/2010/04/20/1-million-iop-article-explained And today they seem to have a demo which produces 1.3 million IOPS! "1 Million IOPS? How about 1.25 Million!": http://communities.intel.com/community/wired/blog/2010/04/22/1-million-iops-how-about-125-million I'm glad for them. The only thing surprises me that none of the Linux vendors, including Intel itself, interested to repeat this test for Linux and fix possible found problems, if any. Ten years ago similar test about Linux TCP scalability limitations comparing with Windows caused massive reaction and great TCP improvements. The way how to do the test is quite straightforward, starting from making for Linux similarly effective test tool as IOMeter on Windows [1]. Maybe, the lack of such tool scares the vendors away? Vlad [1] None of the performance measurement tools for Linux I've seen so far, including disktest (although I've not looked at newer (1-1.5 years) versions) and fio satisfied me for various reasons. there's iometer agent (dynamo) for linux (but the official version has some one fundamental flow, which should be fixed - it doesn't use AIO properly) - you just need a windows desktop to launch the test, and run the dynamo agent on a linux machine. there is also vdbench from sun. --guy -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet, 1.25 million IOPS update
Pasi Kärkkäinen, on 06/11/2010 11:26 AM wrote: On Fri, Feb 05, 2010 at 02:10:32PM +0300, Vladislav Bolkhovitin wrote: Pasi Kärkkäinen, on 01/28/2010 03:36 PM wrote: Hello list, Please check these news items: http://blog.fosketts.net/2010/01/14/microsoft-intel-push-million-iscsi-iops/ http://communities.intel.com/community/openportit/server/blog/2010/01/19/100-iops-with-iscsi--thats-not-a-typo http://www.infostor.com/index/blogs_new/dave_simpson_storage/blogs/infostor/dave_simpon_storage/post987_37501094375591341.html "1,030,000 IOPS over a single 10 Gb Ethernet link" "Specifically, Intel and Microsoft clocked 1,030,000 IOPS (with 512-byte blocks), and more than 2,250MBps with large block sizes (16KB to 256KB) using the Iometer benchmark" So.. who wants to beat that using Linux + open-iscsi? :) I personally, don't like such tests and don't trust them at all. They are pure marketing. The only goal of them is to create impression that X (Microsoft and Windows in this case) is a super-puper ahead of the world. I've seen on the Web a good article about usual tricks used by vendors to cheat benchmarks to get good marketing material, but, unfortunately, can't find link on it at the moment. The problem is that you can't say from such tests if X will also "ahead of the world" on real life usages, because such tests always heavily optimized for particular used benchmarks and such optimizations almost always hurt real life cases. And you hardly find descriptions of those optimizations as well as a scientific description of the tests themself. The results published practically only in marketing documents. Anyway, as far as I can see Linux supports all the used hardware as well as all advance performance modes of it, so if one repeats this test in the same setup, he/she should get not worse results. For me personally it was funny to see how MS presents in the WinHEC presentation (http://download.microsoft.com/download/5/E/6/5E66B27B-988B-4F50-AF3A-C2FF1E62180F/COR-T586_WH08.pptx) that they have 1.1GB/s from 4 connections. In the beginning of 2008 I saw a *single* dd pushing data on that rate over a *single* connection from Linux initiator to iSCSI-SCST target using regular Myricom hardware without any special acceleration. I didn't know how proud I must have been for Linux :). It seems they've described the setup here: http://communities.intel.com/community/wired/blog/2010/04/20/1-million-iop-article-explained And today they seem to have a demo which produces 1.3 million IOPS! "1 Million IOPS? How about 1.25 Million!": http://communities.intel.com/community/wired/blog/2010/04/22/1-million-iops-how-about-125-million I'm glad for them. The only thing surprises me that none of the Linux vendors, including Intel itself, interested to repeat this test for Linux and fix possible found problems, if any. Ten years ago similar test about Linux TCP scalability limitations comparing with Windows caused massive reaction and great TCP improvements. The way how to do the test is quite straightforward, starting from making for Linux similarly effective test tool as IOMeter on Windows [1]. Maybe, the lack of such tool scares the vendors away? Vlad [1] None of the performance measurement tools for Linux I've seen so far, including disktest (although I've not looked at newer (1-1.5 years) versions) and fio satisfied me for various reasons. -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet, 1.25 million IOPS update
On Fri, Feb 05, 2010 at 02:10:32PM +0300, Vladislav Bolkhovitin wrote: > Pasi Kärkkäinen, on 01/28/2010 03:36 PM wrote: >> Hello list, >> >> Please check these news items: >> http://blog.fosketts.net/2010/01/14/microsoft-intel-push-million-iscsi-iops/ >> http://communities.intel.com/community/openportit/server/blog/2010/01/19/100-iops-with-iscsi--thats-not-a-typo >> http://www.infostor.com/index/blogs_new/dave_simpson_storage/blogs/infostor/dave_simpon_storage/post987_37501094375591341.html >> >> "1,030,000 IOPS over a single 10 Gb Ethernet link" >> >> "Specifically, Intel and Microsoft clocked 1,030,000 IOPS (with >> 512-byte blocks), and more than 2,250MBps with large block sizes (16KB >> to 256KB) using the Iometer benchmark" >> >> So.. who wants to beat that using Linux + open-iscsi? :) > > I personally, don't like such tests and don't trust them at all. They > are pure marketing. The only goal of them is to create impression that X > (Microsoft and Windows in this case) is a super-puper ahead of the > world. I've seen on the Web a good article about usual tricks used by > vendors to cheat benchmarks to get good marketing material, but, > unfortunately, can't find link on it at the moment. > > The problem is that you can't say from such tests if X will also "ahead > of the world" on real life usages, because such tests always heavily > optimized for particular used benchmarks and such optimizations almost > always hurt real life cases. And you hardly find descriptions of those > optimizations as well as a scientific description of the tests themself. > The results published practically only in marketing documents. > > Anyway, as far as I can see Linux supports all the used hardware as well > as all advance performance modes of it, so if one repeats this test in > the same setup, he/she should get not worse results. > > For me personally it was funny to see how MS presents in the WinHEC > presentation > (http://download.microsoft.com/download/5/E/6/5E66B27B-988B-4F50-AF3A-C2FF1E62180F/COR-T586_WH08.pptx) > > that they have 1.1GB/s from 4 connections. In the beginning of 2008 I > saw a *single* dd pushing data on that rate over a *single* connection > from Linux initiator to iSCSI-SCST target using regular Myricom hardware > without any special acceleration. I didn't know how proud I must have > been for Linux :). > It seems they've described the setup here: http://communities.intel.com/community/wired/blog/2010/04/20/1-million-iop-article-explained And today they seem to have a demo which produces 1.3 million IOPS! "1 Million IOPS? How about 1.25 Million!": http://communities.intel.com/community/wired/blog/2010/04/22/1-million-iops-how-about-125-million -- Pasi -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet
Pasi Kärkkäinen, on 04/12/2010 11:54 PM wrote: On Fri, Feb 05, 2010 at 02:10:32PM +0300, Vladislav Bolkhovitin wrote: For me personally it was funny to see how MS presents in the WinHEC presentation (http://download.microsoft.com/download/5/E/6/5E66B27B-988B-4F50-AF3A-C2FF1E62180F/COR-T586_WH08.pptx) that they have 1.1GB/s from 4 connections. In the beginning of 2008 I saw a *single* dd pushing data on that rate over a *single* connection from Linux initiator to iSCSI-SCST target using regular Myricom hardware without any special acceleration. I didn't know how proud I must have been for Linux :). Btw was this over 10 Gig Ethernet? Did you have to tweak something special to achieve this, either on the initiator, or on the target? Nothing special. It was for plain dd writes. Vlad -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet
On Fri, Feb 05, 2010 at 02:10:32PM +0300, Vladislav Bolkhovitin wrote: > > For me personally it was funny to see how MS presents in the WinHEC > presentation > (http://download.microsoft.com/download/5/E/6/5E66B27B-988B-4F50-AF3A-C2FF1E62180F/COR-T586_WH08.pptx) > > that they have 1.1GB/s from 4 connections. In the beginning of 2008 I > saw a *single* dd pushing data on that rate over a *single* connection > from Linux initiator to iSCSI-SCST target using regular Myricom hardware > without any special acceleration. I didn't know how proud I must have > been for Linux :). > Btw was this over 10 Gig Ethernet? Did you have to tweak something special to achieve this, either on the initiator, or on the target? -- Pasi -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet
On Mon, Feb 22, 2010 at 02:03:59PM -0800, ByteEnable wrote: > This was achieved by using cache. > What cache? Obviously the iscsi targets were ramdisks, but what do you mean with caching? -- Pasi > On Jan 28, 6:36 am, Pasi Kärkkäinen wrote: > > Hello list, > > > > Please check these news > > items:http://blog.fosketts.net/2010/01/14/microsoft-intel-push-million-iscs...http://communities.intel.com/community/openportit/server/blog/2010/01...http://www.infostor.com/index/blogs_new/dave_simpson_storage/blogs/in... > > > > "1,030,000 IOPS over a single 10 Gb Ethernet link" > > > > "Specifically, Intel and Microsoft clocked 1,030,000 IOPS (with 512-byte > > blocks), > > and more than 2,250MBps with large block sizes (16KB to 256KB) using the > > Iometer benchmark" > > > > So.. who wants to beat that using Linux + open-iscsi? :) > > > > -- Pasi > > -- > You received this message because you are subscribed to the Google Groups > "open-iscsi" group. > To post to this group, send email to open-is...@googlegroups.com. > To unsubscribe from this group, send email to > open-iscsi+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/open-iscsi?hl=en. > -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet
This was achieved by using cache. On Jan 28, 6:36 am, Pasi Kärkkäinen wrote: > Hello list, > > Please check these news > items:http://blog.fosketts.net/2010/01/14/microsoft-intel-push-million-iscs...http://communities.intel.com/community/openportit/server/blog/2010/01...http://www.infostor.com/index/blogs_new/dave_simpson_storage/blogs/in... > > "1,030,000 IOPS over a single 10 Gb Ethernet link" > > "Specifically, Intel and Microsoft clocked 1,030,000 IOPS (with 512-byte > blocks), > and more than 2,250MBps with large block sizes (16KB to 256KB) using the > Iometer benchmark" > > So.. who wants to beat that using Linux + open-iscsi? :) > > -- Pasi -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet
Pasi Kärkkäinen, on 02/08/2010 02:58 PM wrote: On Fri, Feb 05, 2010 at 02:10:32PM +0300, Vladislav Bolkhovitin wrote: Pasi Kärkkäinen, on 01/28/2010 03:36 PM wrote: Hello list, Please check these news items: http://blog.fosketts.net/2010/01/14/microsoft-intel-push-million-iscsi-iops/ http://communities.intel.com/community/openportit/server/blog/2010/01/19/100-iops-with-iscsi--thats-not-a-typo http://www.infostor.com/index/blogs_new/dave_simpson_storage/blogs/infostor/dave_simpon_storage/post987_37501094375591341.html "1,030,000 IOPS over a single 10 Gb Ethernet link" "Specifically, Intel and Microsoft clocked 1,030,000 IOPS (with 512-byte blocks), and more than 2,250MBps with large block sizes (16KB to 256KB) using the Iometer benchmark" So.. who wants to beat that using Linux + open-iscsi? :) I personally, don't like such tests and don't trust them at all. They are pure marketing. The only goal of them is to create impression that X (Microsoft and Windows in this case) is a super-puper ahead of the world. I've seen on the Web a good article about usual tricks used by vendors to cheat benchmarks to get good marketing material, but, unfortunately, can't find link on it at the moment. The problem is that you can't say from such tests if X will also "ahead of the world" on real life usages, because such tests always heavily optimized for particular used benchmarks and such optimizations almost always hurt real life cases. And you hardly find descriptions of those optimizations as well as a scientific description of the tests themself. The results published practically only in marketing documents. Anyway, as far as I can see Linux supports all the used hardware as well as all advance performance modes of it, so if one repeats this test in the same setup, he/she should get not worse results. For me personally it was funny to see how MS presents in the WinHEC presentation (http://download.microsoft.com/download/5/E/6/5E66B27B-988B-4F50-AF3A-C2FF1E62180F/COR-T586_WH08.pptx) that they have 1.1GB/s from 4 connections. In the beginning of 2008 I saw a *single* dd pushing data on that rate over a *single* connection from Linux initiator to iSCSI-SCST target using regular Myricom hardware without any special acceleration. I didn't know how proud I must have been for Linux :). Hehe, congrats :) Did you ever benchmark/measure what kind of IOPS numbers you can get? No. I was solving a task to get max linear throughput from a single initiator, target, link and connection with a HDD RAID6 backstorage on the target. Vlad -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet
On Fri, Feb 05, 2010 at 02:10:32PM +0300, Vladislav Bolkhovitin wrote: > Pasi Kärkkäinen, on 01/28/2010 03:36 PM wrote: >> Hello list, >> >> Please check these news items: >> http://blog.fosketts.net/2010/01/14/microsoft-intel-push-million-iscsi-iops/ >> http://communities.intel.com/community/openportit/server/blog/2010/01/19/100-iops-with-iscsi--thats-not-a-typo >> http://www.infostor.com/index/blogs_new/dave_simpson_storage/blogs/infostor/dave_simpon_storage/post987_37501094375591341.html >> >> "1,030,000 IOPS over a single 10 Gb Ethernet link" >> >> "Specifically, Intel and Microsoft clocked 1,030,000 IOPS (with >> 512-byte blocks), and more than 2,250MBps with large block sizes (16KB >> to 256KB) using the Iometer benchmark" >> >> So.. who wants to beat that using Linux + open-iscsi? :) > > I personally, don't like such tests and don't trust them at all. They > are pure marketing. The only goal of them is to create impression that X > (Microsoft and Windows in this case) is a super-puper ahead of the > world. I've seen on the Web a good article about usual tricks used by > vendors to cheat benchmarks to get good marketing material, but, > unfortunately, can't find link on it at the moment. > > The problem is that you can't say from such tests if X will also "ahead > of the world" on real life usages, because such tests always heavily > optimized for particular used benchmarks and such optimizations almost > always hurt real life cases. And you hardly find descriptions of those > optimizations as well as a scientific description of the tests themself. > The results published practically only in marketing documents. > > Anyway, as far as I can see Linux supports all the used hardware as well > as all advance performance modes of it, so if one repeats this test in > the same setup, he/she should get not worse results. > > For me personally it was funny to see how MS presents in the WinHEC > presentation > (http://download.microsoft.com/download/5/E/6/5E66B27B-988B-4F50-AF3A-C2FF1E62180F/COR-T586_WH08.pptx) > > that they have 1.1GB/s from 4 connections. In the beginning of 2008 I > saw a *single* dd pushing data on that rate over a *single* connection > from Linux initiator to iSCSI-SCST target using regular Myricom hardware > without any special acceleration. I didn't know how proud I must have > been for Linux :). > Hehe, congrats :) Did you ever benchmark/measure what kind of IOPS numbers you can get? -- Pasi -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet
Nicholas A. Bellinger, on 01/29/2010 07:25 PM wrote: On Thu, 2010-01-28 at 20:45 +0200, Pasi Kärkkäinen wrote: On Thu, Jan 28, 2010 at 07:38:28PM +0100, Bart Van Assche wrote: On Thu, Jan 28, 2010 at 4:01 PM, Joe Landman wrote: Pasi Kärkkäinen wrote: Please check these news items: http://blog.fosketts.net/2010/01/14/microsoft-intel-push-million-iscsi-iops/ http://communities.intel.com/community/openportit/server/blog/2010/01/19/100-iops-with-iscsi--thats-not-a-typo http://www.infostor.com/index/blogs_new/dave_simpson_storage/blogs/infostor/dave_simpon_storage/post987_37501094375591341.html "1,030,000 IOPS over a single 10 Gb Ethernet link" This is less than 1us per IOP. Interesting. Their hardware may not actually support this. 10GbE typically is 7-10us, though ConnectX and some others get down to 2ish. Which I/O depth has been used in the test ? Latency matters most with an I/O depth of one and is almost irrelevant for high I/O depth values. iirc outstanding I/Os was 20 in that benchmark. Also of interest, according to the following link http://gestaltit.com/featured/top/stephen/wirespeed-10-gb-iscsi/ is that I/Os are being multiplexed across multiple TCP connections using RFC-3720 defined Multiple Connection per Session (MC/S) logic between the MSFT Initiator Nehalem machine and Netapp Target array: "The configuration tested (on the initiator side) was an IBM x3550 with dual 2 GHz CPUs, 4 GB of RAM, and an Intel 82598 adapter. This is not a special server – in fact, it’s pretty low-end! The connection was tuned with RSS, NetDMA, LRO, LSO, and jumbo frames and maxed out over 4 MCS connections per second. I’m not sure what kind of access they were doing (I’ll ask Suzanne), but it’s pretty impressive that the NetApp Filer could push 1,174 megabytes per second!' It just goes to show that software iSCSI MC/S can really scale to some very impressive results with enough x86_64 horsepower behind it.. Well, if on Windows MC/S scales better than MPIO on random IO tests, as it is seen from the WinHEC presentation, it must mean that MPIO on Windows has serious scalability problems. It would well explain why Microsoft is the only OS vendor who pushes MC/S-capable initiator. I'm sure, if in the next version they fix those scalability problems, it will be presented as a great achievement ;). It isn't necessary Linux should also have such problems. (MC/S requires to serialize all the commands across all connections according to commands' SNs, even if preserving the commands delivery order isn't needed. This is a known scalability limitation. MPIO does not require any such commands serialization, hence doesn't have such limitation. It is especially seen on high IOPS tests. For MPIO you can assign each IO thread to particular CPU and make network hardware to put data for corresponding connection on that CPU. It will allow the best CPU caches usage as well as avoid the "cache ping-pong" between CPUs. With MC/S such setup isn't possible, because all commands from all connections must pass single serialization point.) Vlad -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet
Joe Landman, on 01/28/2010 06:01 PM wrote: Pasi Kärkkäinen wrote: Hello list, Please check these news items: http://blog.fosketts.net/2010/01/14/microsoft-intel-push-million-iscsi-iops/ http://communities.intel.com/community/openportit/server/blog/2010/01/19/100-iops-with-iscsi--thats-not-a-typo http://www.infostor.com/index/blogs_new/dave_simpson_storage/blogs/infostor/dave_simpon_storage/post987_37501094375591341.html "1,030,000 IOPS over a single 10 Gb Ethernet link" This is less than 1us per IOP. Interesting. Their hardware may not actually support this. 10GbE typically is 7-10us, though ConnectX and some others get down to 2ish. You can't calculate any latency for this test as 1/IOPS number. Otherwise you can measure that a half a second satellite link has lower latency, than your local 100Mbps net ;). You know, each link has 2 properties: latency and bandwidth. They are *independent* (or orthogonal), this is fundamental. This means that if you know one parameter, you can't figure out another one. So, if you send single 1 byte IO at time, 1/IOPS will mean latency (I suppose 1/bandwidth << 1/IOPS, otherwise latency is 1/IOPS - 1/bandwidth), but if you fully fill the link, i.e. have at least latency*bandwidth data in-flight, with 1 byte IOs 1/IOPS will mean time to transfer each IO, i.e. 1/bandwidth. Hence, tests which fully fill the link, as in this case, are latency insensitive and only test bandwidth, so you can't figure out any latency from them. You will get the same IOPS results (millions, possibly) for any link's latency, even if it's hundreds of seconds. Initiator and target systems can also be considered as "links" with the same 2 independent parameters: latency (time to process each IO) and bandwidth (how many IOs at time can be processed), where bandwidth != latency * CPUs_num, because SMP systems don't scale linearly. Plus, in this test there are many targets working in parallel. Thus, in this test IOs sent through many links, including the above "links", and fully filled them. So, for this test 1/IOPS is close to IO_size/the_worst_link_bandwidth. Vlad -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet
Pasi Kärkkäinen, on 01/28/2010 03:36 PM wrote: Hello list, Please check these news items: http://blog.fosketts.net/2010/01/14/microsoft-intel-push-million-iscsi-iops/ http://communities.intel.com/community/openportit/server/blog/2010/01/19/100-iops-with-iscsi--thats-not-a-typo http://www.infostor.com/index/blogs_new/dave_simpson_storage/blogs/infostor/dave_simpon_storage/post987_37501094375591341.html "1,030,000 IOPS over a single 10 Gb Ethernet link" "Specifically, Intel and Microsoft clocked 1,030,000 IOPS (with 512-byte blocks), and more than 2,250MBps with large block sizes (16KB to 256KB) using the Iometer benchmark" So.. who wants to beat that using Linux + open-iscsi? :) I personally, don't like such tests and don't trust them at all. They are pure marketing. The only goal of them is to create impression that X (Microsoft and Windows in this case) is a super-puper ahead of the world. I've seen on the Web a good article about usual tricks used by vendors to cheat benchmarks to get good marketing material, but, unfortunately, can't find link on it at the moment. The problem is that you can't say from such tests if X will also "ahead of the world" on real life usages, because such tests always heavily optimized for particular used benchmarks and such optimizations almost always hurt real life cases. And you hardly find descriptions of those optimizations as well as a scientific description of the tests themself. The results published practically only in marketing documents. Anyway, as far as I can see Linux supports all the used hardware as well as all advance performance modes of it, so if one repeats this test in the same setup, he/she should get not worse results. For me personally it was funny to see how MS presents in the WinHEC presentation (http://download.microsoft.com/download/5/E/6/5E66B27B-988B-4F50-AF3A-C2FF1E62180F/COR-T586_WH08.pptx) that they have 1.1GB/s from 4 connections. In the beginning of 2008 I saw a *single* dd pushing data on that rate over a *single* connection from Linux initiator to iSCSI-SCST target using regular Myricom hardware without any special acceleration. I didn't know how proud I must have been for Linux :). Vlad -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet
On Mon, Feb 1, 2010 at 10:33 AM, Pasi Kärkkäinen wrote: > > On Thu, Jan 28, 2010 at 01:44:00PM -0500, Joe Landman wrote: > > Pasi Kärkkäinen wrote: > > I suspect that they really aren't seeing ~1us latencies, but that with > > some neat tricks, it appears to be this. > > > > Physically, it isn't there. I'd classify this as a "marketing number" > > (e.g. unachievable in applications that matter). I'd be happy to be > > proven wrong, but there really isn't much to suggest that I am wrong. > > > > I hope I get to test some 10 Gbit Ethernet equipment soon.. let's see > what I can get myself. If you want to repeat this test, please keep in mind that this test is CPU or memory-bound, not I/O-bound. Letting the block layer combine 512-byte blocks into larger blocks stresses the CPU and memory system. As an example, the command below reported between 200.000 and 400.000 IOPS, depending on the performance of the underlying hardware of the system the command was run on: fio --bs=512 --size=1G --buffered=1 --ioengine=sync --numjobs=1 --group_reporting --name=dev-null /dev/null Bart. -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet
On Mon, Feb 01, 2010 at 11:33:53AM +0200, Pasi Kärkkäinen wrote: > On Thu, Jan 28, 2010 at 01:44:00PM -0500, Joe Landman wrote: > > Pasi Kärkkäinen wrote: > > > >> I think SFP+ 10 Gbit has 0.6usec latency.. ? 10GBase-T is 2.6 usec. > > > > > > http://www.mellanox.com/pdf/whitepapers/wp_mellanox_en_Arista.pdf > > > > This one is from 2008.. > > > They are reporting 7+ us latency. ConnectX are a bit better on latency > > than the Intel NICs. > > > > http://www.ednasia.com/article-24923-solarflarearistanetworkspublishtestreportdemonstratinglowlatencywith10gbe-Asia.html > > > > shows latency, server to server of ~5us. This is about what you expect > > (and why 10GbE isn't quite to IB capability yet in low latency apps). > > > > Yeah.. well, I don't seem to be able to find better data :) > Marketing stuff again: http://www.aristanetworks.com/media/system/pdf/CloudNetworkLatency.pdf "Arista Cut-through Design: Intra-Rack Latency: 0.6us, Inter-Rack Latency: 2.4us". > > I suspect that they really aren't seeing ~1us latencies, but that with > > some neat tricks, it appears to be this. > > > > Physically, it isn't there. I'd classify this as a "marketing number" > > (e.g. unachievable in applications that matter). I'd be happy to be > > proven wrong, but there really isn't much to suggest that I am wrong. > > > Thinking about this again.. how can they cheat? That >million IOPS was the iometer reported benchmark result? Of course they used ramdisks on the iscsi targets etc.. -- Pasi -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet
On Thu, Jan 28, 2010 at 01:44:00PM -0500, Joe Landman wrote: > Pasi Kärkkäinen wrote: > >> I think SFP+ 10 Gbit has 0.6usec latency.. ? 10GBase-T is 2.6 usec. > > > http://www.mellanox.com/pdf/whitepapers/wp_mellanox_en_Arista.pdf > This one is from 2008.. > They are reporting 7+ us latency. ConnectX are a bit better on latency > than the Intel NICs. > > http://www.ednasia.com/article-24923-solarflarearistanetworkspublishtestreportdemonstratinglowlatencywith10gbe-Asia.html > > shows latency, server to server of ~5us. This is about what you expect > (and why 10GbE isn't quite to IB capability yet in low latency apps). > Yeah.. well, I don't seem to be able to find better data :) > I suspect that they really aren't seeing ~1us latencies, but that with > some neat tricks, it appears to be this. > > Physically, it isn't there. I'd classify this as a "marketing number" > (e.g. unachievable in applications that matter). I'd be happy to be > proven wrong, but there really isn't much to suggest that I am wrong. > I hope I get to test some 10 Gbit Ethernet equipment soon.. let's see what I can get myself. -- Pasi -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet
Bart Van Assche wrote: "1,030,000 IOPS over a single 10 Gb Ethernet link" This is less than 1us per IOP. Interesting. Their hardware may not actually support this. 10GbE typically is 7-10us, though ConnectX and some others get down to 2ish. Which I/O depth has been used in the test ? Latency matters most with an I/O depth of one and is almost irrelevant for high I/O depth values. Yes, but then they are measuring the efficiency of their caching/queuing, and not of actual physical IOPs. For one of our storage clusters at a customer site, we have demonstrated really fast writes into cache across 8000+ simultaneous writers. Its just that this number is ... well ... meaningless. Bart. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/jackrabbit phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet
Pasi Kärkkäinen wrote: I think SFP+ 10 Gbit has 0.6usec latency.. ? 10GBase-T is 2.6 usec. http://www.mellanox.com/pdf/whitepapers/wp_mellanox_en_Arista.pdf They are reporting 7+ us latency. ConnectX are a bit better on latency than the Intel NICs. http://www.ednasia.com/article-24923-solarflarearistanetworkspublishtestreportdemonstratinglowlatencywith10gbe-Asia.html shows latency, server to server of ~5us. This is about what you expect (and why 10GbE isn't quite to IB capability yet in low latency apps). I suspect that they really aren't seeing ~1us latencies, but that with some neat tricks, it appears to be this. Physically, it isn't there. I'd classify this as a "marketing number" (e.g. unachievable in applications that matter). I'd be happy to be proven wrong, but there really isn't much to suggest that I am wrong. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/jackrabbit phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet
On Thu, 2010-01-28 at 20:45 +0200, Pasi Kärkkäinen wrote: > On Thu, Jan 28, 2010 at 07:38:28PM +0100, Bart Van Assche wrote: > > On Thu, Jan 28, 2010 at 4:01 PM, Joe Landman > > wrote: > > > Pasi Kärkkäinen wrote: > > >> > > >> Please check these news items: > > >> > > >> http://blog.fosketts.net/2010/01/14/microsoft-intel-push-million-iscsi-iops/ > > >> > > >> http://communities.intel.com/community/openportit/server/blog/2010/01/19/100-iops-with-iscsi--thats-not-a-typo > > >> > > >> http://www.infostor.com/index/blogs_new/dave_simpson_storage/blogs/infostor/dave_simpon_storage/post987_37501094375591341.html > > >> > > >> "1,030,000 IOPS over a single 10 Gb Ethernet link" > > > > > > This is less than 1us per IOP. Interesting. Their hardware may not > > > actually support this. 10GbE typically is 7-10us, though ConnectX and > > > some > > > others get down to 2ish. > > > > Which I/O depth has been used in the test ? Latency matters most with > > an I/O depth of one and is almost irrelevant for high I/O depth > > values. > > > > iirc outstanding I/Os was 20 in that benchmark. Also of interest, according to the following link http://gestaltit.com/featured/top/stephen/wirespeed-10-gb-iscsi/ is that I/Os are being multiplexed across multiple TCP connections using RFC-3720 defined Multiple Connection per Session (MC/S) logic between the MSFT Initiator Nehalem machine and Netapp Target array: "The configuration tested (on the initiator side) was an IBM x3550 with dual 2 GHz CPUs, 4 GB of RAM, and an Intel 82598 adapter. This is not a special server – in fact, it’s pretty low-end! The connection was tuned with RSS, NetDMA, LRO, LSO, and jumbo frames and maxed out over 4 MCS connections per second. I’m not sure what kind of access they were doing (I’ll ask Suzanne), but it’s pretty impressive that the NetApp Filer could push 1,174 megabytes per second!' It just goes to show that software iSCSI MC/S can really scale to some very impressive results with enough x86_64 horsepower behind it.. --nab > > -- Pasi > -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet
On Thu, Jan 28, 2010 at 07:38:28PM +0100, Bart Van Assche wrote: > On Thu, Jan 28, 2010 at 4:01 PM, Joe Landman > wrote: > > Pasi Kärkkäinen wrote: > >> > >> Please check these news items: > >> > >> http://blog.fosketts.net/2010/01/14/microsoft-intel-push-million-iscsi-iops/ > >> > >> http://communities.intel.com/community/openportit/server/blog/2010/01/19/100-iops-with-iscsi--thats-not-a-typo > >> > >> http://www.infostor.com/index/blogs_new/dave_simpson_storage/blogs/infostor/dave_simpon_storage/post987_37501094375591341.html > >> > >> "1,030,000 IOPS over a single 10 Gb Ethernet link" > > > > This is less than 1us per IOP. Interesting. Their hardware may not > > actually support this. 10GbE typically is 7-10us, though ConnectX and some > > others get down to 2ish. > > Which I/O depth has been used in the test ? Latency matters most with > an I/O depth of one and is almost irrelevant for high I/O depth > values. > iirc outstanding I/Os was 20 in that benchmark. -- Pasi -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet
On Thu, Jan 28, 2010 at 4:01 PM, Joe Landman wrote: > Pasi Kärkkäinen wrote: >> >> Please check these news items: >> >> http://blog.fosketts.net/2010/01/14/microsoft-intel-push-million-iscsi-iops/ >> >> http://communities.intel.com/community/openportit/server/blog/2010/01/19/100-iops-with-iscsi--thats-not-a-typo >> >> http://www.infostor.com/index/blogs_new/dave_simpson_storage/blogs/infostor/dave_simpon_storage/post987_37501094375591341.html >> >> "1,030,000 IOPS over a single 10 Gb Ethernet link" > > This is less than 1us per IOP. Interesting. Their hardware may not > actually support this. 10GbE typically is 7-10us, though ConnectX and some > others get down to 2ish. Which I/O depth has been used in the test ? Latency matters most with an I/O depth of one and is almost irrelevant for high I/O depth values. Bart. -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet
On Thu, Jan 28, 2010 at 10:01:39AM -0500, Joe Landman wrote: > Pasi Kärkkäinen wrote: >> Hello list, >> >> Please check these news items: >> http://blog.fosketts.net/2010/01/14/microsoft-intel-push-million-iscsi-iops/ >> http://communities.intel.com/community/openportit/server/blog/2010/01/19/100-iops-with-iscsi--thats-not-a-typo >> http://www.infostor.com/index/blogs_new/dave_simpson_storage/blogs/infostor/dave_simpon_storage/post987_37501094375591341.html >> >> "1,030,000 IOPS over a single 10 Gb Ethernet link" > > This is less than 1us per IOP. Interesting. Their hardware may not > actually support this. 10GbE typically is 7-10us, though ConnectX and > some others get down to 2ish. > I think SFP+ 10 Gbit has 0.6usec latency.. ? 10GBase-T is 2.6 usec. -- Pasi -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet
On Thu, Jan 28, 2010 at 05:35:25PM +0100, Bart Van Assche wrote: >On Thu, Jan 28, 2010 at 1:36 PM, Pasi Kärkkäinen <[1]pa...@iki.fi> wrote: > > Please check these news items: > > [2]http://blog.fosketts.net/2010/01/14/microsoft-intel-push-million-iscsi-iops/ > > [3]http://communities.intel.com/community/openportit/server/blog/2010/01/19/100-iops-with-iscsi--thats-not-a-typo > > [4]http://www.infostor.com/index/blogs_new/dave_simpson_storage/blogs/infostor/dave_simpon_storage/post987_37501094375591341.html > > "1,030,000 IOPS over a single 10 Gb Ethernet link" > > "Specifically, Intel and Microsoft clocked 1,030,000 IOPS (with 512-byte > blocks), > and more than 2,250MBps with large block sizes (16KB to 256KB) using the > Iometer benchmark" > > So.. who wants to beat that using Linux + open-iscsi? :) > >A few comments: >* A throughput of 2250 MB/s over a 10 Gb/s link is only possible when >running read and write tests simultaneously and when counting the traffic >that flows in both directions. Obviously.. >* These results say more about the NIC used than about they say about the >iSCSI initiator "software" used. A quote from > > [5]http://msevents.microsoft.com/CUI/WebCastEventDetails.aspx?culture=en-US&EventID=1032432957&CountryCode=US: >Topics we discuss include [ ... ] Advanced iSCSI acceleration features in >Intel Ethernet Server Adapters and how they work with the native iSCSI >support in Windows Sever 2008 R2. > We were just trying to figure out if they used some "Advanced iSCSI acceleration" or not.. Afaik Intel NICs don't really contain much iSCSI acceleration, in addition to the usual TCP/IP acceleration/offloading features.. -- Pasi -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet
On Thu, Jan 28, 2010 at 10:16:24AM -0600, Mike Christie wrote: > On 01/28/2010 06:36 AM, Pasi Kärkkäinen wrote: >> Hello list, >> >> Please check these news items: >> http://blog.fosketts.net/2010/01/14/microsoft-intel-push-million-iscsi-iops/ >> http://communities.intel.com/community/openportit/server/blog/2010/01/19/100-iops-with-iscsi--thats-not-a-typo >> http://www.infostor.com/index/blogs_new/dave_simpson_storage/blogs/infostor/dave_simpon_storage/post987_37501094375591341.html >> >> "1,030,000 IOPS over a single 10 Gb Ethernet link" >> >> "Specifically, Intel and Microsoft clocked 1,030,000 IOPS (with 512-byte >> blocks), > > Did it look like they were using ioat? I think for Windows they support > iscsi and ioat, right? > Hmm.. let's see: http://dlbmodigital.microsoft.com/ppt/TN-100114-JSchwartz_SMorgan_JPlawner-1032432956-FINAL.pdf Performance factors: - iSCSI initiator perf optimizations - Network stack optimizations - Receive Side Scaling (RSS) - Intel Xeon 5500 QPI and integrated memory controller - Intel 82599: HW Acceleration, multi-core scaling with RSS, MSI-X iSCSI and Storage Enhancements in (Windows 2008) R2: - iSCSI Multi-Core and Numa IO - DPC redirection - Dynamic Load Balancing - Storage IO monitoring - CRC Digest Offload - Support for 32 paths at boot time Intel Xeon 5500 Processor Series, Shatters previous iSCSI performance Architecture increases I/O Bandwidth and CPU efficiency - New memory subsystem - Intel Quickpath Interconnect - New I/O subsystem w/ PCIe Gen2 and CRC32-C instruction set Intel Ethernet Adapters, Elements of iSCSI connectivity .. .. 3. Performance - Transport HW off-loads: Ethernet, TCP/IP and IPSEC - Multi-Core I/O scaling integrated with Windows Server - Intel VMDQ: VM mapping off-load with Hyper-V - Intel Xeon Processor 5500 iSCSI CRC digest instruction set Intel Ethernet iSCSI Acceleration 1. Largest portion of storage I/O host processing is in the application & SCSI layers. No off-load possible. 2. Integrated initiator compute insignificant at run time. CPU CRC computation offers maximum data protection. CRC instruction off-load. 3. Header created in SW, Segmentation and checksum off-loaded. Transport Off-load. 4. I/O mapped to cores via classes and queues. Mapping off-load. 5. Transport layer: LRO, LSO, Cksum Rx/Tx, RSS, IPSEC HW Off-load. Transport off-load. So yeah.. no idea if those are IOAT or not. -- Pasi -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet
On Thu, Jan 28, 2010 at 1:36 PM, Pasi Kärkkäinen wrote: > > Please check these news items: > > http://blog.fosketts.net/2010/01/14/microsoft-intel-push-million-iscsi-iops/ > > http://communities.intel.com/community/openportit/server/blog/2010/01/19/100-iops-with-iscsi--thats-not-a-typo > > http://www.infostor.com/index/blogs_new/dave_simpson_storage/blogs/infostor/dave_simpon_storage/post987_37501094375591341.html > > "1,030,000 IOPS over a single 10 Gb Ethernet link" > > "Specifically, Intel and Microsoft clocked 1,030,000 IOPS (with 512-byte > blocks), > and more than 2,250MBps with large block sizes (16KB to 256KB) using the > Iometer benchmark" > > So.. who wants to beat that using Linux + open-iscsi? :) > A few comments: * A throughput of 2250 MB/s over a 10 Gb/s link is only possible when running read and write tests simultaneously and when counting the traffic that flows in both directions. * These results say more about the NIC used than about they say about the iSCSI initiator "software" used. A quote from http://msevents.microsoft.com/CUI/WebCastEventDetails.aspx?culture=en-US&EventID=1032432957&CountryCode=US: Topics we discuss include [ ... ] Advanced iSCSI acceleration features in Intel Ethernet Server Adapters and how they work with the native iSCSI support in Windows Sever 2008 R2. Bart. -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet
Pasi Kärkkäinen wrote: Hello list, Please check these news items: http://blog.fosketts.net/2010/01/14/microsoft-intel-push-million-iscsi-iops/ http://communities.intel.com/community/openportit/server/blog/2010/01/19/100-iops-with-iscsi--thats-not-a-typo http://www.infostor.com/index/blogs_new/dave_simpson_storage/blogs/infostor/dave_simpon_storage/post987_37501094375591341.html "1,030,000 IOPS over a single 10 Gb Ethernet link" This is less than 1us per IOP. Interesting. Their hardware may not actually support this. 10GbE typically is 7-10us, though ConnectX and some others get down to 2ish. "Specifically, Intel and Microsoft clocked 1,030,000 IOPS (with 512-byte blocks), and more than 2,250MBps with large block sizes (16KB to 256KB) using the Iometer benchmark" So.. who wants to beat that using Linux + open-iscsi? :) -- Pasi -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics, Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/jackrabbit phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet
On 01/28/2010 06:36 AM, Pasi Kärkkäinen wrote: Hello list, Please check these news items: http://blog.fosketts.net/2010/01/14/microsoft-intel-push-million-iscsi-iops/ http://communities.intel.com/community/openportit/server/blog/2010/01/19/100-iops-with-iscsi--thats-not-a-typo http://www.infostor.com/index/blogs_new/dave_simpson_storage/blogs/infostor/dave_simpon_storage/post987_37501094375591341.html "1,030,000 IOPS over a single 10 Gb Ethernet link" "Specifically, Intel and Microsoft clocked 1,030,000 IOPS (with 512-byte blocks), Did it look like they were using ioat? I think for Windows they support iscsi and ioat, right? -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Over one million IOPS using software iSCSI and 10 Gbit Ethernet
On Thu, Jan 28, 2010 at 02:36:09PM +0200, Pasi Kärkkäinen wrote: > Hello list, > > Please check these news items: > http://blog.fosketts.net/2010/01/14/microsoft-intel-push-million-iscsi-iops/ > http://communities.intel.com/community/openportit/server/blog/2010/01/19/100-iops-with-iscsi--thats-not-a-typo > http://www.infostor.com/index/blogs_new/dave_simpson_storage/blogs/infostor/dave_simpon_storage/post987_37501094375591341.html > > "1,030,000 IOPS over a single 10 Gb Ethernet link" > > "Specifically, Intel and Microsoft clocked 1,030,000 IOPS (with 512-byte > blocks), > and more than 2,250MBps with large block sizes (16KB to 256KB) using the > Iometer benchmark" > > So.. who wants to beat that using Linux + open-iscsi? :) > Some more information about the benchmark, and MS marketing stuff: http://dlbmodigital.microsoft.com/ppt/TN-100114-JSchwartz_SMorgan_JPlawner-1032432956-FINAL.pdf And here's earlier benchmark using older hardware, from 03/2009: http://gestaltit.com/featured/top/stephen/wirespeed-10-gb-iscsi/ -- Pasi -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Over one million IOPS using software iSCSI and 10 Gbit Ethernet
Hello list, Please check these news items: http://blog.fosketts.net/2010/01/14/microsoft-intel-push-million-iscsi-iops/ http://communities.intel.com/community/openportit/server/blog/2010/01/19/100-iops-with-iscsi--thats-not-a-typo http://www.infostor.com/index/blogs_new/dave_simpson_storage/blogs/infostor/dave_simpon_storage/post987_37501094375591341.html "1,030,000 IOPS over a single 10 Gb Ethernet link" "Specifically, Intel and Microsoft clocked 1,030,000 IOPS (with 512-byte blocks), and more than 2,250MBps with large block sizes (16KB to 256KB) using the Iometer benchmark" So.. who wants to beat that using Linux + open-iscsi? :) -- Pasi -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
