Re: [Aoetools-discuss] AoE blog article
On Thu, Apr 26, 2012 at 03:46:17PM +0200, Lars Täuber spake thusly: > 10GBit ggaoed offering LVs on DRBD on RAID6s I have always used the vblade from Coraid, not ggaoed. I would give it a try but I really don't want to invest any more time in this issue and whether ggaoed is going to be supported long-term. -- Tracy Reed Digital signature attached for your safety. CopilotcoProfessionally Managed PCI Compliant Secure Hosting 866-MY-COPILOT x101 http://copilotco.com pgpktaukxW9Jz.pgp Description: PGP signature -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Aoetools-discuss mailing list Aoetools-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/aoetools-discuss
Re: [Aoetools-discuss] AoE blog article
On Thu, Apr 26, 2012 at 01:17:43PM +0200, Torbjørn Thorsen spake thusly: > * How will using iSCSI change the partition alignment problem ? > I'm guessing you have found the difference the only way that works; by trying. > It seems unclear why AoE would cause the unaligned writes, so maybe there > isn't > a good answer to this question. I presume it changes the partition alignment problem but not actually introducing one. I don't know how it is possible that an alignment problem is being introduced by using a VM. It makes no sense to me. But the performance is what it is. And when doing writes I see lots of corresponding reads indicative of an alignment problem. And it isn't necessary a partition alignment problem: I'm pretty sure that at some point I actually exported a raw unpartitioned hard drive into then Xen VM and still had issues. It almost seems like Xen itself is causing it. > * How do you actually see from the output of iostat that you're writes > aren't aligned ? > I usually use "iostat -dxm 1" to get an overview, but I don't fully > understand all the output. I run iostat and watch the "rrqm/s wrqm/s r/s w/s" columns. If I am inside the VM and I do a: dd if=/dev/zero of=testfile bs=4M then in the iostat output I would expect to see a lot of write requests merged (wrqm/s) and writes (w/s) per second. I would expect to see few if any reads. Instead I see a significant number of reads perhaps totalling around 1/4 as many writes. This causes lots of extra seeking which really slows down IO. > In a non-AoE context, I have a dm RAID6, where I see all the disks and > the md device > when using iostat. > I can there see the md device hitting 100% utilization, while the disk > don't go above ~60%. > Is that a symptom of unaligned writes ? Not necessarily. The main indicator of unaligned writes is seeing a lot of reads when only writes should be happening. -- Tracy Reed pgpr0UQ5TOEvP.pgp Description: PGP signature -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Aoetools-discuss mailing list Aoetools-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/aoetools-discuss
Re: [Aoetools-discuss] AoE blog article
Am Thu, 26 Apr 2012 13:17:40 +0200 Lars Erik Dangvard Jensen schrieb: > FYI: > Using XenServer 5.6 SP2 with AoE-targets (both coraid appliances and > ggaoed) I get the same diskspeeds in XenServer dom0 and VM domU. This is my experience too. But with xen_major : 4 xen_minor : 0 xen_extra : .2_52-0.2.1 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p aoe v47 1GBit => 10GBit ggaoed offering LVs on DRBD on RAID6s Lars -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Aoetools-discuss mailing list Aoetools-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/aoetools-discuss
Re: [Aoetools-discuss] AoE blog article
Very interesting answer, thank you. Partition alignment, and IO performance in general, is something I'm trying to get better grips on. FWIW, we're on AoE 47 (stock Debian kernel), with AoE clients using two AoE devices as a backing device for a RAID1 device. It works well enough, but performance isn't exactly stellar. I have some questions that would be grateful to get the answer to. * How will using iSCSI change the partition alignment problem ? I'm guessing you have found the difference the only way that works; by trying. It seems unclear why AoE would cause the unaligned writes, so maybe there isn't a good answer to this question. * How do you actually see from the output of iostat that you're writes aren't aligned ? I usually use "iostat -dxm 1" to get an overview, but I don't fully understand all the output. In a non-AoE context, I have a dm RAID6, where I see all the disks and the md device when using iostat. I can there see the md device hitting 100% utilization, while the disk don't go above ~60%. Is that a symptom of unaligned writes ? Thanks for your insightful answers. On Thu, Apr 26, 2012 at 13:02, Tracy Reed wrote: > On Wed, Apr 25, 2012 at 01:47:23PM +0200, Lars Täuber spake thusly: >> Tracy Reed schrieb: >> > Reasons why I am currently migrating away from AoE to iSCSI (*sigh*): >> > >> > 1. Disk alignment between Xen VMs and the target. >> >> What do you mean by that? > > I mean that when I configure a Xen VM to use an AoE block device I always had > mis-aligned writes. The Xen dom0 has the block device in /dev/etherd and I put > the block device in the Xen VM config file and make it /dev/xvda inside the > VM. > > If I access the device from dom0 everything is fine. Very fast writes, no > misalignment. But accessing the block device from within the VM causes the > problem. This makes no sense to me and I don't see anything that could cause > alignment to change but it clearly did somehow. This got to be very noticeable > performance-wise and when doing a pure-write benchmark while running iostat on > the target I could see lots of reads happening to backfill partial pages due > to > the misaligned write. > > -- > Tracy Reed > > -- > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > ___ > Aoetools-discuss mailing list > Aoetools-discuss@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/aoetools-discuss > -- Vennlig hilsen Torbjørn Thorsen Utvikler / driftstekniker Trollweb Solutions AS - Professional Magento Partner www.trollweb.no Telefon dagtid: +47 51215300 Telefon kveld/helg: For kunder med Serviceavtale Besøksadresse: Luramyrveien 40, 4313 Sandnes Postadresse: Maurholen 57, 4316 Sandnes Husk at alle våre standard-vilkår alltid er gjeldende -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Aoetools-discuss mailing list Aoetools-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/aoetools-discuss
Re: [Aoetools-discuss] AoE blog article
Den 26/04/2012 kl. 13.02 skrev Tracy Reed: >>> >>> 1. Disk alignment between Xen VMs and the target. >> >> What do you mean by that? > > I mean that when I configure a Xen VM to use an AoE block device I always had > mis-aligned writes. The Xen dom0 has the block device in /dev/etherd and I put > the block device in the Xen VM config file and make it /dev/xvda inside the > VM. > > If I access the device from dom0 everything is fine. Very fast writes, no > misalignment. But accessing the block device from within the VM causes the > problem. This makes no sense to me and I don't see anything that could cause > alignment to change but it clearly did somehow. This got to be very noticeable > performance-wise and when doing a pure-write benchmark while running iostat on > the target I could see lots of reads happening to backfill partial pages due > to > the misaligned write. FYI: Using XenServer 5.6 SP2 with AoE-targets (both coraid appliances and ggaoed) I get the same diskspeeds in XenServer dom0 and VM domU. So /Lars -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Aoetools-discuss mailing list Aoetools-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/aoetools-discuss
Re: [Aoetools-discuss] AoE blog article
On Wed, Apr 25, 2012 at 01:47:23PM +0200, Lars Täuber spake thusly: > Tracy Reed schrieb: > > Reasons why I am currently migrating away from AoE to iSCSI (*sigh*): > > > > 1. Disk alignment between Xen VMs and the target. > > What do you mean by that? I mean that when I configure a Xen VM to use an AoE block device I always had mis-aligned writes. The Xen dom0 has the block device in /dev/etherd and I put the block device in the Xen VM config file and make it /dev/xvda inside the VM. If I access the device from dom0 everything is fine. Very fast writes, no misalignment. But accessing the block device from within the VM causes the problem. This makes no sense to me and I don't see anything that could cause alignment to change but it clearly did somehow. This got to be very noticeable performance-wise and when doing a pure-write benchmark while running iostat on the target I could see lots of reads happening to backfill partial pages due to the misaligned write. -- Tracy Reed pgp9Cv6KCRmlg.pgp Description: PGP signature -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Aoetools-discuss mailing list Aoetools-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/aoetools-discuss
Re: [Aoetools-discuss] AoE blog article
Hi Tracy, Am Tue, 24 Apr 2012 14:42:33 -0700 Tracy Reed schrieb: > Reasons why I am currently migrating away from AoE to iSCSI (*sigh*): > > 1. Disk alignment between Xen VMs and the target. What do you mean by that? Lars -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Aoetools-discuss mailing list Aoetools-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/aoetools-discuss
Re: [Aoetools-discuss] AoE blog article
On Tue, Apr 24, 2012 at 03:50:33PM +, Jeff Sturm spake thusly: > (AoE multipath really demonstrates the advantages of a simple > protocol--there's nothing involved in making it work other than plugging in > additional interfaces.) Agree completely. Simplicity has always been one of the primary reasons I used AoE for so long. Reasons why I am currently migrating away from AoE to iSCSI (*sigh*): 1. Disk alignment between Xen VMs and the target. I've never figured it out and got it working reliably. I've played with partitioning and disk geometry and partition offsets and all kinds of things. I've just never made it work properly and can't pay the performance penalty. Direct machine-to-machine AoE is blazing fast and typically faster than iSCSI. But my use case has always been VMs, from the very beginning of my initial use of AoE. I've really only used it outside of VMs just once. 2. Lack of integration with RHEL/CentOS. iSCSI gets all the work. I can write init scripts and fix this stuff myself if necessary, and have to some degree, it's just more work. > Multipath was important to us for reliability, so the loss of a single switch > would not impact the storage device. I've been using LACP but with multipath you can span switches without needing fancy expensive stacking switches which is pretty cool. I really should consider whether I want to go multipath with iSCSI. -- Tracy Reed pgp72iBSsRUb5.pgp Description: PGP signature -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Aoetools-discuss mailing list Aoetools-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/aoetools-discuss
Re: [Aoetools-discuss] AoE blog article
> -Original Message- > From: Lars Täuber [mailto:taeu...@bbaw.de] > Sent: Tuesday, April 24, 2012 3:52 AM > > Thanks to this thread I just recognized, that the vanilla kernel driver got > stuck at > version 47. While the coraid driver version is 79. > We used opensuse and now are using ubuntu server and both distributions still > distribute version 47. > > Are there known problems/bugs with the driver version 47 or are the > differences > »only« performance wise? If there are bugs with the vanilla linux driver, we haven't stumbled into any. That driver is old but stable. I think you have to install a newer driver to get support for jumbo frames or multipath however--those are among several performance enhancements Coraid has added to the driver. Multipath was important to us for reliability, so the loss of a single switch would not impact the storage device. It also helps for maximum throughput if you are stuck on gigabit Ethernet. (AoE multipath really demonstrates the advantages of a simple protocol--there's nothing involved in making it work other than plugging in additional interfaces.) -Jeff -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Aoetools-discuss mailing list Aoetools-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/aoetools-discuss
Re: [Aoetools-discuss] AoE blog article
Hi there! Am Tue, 24 Apr 2012 03:25:37 + Jeff Sturm schrieb: [...] > They should have merged the module updates into the upstream kernel. Yes. That's my opinion too. Thanks to this thread I just recognized, that the vanilla kernel driver got stuck at version 47. While the coraid driver version is 79. We used opensuse and now are using ubuntu server and both distributions still distribute version 47. Are there known problems/bugs with the driver version 47 or are the differences »only« performance wise? We serve a redundant 12TB SAN over aoe for virtualization. So far we don't had any serious issues with aoe for several years now. Thanks Lars -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Aoetools-discuss mailing list Aoetools-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/aoetools-discuss
Re: [Aoetools-discuss] AoE blog article
> -Original Message- > From: Joshua J. Kugler [mailto:jos...@azariah.com] > Sent: Monday, April 23, 2012 5:54 PM > > > http://www.typinganimal.net/wp/2012/04/16/red-hat-just-doesnt-get-aoe > > Sounds like Coraid needs to build an RPM package that takes care of the > deficiencies > noted in that article. That would save their customers some pain, and > probably > encourage adoption. They should have merged the module updates into the upstream kernel. Red Hat doesn't really like maintaining kernel patches, and won't put in the effort unless it's deemed important to their user base. No need for a separate RPM if a recent aoe driver is part of the kernel. -Jeff -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Aoetools-discuss mailing list Aoetools-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/aoetools-discuss
Re: [Aoetools-discuss] AoE blog article
On Mon, Apr 23, 2012 at 01:54:28PM -0800, Joshua J. Kugler wrote: >On Monday, April 23, 2012, Tracy Reed elucidated thus: >> Here's a good blog article on some of the current problems with AoE >> that are killing its adoption and therefore utility to potential >> Coraid customers: >> >> http://www.typinganimal.net/wp/2012/04/16/red-hat-just-doesnt-get-aoe >> / > >Sounds like Coraid needs to build an RPM package that takes care of the >deficiencies noted in that article. That would save their customers >some pain, and probably encourage adoption. Charlie links to my aoe-kmod package there though [1], which provides this for at least RHEL 5 and 6. I've also got aoetools and cec packages in that repo, so all those bits are just a yum install away once you've configured the repo [2]. I've also notified Coraid about the kmod, but have had nothing back. Of course, it would be nicer to have this stuff supported directly by RedHat, as Charlie is suggesting. Cheers, Gavin [1] http://www.openfusion.net/linux/aoe_on_rhel_centos [2] http://www.openfusion.net/linux/openfusion_rpm_repository -- For Developers, A Lot Can Happen In A Second. Boundary is the first to Know...and Tell You. Monitor Your Applications in Ultra-Fine Resolution. Try it FREE! http://p.sf.net/sfu/Boundary-d2dvs2 ___ Aoetools-discuss mailing list Aoetools-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/aoetools-discuss
Re: [Aoetools-discuss] AoE blog article
On Monday, April 23, 2012, Tracy Reed elucidated thus: > Here's a good blog article on some of the current problems with AoE > that are killing its adoption and therefore utility to potential > Coraid customers: > > http://www.typinganimal.net/wp/2012/04/16/red-hat-just-doesnt-get-aoe > / Sounds like Coraid needs to build an RPM package that takes care of the deficiencies noted in that article. That would save their customers some pain, and probably encourage adoption. j -- Joshua J. Kugler - Fairbanks, Alaska Azariah Enterprises - Programming and Website Design jos...@azariah.com - Jabber: pedah...@gmail.com PGP Key: http://pgp.mit.edu/ ID 0x73B13B6A -- For Developers, A Lot Can Happen In A Second. Boundary is the first to Know...and Tell You. Monitor Your Applications in Ultra-Fine Resolution. Try it FREE! http://p.sf.net/sfu/Boundary-d2dvs2 ___ Aoetools-discuss mailing list Aoetools-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/aoetools-discuss
[Aoetools-discuss] AoE blog article
Here's a good blog article on some of the current problems with AoE that are killing its adoption and therefore utility to potential Coraid customers: http://www.typinganimal.net/wp/2012/04/16/red-hat-just-doesnt-get-aoe/ -- Tracy Reed pgpHVHTwd28x9.pgp Description: PGP signature -- For Developers, A Lot Can Happen In A Second. Boundary is the first to Know...and Tell You. Monitor Your Applications in Ultra-Fine Resolution. Try it FREE! http://p.sf.net/sfu/Boundary-d2dvs2 ___ Aoetools-discuss mailing list Aoetools-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/aoetools-discuss
