Re: [PERFORM] Testing FusionIO
On Mon, 2010-03-08 at 09:38 -0800, Ben Chobot wrote: We've enjoyed our FusionIO drives very much. They can do 100k iops without breaking a sweat. Yeah, performance is excellent. I bet we could get more, but CPU was bottleneck in our test, since it was just a demo server :( -- Devrim GÜNDÜZ PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer PostgreSQL RPM Repository: http://yum.pgrpms.org Community: devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr http://www.gunduz.org Twitter: http://twitter.com/devrimgunduz signature.asc Description: This is a digitally signed message part
Re: [PERFORM] Testing FusionIO
On Wed, 2010-03-17 at 14:30 +0200, Devrim GÜNDÜZ wrote: On Mon, 2010-03-08 at 09:38 -0800, Ben Chobot wrote: We've enjoyed our FusionIO drives very much. They can do 100k iops without breaking a sweat. Yeah, performance is excellent. I bet we could get more, but CPU was bottleneck in our test, since it was just a demo server :( Did you test the drive in all three modes? If so, what sort of differences did you see. I've been hearing bad things from some folks about the quality of the FusionIO drives from a durability standpoint. I'm Unsure if this is vendor specific bias or not, but considering the source (which not vendor specific), I don't think so. -- Brad Nicholson 416-673-4106 Database Administrator, Afilias Canada Corp. -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Testing FusionIO
On Wed, 2010-03-17 at 09:11 -0400, Justin Pitts wrote: On Mar 17, 2010, at 9:03 AM, Brad Nicholson wrote: I've been hearing bad things from some folks about the quality of the FusionIO drives from a durability standpoint. Can you be more specific about that? Durability over what time frame? How many devices in the sample set? How did FusionIO deal with the issue? I didn't get any specifics - as we are looking at other products. It did center around how FusionIO did wear-leveling though. -- Brad Nicholson 416-673-4106 Database Administrator, Afilias Canada Corp. -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Testing FusionIO
On Wed, 2010-03-17 at 09:52 -0400, Justin Pitts wrote: FusionIO is publicly claiming 24 years @ 5TB/day on the 80GB SLC device, which wear levels across 100GB of actual installed capacity. http://community.fusionio.com/forums/p/34/258.aspx#258 20% of overall capacity free for levelling doesn't strike me as a lot. Some of the Enterprise grade stuff we are looking into (like TMS RamSan) leaves 40% (with much larger overall capacity). Also, running that drive at 80GB is the Maximum Capacity mode, which decreases the write performance. Max drive performance would be about 41TB/day, which coincidently works out very close to the 3 year warranty they have on the devices. To counter that: http://www.tomshardware.com/reviews/fusioinio-iodrive-flash,2140-2.html Fusion-io’s wear leveling algorithm is based on a cycle of 5 TB write/erase volume per day, resulting in 24 years run time for the 80 GB model, 48 years for the 160 GB version and 16 years for the MLC-based 320 GB type. However, since 5 TB could be written or erased rather quickly given the performance level, we recommend not relying on these approximations too much. FusionIO's claim _seems_ credible. I'd love to see some evidence to the contrary. Vendor claims always seem credible. The key is to separate the marketing hype from the actual details. Again, I'm just passing along what I heard - which was from a vendor-neutral, major storage consulting firm that decided to stop recommending these drives to clients. Make of that what you will. As an aside, some folks in our Systems Engineering department here did do some testing of FusionIO, and they found that the helper daemons were inefficient and placed a fair amount of load on the server. That might be something to watch of for for those that are testing them. On Mar 17, 2010, at 9:18 AM, Brad Nicholson wrote: On Wed, 2010-03-17 at 09:11 -0400, Justin Pitts wrote: On Mar 17, 2010, at 9:03 AM, Brad Nicholson wrote: I've been hearing bad things from some folks about the quality of the FusionIO drives from a durability standpoint. Can you be more specific about that? Durability over what time frame? How many devices in the sample set? How did FusionIO deal with the issue? I didn't get any specifics - as we are looking at other products. It did center around how FusionIO did wear-leveling though. -- Brad Nicholson 416-673-4106 Database Administrator, Afilias Canada Corp. -- Brad Nicholson 416-673-4106 Database Administrator, Afilias Canada Corp. -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Testing FusionIO
On Mar 17, 2010, at 10:41 AM, Brad Nicholson wrote: On Wed, 2010-03-17 at 09:52 -0400, Justin Pitts wrote: FusionIO is publicly claiming 24 years @ 5TB/day on the 80GB SLC device, which wear levels across 100GB of actual installed capacity. http://community.fusionio.com/forums/p/34/258.aspx#258 20% of overall capacity free for levelling doesn't strike me as a lot. I don't have any idea how to judge what amount would be right. Some of the Enterprise grade stuff we are looking into (like TMS RamSan) leaves 40% (with much larger overall capacity). Also, running that drive at 80GB is the Maximum Capacity mode, which decreases the write performance. Very fair. In my favor, my proposed use case is probably at half capacity or less. I am getting the impression that partitioning/formatting the drive for the intended usage, and not the max capacity, is the way to go. Capacity isn't an issue with this workload. I cannot fit enough drives into these servers to get a tenth of the IOPS that even Tom's documents the ioDrive is capable of at reduced performance levels. Max drive performance would be about 41TB/day, which coincidently works out very close to the 3 year warranty they have on the devices. To counter that: http://www.tomshardware.com/reviews/fusioinio-iodrive-flash,2140-2.html Fusion-io’s wear leveling algorithm is based on a cycle of 5 TB write/erase volume per day, resulting in 24 years run time for the 80 GB model, 48 years for the 160 GB version and 16 years for the MLC-based 320 GB type. However, since 5 TB could be written or erased rather quickly given the performance level, we recommend not relying on these approximations too much. I'm not sure if that is a counter or a supporting claim :) FusionIO's claim _seems_ credible. I'd love to see some evidence to the contrary. Vendor claims always seem credible. The key is to separate the marketing hype from the actual details. I'm hoping to get my hands on a sample in the next few weeks. Again, I'm just passing along what I heard - which was from a vendor-neutral, major storage consulting firm that decided to stop recommending these drives to clients. Make of that what you will. As an aside, some folks in our Systems Engineering department here did do some testing of FusionIO, and they found that the helper daemons were inefficient and placed a fair amount of load on the server. That might be something to watch of for for those that are testing them. That is a wonderful little nugget of knowledge that I shall put on my test plan. -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Testing FusionIO
On Mar 17, 2010, at 9:03 AM, Brad Nicholson wrote: I've been hearing bad things from some folks about the quality of the FusionIO drives from a durability standpoint. Can you be more specific about that? Durability over what time frame? How many devices in the sample set? How did FusionIO deal with the issue? -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Testing FusionIO
FusionIO is publicly claiming 24 years @ 5TB/day on the 80GB SLC device, which wear levels across 100GB of actual installed capacity. http://community.fusionio.com/forums/p/34/258.aspx#258 Max drive performance would be about 41TB/day, which coincidently works out very close to the 3 year warranty they have on the devices. FusionIO's claim _seems_ credible. I'd love to see some evidence to the contrary. On Mar 17, 2010, at 9:18 AM, Brad Nicholson wrote: On Wed, 2010-03-17 at 09:11 -0400, Justin Pitts wrote: On Mar 17, 2010, at 9:03 AM, Brad Nicholson wrote: I've been hearing bad things from some folks about the quality of the FusionIO drives from a durability standpoint. Can you be more specific about that? Durability over what time frame? How many devices in the sample set? How did FusionIO deal with the issue? I didn't get any specifics - as we are looking at other products. It did center around how FusionIO did wear-leveling though. -- Brad Nicholson 416-673-4106 Database Administrator, Afilias Canada Corp. -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Testing FusionIO
Greg, Did you ever contact them and get your hands on one? We eventually did see long SSD rebuild times on server crash as well. But data came back uncorrupted per my blog post. This is a good case for Slony Slaves. Anyone in a high TX low downtime environment would have already engineered around needing to wait for rebuild/recover times anyway. So it's not a deal killer in my view. -kg On Mar 8, 2010, at 12:50 PM, Greg Smith wrote: Ben Chobot wrote: We've enjoyed our FusionIO drives very much. They can do 100k iops without breaking a sweat. Just make sure you shut them down cleanly - it can up to 30 minutes per card to recover from a crash/plug pull test. Yeah...I got into an argument with Kenny Gorman over my concerns with how they were handling durability issues on his blog, the reading I did about them never left me satisfied Fusion was being completely straight with everyone about this area: http://www.kennygorman.com/wordpress/?p=398 If it takes 30 minutes to recover, but it does recover, I guess that's better than I feared was the case with them. Thanks for reporting the plug pull tests--I don't trust any report from anyone about new storage hardware that doesn't include that little detail as part of the testing. You're just asking to have your data get lost without that basic due diligence, and I'm sure not going to even buy eval hardware from a vendor that appears evasive about it. There's a reason I don't personally own any SSD hardware yet. -- Greg Smith 2ndQuadrant US Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.us -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Testing FusionIO
On Wed, 2010-03-17 at 14:11 -0400, Justin Pitts wrote: On Mar 17, 2010, at 10:41 AM, Brad Nicholson wrote: On Wed, 2010-03-17 at 09:52 -0400, Justin Pitts wrote: FusionIO is publicly claiming 24 years @ 5TB/day on the 80GB SLC device, which wear levels across 100GB of actual installed capacity. http://community.fusionio.com/forums/p/34/258.aspx#258 20% of overall capacity free for levelling doesn't strike me as a lot. I don't have any idea how to judge what amount would be right. Some of the Enterprise grade stuff we are looking into (like TMS RamSan) leaves 40% (with much larger overall capacity). Also, running that drive at 80GB is the Maximum Capacity mode, which decreases the write performance. Very fair. In my favor, my proposed use case is probably at half capacity or less. I am getting the impression that partitioning/formatting the drive for the intended usage, and not the max capacity, is the way to go. Capacity isn't an issue with this workload. I cannot fit enough drives into these servers to get a tenth of the IOPS that even Tom's documents the ioDrive is capable of at reduced performance levels. The actual media is only good for a very limited number of write cycles. The way that the drives get around to be reliable is to constantly write to different areas. The more you have free, the less you have to re-use, the longer the lifespan. This is done by the drives wear levelling algorithms, not by using partitioning utilities btw. -- Brad Nicholson 416-673-4106 Database Administrator, Afilias Canada Corp. -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Testing FusionIO
On Wed, 17 Mar 2010, Brad Nicholson wrote: On Wed, 2010-03-17 at 14:11 -0400, Justin Pitts wrote: On Mar 17, 2010, at 10:41 AM, Brad Nicholson wrote: On Wed, 2010-03-17 at 09:52 -0400, Justin Pitts wrote: FusionIO is publicly claiming 24 years @ 5TB/day on the 80GB SLC device, which wear levels across 100GB of actual installed capacity. http://community.fusionio.com/forums/p/34/258.aspx#258 20% of overall capacity free for levelling doesn't strike me as a lot. I don't have any idea how to judge what amount would be right. Some of the Enterprise grade stuff we are looking into (like TMS RamSan) leaves 40% (with much larger overall capacity). Also, running that drive at 80GB is the Maximum Capacity mode, which decreases the write performance. Very fair. In my favor, my proposed use case is probably at half capacity or less. I am getting the impression that partitioning/formatting the drive for the intended usage, and not the max capacity, is the way to go. Capacity isn't an issue with this workload. I cannot fit enough drives into these servers to get a tenth of the IOPS that even Tom's documents the ioDrive is capable of at reduced performance levels. The actual media is only good for a very limited number of write cycles. The way that the drives get around to be reliable is to constantly write to different areas. The more you have free, the less you have to re-use, the longer the lifespan. This is done by the drives wear levelling algorithms, not by using partitioning utilities btw. true, but if the drive is partitioned so that parts of it are never written to by the OS, the drive knows that those parts don't contain data and so can treat them as unallocated. once the OS writes to a part of the drive, unless the OS issues a trim command the drive can't know that the data there is worthless and can be ignored, it has to try and preserve that data, which makes doing the wear leveling harder and slower. David Lang -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Testing FusionIO
On Mar 17, 2010, at 7:41 AM, Brad Nicholson wrote: As an aside, some folks in our Systems Engineering department here did do some testing of FusionIO, and they found that the helper daemons were inefficient and placed a fair amount of load on the server. That might be something to watch of for for those that are testing them. As another anecdote, we have 4 of the 160GB cards in a 24-core Istanbul server. I don't know how efficient the helper daemons are, but they do take up about half of one core's cycles, regardless of how busy the box actually is. So that sounds bad until you take into account how much that one core costs, and compare it to how much it would cost to have the same amount of IOPs in a different form. -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
[PERFORM] Testing FusionIO
Hi, I have a FusionIO drive to test for a few days. I already ran iozone and bonnie++ against it. Does anyone have more suggestions for it? It is a single drive (unfortunately). Regards, -- Devrim GÜNDÜZ PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer PostgreSQL RPM Repository: http://yum.pgrpms.org Community: devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr http://www.gunduz.org Twitter: http://twitter.com/devrimgunduz signature.asc Description: This is a digitally signed message part
Re: [PERFORM] Testing FusionIO
Devrim GÜNDÜZ wrote: Hi, I have a FusionIO drive Cool!! to test for a few days. I already ran iozone and bonnie++ against it. Does anyone have more suggestions for it? Oracle has a tool to test drives specifically for database loads kinds called orion - its free software and comes with a good manual. Download without registration etc at http://www.oracle.com/technology/software/tech/orion/index.html Quickstart create file named named 'fusion.lun' with the device name, e.g. /dev/sda1 Invoke orion tool with something like orion binary -run advanced -testname fusion -num_disks 50 -size_small 4 -size_large 1024 -type rand -simulate concat -verbose -write 25 -duration 15 -matrix detailed -cache_size 256 cache size is in MB's but not so important for random io. num disks doesn't have to match physical disks but it's used by the tool to determine how large the test matrix should be. E.g. 1 disk gives a small matrix with small number of concurrent io requests. So I set it to 50. Another idea: pgbench? regards, Yeb Havinga -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Testing FusionIO
2010/3/8 Devrim GÜNDÜZ dev...@gunduz.org: Hi, I have a FusionIO drive to test for a few days. I already ran iozone and bonnie++ against it. Does anyone have more suggestions for it? It is a single drive (unfortunately). vdbench -- Łukasz Jagiełło System Administrator G-Forces Web Management Polska sp. z o.o. (www.gforces.pl) Ul. Kruczkowskiego 12, 80-288 Gdańsk Spółka wpisana do KRS pod nr 246596 decyzją Sądu Rejonowego Gdańsk-Północ -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Testing FusionIO
We've enjoyed our FusionIO drives very much. They can do 100k iops without breaking a sweat. Just make sure you shut them down cleanly - it can up to 30 minutes per card to recover from a crash/plug pull test. I also have serious questions about their longevity and failure mode when the flash finally burns out. Our hardware guys claim they have overbuilt the amount of flash on the card to be able to do their heavy writes for 5 years, but I remain skeptical. On Mar 8, 2010, at 6:41 AM, Devrim GÜNDÜZ wrote: Hi, I have a FusionIO drive to test for a few days. I already ran iozone and bonnie++ against it. Does anyone have more suggestions for it? It is a single drive (unfortunately). Regards, -- Devrim GÜNDÜZ PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer PostgreSQL RPM Repository: http://yum.pgrpms.org Community: devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr http://www.gunduz.org Twitter: http://twitter.com/devrimgunduz -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Testing FusionIO
Ben Chobot wrote: We've enjoyed our FusionIO drives very much. They can do 100k iops without breaking a sweat. Just make sure you shut them down cleanly - it can up to 30 minutes per card to recover from a crash/plug pull test. Yeah...I got into an argument with Kenny Gorman over my concerns with how they were handling durability issues on his blog, the reading I did about them never left me satisfied Fusion was being completely straight with everyone about this area: http://www.kennygorman.com/wordpress/?p=398 If it takes 30 minutes to recover, but it does recover, I guess that's better than I feared was the case with them. Thanks for reporting the plug pull tests--I don't trust any report from anyone about new storage hardware that doesn't include that little detail as part of the testing. You're just asking to have your data get lost without that basic due diligence, and I'm sure not going to even buy eval hardware from a vendor that appears evasive about it. There's a reason I don't personally own any SSD hardware yet. -- Greg Smith 2ndQuadrant US Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.us -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Testing FusionIO
On Mar 8, 2010, at 12:50 PM, Greg Smith wrote: Ben Chobot wrote: We've enjoyed our FusionIO drives very much. They can do 100k iops without breaking a sweat. Just make sure you shut them down cleanly - it can up to 30 minutes per card to recover from a crash/plug pull test. Yeah...I got into an argument with Kenny Gorman over my concerns with how they were handling durability issues on his blog, the reading I did about them never left me satisfied Fusion was being completely straight with everyone about this area: http://www.kennygorman.com/wordpress/?p=398 If it takes 30 minutes to recover, but it does recover, I guess that's better than I feared was the case with them. Thanks for reporting the plug pull tests--I don't trust any report from anyone about new storage hardware that doesn't include that little detail as part of the testing. You're just asking to have your data get lost without that basic due diligence, and I'm sure not going to even buy eval hardware from a vendor that appears evasive about it. There's a reason I don't personally own any SSD hardware yet. Of course, the plug pull test can never be conclusive, but we never lost any data the handful of times we did it. Normally we'd do it more, but with such a long reboot cycle But from everything we can tell, FusionIO does do reliability right. -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
