Re: [SLUG] Petabytes on a budget
Jake, Most of thenumbers you are showing are just the clocked speeds on busses and cables. Certainly the components can clock at those speeds, but the biggest issue with lower-end components is whether they can actually feed and sustain data at that rate, and how well they handle contention for resources. Generally this comes down to size of buffers, and whether have fast or wide enough processors at those interface points. But it sounds like for the business Backblaze is in they are building something that is big rather than fast (or at least something that is fast enough). And certainly if you can parallelize the system enough (and maintain reliablity) then you probably even achieve cheap and fast. Regards, Martin martinvisse...@gmail.com On Thu, Sep 3, 2009 at 12:40 PM, Jake Anderson ya...@vapourforge.comwrote: On 03/09/09 10:37, Mark Walkom wrote: I was thinking the same, but I reckon because they are just backing up/archiving data it wouldn't be too bad. ie They aren't looking for huge performance, just huge, cheap storage. 2009/9/3 Morgan Storeym...@morganstorey.com I know I am a geek but that is hot. I am wondering if they see any throughput issues with the sata backplanes and pci sata cards. The backplanes are probably fine, each sata cable is good for ~300mbytes/sec most physical disks couldn't hope to hit that. lesse what their maximum xfer rate is. each drive can hit 103mb/sec average (better than I thought) each sata channel will max out at 250mbyte/sec so they are going to be loosing some bandwidth there. their backplanes take 5 disks, so a potential bandwidth of well call it 500mbyte/second so 50% is out the window there actual bandwidth per 5 disks is going to be 250mbyte they have 9 of these channels for a total bandwith available of 2250mbyte/sec (2 gigabytes a second, that'll rip some dvds fast) standard PCI tops out at 133mbyte/sec so thats out ;- it looks like they are using PCI-E SATA cards the mbo they have and the cards they are using indicate they have 3x of something like this http://www.syba.com/index.php?controller=Productaction=InfoId=861 which maxes out at 250mbyte/sec per (1x PCI-E 1x lane) and one 4 port card which if it comes from that mob must be a PCI by the look of things. but I'll assume that its PCI-E and at least 4 lanes. so the total xfer rate is 1750mbyte/sec (or 883 if they are using the PCI card) Vs the total possible xfer rate of 4500 they aren't doing *too* badly given that on a gigabit ethernet connection you are lucky to push 30mbyte/sec (or 60 if you tweak it), I think its not going to be a big issue ;- If they wanted more oomph their best bet would be to put 2x 16 port PCI-E 16x cards into a motherboard that supported it (most decent SLI motherboards will do that) better still one with 4x pci-E 16 slots so that you can put some 10gigE cards in as well something like http://www.aria.co.uk/Products/Components/Motherboards/Socket+AM3+(AMD)/MSI+790FX-GD70+AM3+Motherboard+with+4+x+PCIe+x+16+?productId=36604http://www.aria.co.uk/Products/Components/Motherboards/Socket+AM3+%28AMD%29/MSI+790FX-GD70+AM3+Motherboard+with+4+x+PCIe+x+16+?productId=36604 say (but with intel of course ;-) That should net you (assuming you use dual port 10gig-E nics) an xfer rate out of the box of around 2400mbytes/sec almost fast enough to spy on teh entirez intarwebz!! -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Petabytes on a budget
Also the constraining factor would be the network speed coming in so the actual speed of the hardware would probably not affect it that much. Ken -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Petabytes on a budget
On 03/09/09 16:09, Martin Visser wrote: Jake, Most of thenumbers you are showing are just the clocked speeds on busses and cables. actually no they are benchmarked sustained and average transfer rates for the devices. sata 2 line speed is 3gbit, with overhead it will saturate at 250mbyte/sec the hdd's min transfer rate is 60mbyte/sec and max is 160. Certainly the components can clock at those speeds, but the biggest issue with lower-end components is whether they can actually feed and sustain data at that rate, and how well they handle contention for resources. Generally this comes down to size of buffers, and whether have fast or wide enough processors at those interface points. They aren't really lower end components There is little difference in performance between SATA and SAS these days. PCI/PCI-X will sustain those transfer rates by design, the cards themselves are being little more than a slightly bent pipe so there should be no real bottleneck there. But it sounds like for the business Backblaze is in they are building something that is big rather than fast (or at least something that is fast enough). And certainly if you can parallelize the system enough (and maintain reliablity) then you probably even achieve cheap and fast. Their biggest issue would be seek times. Regards, Martin martinvisse...@gmail.com On Thu, Sep 3, 2009 at 12:40 PM, Jake Andersonya...@vapourforge.comwrote: On 03/09/09 10:37, Mark Walkom wrote: I was thinking the same, but I reckon because they are just backing up/archiving data it wouldn't be too bad. ie They aren't looking for huge performance, just huge, cheap storage. 2009/9/3 Morgan Storeym...@morganstorey.com I know I am a geek but that is hot. I am wondering if they see any throughput issues with the sata backplanes and pci sata cards. The backplanes are probably fine, each sata cable is good for ~300mbytes/sec most physical disks couldn't hope to hit that. lesse what their maximum xfer rate is. each drive can hit 103mb/sec average (better than I thought) each sata channel will max out at 250mbyte/sec so they are going to be loosing some bandwidth there. their backplanes take 5 disks, so a potential bandwidth of well call it 500mbyte/second so 50% is out the window there actual bandwidth per 5 disks is going to be 250mbyte they have 9 of these channels for a total bandwith available of 2250mbyte/sec (2 gigabytes a second, that'll rip some dvds fast) standard PCI tops out at 133mbyte/sec so thats out ;- it looks like they are using PCI-E SATA cards the mbo they have and the cards they are using indicate they have 3x of something like this http://www.syba.com/index.php?controller=Productaction=InfoId=861 which maxes out at 250mbyte/sec per (1x PCI-E 1x lane) and one 4 port card which if it comes from that mob must be a PCI by the look of things. but I'll assume that its PCI-E and at least 4 lanes. so the total xfer rate is 1750mbyte/sec (or 883 if they are using the PCI card) Vs the total possible xfer rate of 4500 they aren't doing *too* badly given that on a gigabit ethernet connection you are lucky to push 30mbyte/sec (or 60 if you tweak it), I think its not going to be a big issue ;- If they wanted more oomph their best bet would be to put 2x 16 port PCI-E 16x cards into a motherboard that supported it (most decent SLI motherboards will do that) better still one with 4x pci-E 16 slots so that you can put some 10gigE cards in as well something like http://www.aria.co.uk/Products/Components/Motherboards/Socket+AM3+(AMD)/MSI+790FX-GD70+AM3+Motherboard+with+4+x+PCIe+x+16+?productId=36604http://www.aria.co.uk/Products/Components/Motherboards/Socket+AM3+%28AMD%29/MSI+790FX-GD70+AM3+Motherboard+with+4+x+PCIe+x+16+?productId=36604 say (but with intel of course ;-) That should net you (assuming you use dual port 10gig-E nics) an xfer rate out of the box of around 2400mbytes/sec almost fast enough to spy on teh entirez intarwebz!! -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html On 03/09/09 16:09, Martin Visser wrote: Jake, Most of thenumbers you are showing are just the clocked speeds on busses and cables. Certainly the components can clock at those speeds, but the biggest issue with lower-end components is whether they can actually feed and sustain data at that rate, and how well they handle contention for resources. Generally this comes down to size of buffers, and whether have fast or wide enough processors at those interface points. But it sounds like for the business Backblaze is in they are building something that is big rather than fast (or at least something that is fast enough). And certainly if you can parallelize the system enough (and maintain reliablity) then you probably even achieve cheap and fast. Regards, Martin martinvisse...@gmail.com On Thu, Sep 3, 2009 at
Re: [SLUG] Petabytes on a budget
Jake == Jake Anderson ya...@vapourforge.com writes: Jake On 03/09/09 16:09, Martin Visser wrote: Jake, Most of thenumbers you are showing are just the clocked speeds on busses and cables. Jake actually no they are benchmarked sustained and average transfer Jake rates for the devices. sata 2 line speed is 3gbit, with Jake overhead it will saturate at 250mbyte/sec the hdd's min transfer Jake rate is 60mbyte/sec and max is 160. Certainly the components can clock at those speeds, but the biggest issue with lower-end components is whether they can actually feed and sustain data at that rate, and how well they handle contention for resources. Generally this comes down to size of buffers, and whether have fast or wide enough processors at those interface points. Jake They aren't really lower end components There is little Jake difference in performance between SATA and SAS these days. Jake PCI/PCI-X will sustain those transfer rates by design, the cards Jake themselves are being little more than a slightly bent pipe so Jake there should be no real bottleneck there. We've seen some problems with some SATA port multipliere, that don't allow more than one drive at a time to be working --- effectively slowing the transfer rate to that of a single spindle. I don't know if the ones these people are using have that problem. -- Dr Peter Chubb http://www.gelato.unsw.edu.au peterc AT gelato.unsw.edu.au http://www.ertos.nicta.com.au ERTOS within National ICT Australia -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
[SLUG] Petabytes on a budget
Thought people might find this interesting. It doesn't get to the cloud bit, but describes the building block they use. Petabytes on a budget: How to build cheap cloud storage http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/ At Backblaze, we provide unlimited storage to our customers for only $5 per month, so we had to figure out how to store hundreds of petabytes of customer data in a reliable, scalable way—and keep our costs low. After looking at several overpriced commercial solutions, we decided to build our own custom Backblaze Storage Pods: 67 terabyte 4U servers for $7,867. -- Phil Scarratt Draxsen Technologies m: 0403 531 271 -- The trouble with having an open mind, of course, is that people will insist on coming along and trying to put things in it. - Terry Pratchett -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Petabytes on a budget
I know I am a geek but that is hot. I am wondering if they see any throughput issues with the sata backplanes and pci sata cards. On Wed, Sep 2, 2009 at 10:55 PM, Phil Scarratt f...@draxsen.com wrote: Thought people might find this interesting. It doesn't get to the cloud bit, but describes the building block they use. Petabytes on a budget: How to build cheap cloud storage http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/ At Backblaze, we provide unlimited storage to our customers for only $5 per month, so we had to figure out how to store hundreds of petabytes of customer data in a reliable, scalable way—and keep our costs low. After looking at several overpriced commercial solutions, we decided to build our own custom Backblaze Storage Pods: 67 terabyte 4U servers for $7,867. -- Phil Scarratt Draxsen Technologies m: 0403 531 271 -- The trouble with having an open mind, of course, is that people will insist on coming along and trying to put things in it. - Terry Pratchett -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Petabytes on a budget
I was thinking the same, but I reckon because they are just backing up/archiving data it wouldn't be too bad. ie They aren't looking for huge performance, just huge, cheap storage. 2009/9/3 Morgan Storey m...@morganstorey.com I know I am a geek but that is hot. I am wondering if they see any throughput issues with the sata backplanes and pci sata cards. On Wed, Sep 2, 2009 at 10:55 PM, Phil Scarratt f...@draxsen.com wrote: Thought people might find this interesting. It doesn't get to the cloud bit, but describes the building block they use. Petabytes on a budget: How to build cheap cloud storage http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/ At Backblaze, we provide unlimited storage to our customers for only $5 per month, so we had to figure out how to store hundreds of petabytes of customer data in a reliable, scalable way—and keep our costs low. After looking at several overpriced commercial solutions, we decided to build our own custom Backblaze Storage Pods: 67 terabyte 4U servers for $7,867. -- Phil Scarratt Draxsen Technologies m: 0403 531 271 -- The trouble with having an open mind, of course, is that people will insist on coming along and trying to put things in it. - Terry Pratchett -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Petabytes on a budget
On 03/09/09 10:37, Mark Walkom wrote: I was thinking the same, but I reckon because they are just backing up/archiving data it wouldn't be too bad. ie They aren't looking for huge performance, just huge, cheap storage. 2009/9/3 Morgan Storeym...@morganstorey.com I know I am a geek but that is hot. I am wondering if they see any throughput issues with the sata backplanes and pci sata cards. The backplanes are probably fine, each sata cable is good for ~300mbytes/sec most physical disks couldn't hope to hit that. lesse what their maximum xfer rate is. each drive can hit 103mb/sec average (better than I thought) each sata channel will max out at 250mbyte/sec so they are going to be loosing some bandwidth there. their backplanes take 5 disks, so a potential bandwidth of well call it 500mbyte/second so 50% is out the window there actual bandwidth per 5 disks is going to be 250mbyte they have 9 of these channels for a total bandwith available of 2250mbyte/sec (2 gigabytes a second, that'll rip some dvds fast) standard PCI tops out at 133mbyte/sec so thats out ;- it looks like they are using PCI-E SATA cards the mbo they have and the cards they are using indicate they have 3x of something like this http://www.syba.com/index.php?controller=Productaction=InfoId=861 which maxes out at 250mbyte/sec per (1x PCI-E 1x lane) and one 4 port card which if it comes from that mob must be a PCI by the look of things. but I'll assume that its PCI-E and at least 4 lanes. so the total xfer rate is 1750mbyte/sec (or 883 if they are using the PCI card) Vs the total possible xfer rate of 4500 they aren't doing *too* badly given that on a gigabit ethernet connection you are lucky to push 30mbyte/sec (or 60 if you tweak it), I think its not going to be a big issue ;- If they wanted more oomph their best bet would be to put 2x 16 port PCI-E 16x cards into a motherboard that supported it (most decent SLI motherboards will do that) better still one with 4x pci-E 16 slots so that you can put some 10gigE cards in as well something like http://www.aria.co.uk/Products/Components/Motherboards/Socket+AM3+(AMD)/MSI+790FX-GD70+AM3+Motherboard+with+4+x+PCIe+x+16+?productId=36604 say (but with intel of course ;-) That should net you (assuming you use dual port 10gig-E nics) an xfer rate out of the box of around 2400mbytes/sec almost fast enough to spy on teh entirez intarwebz!! -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Petabytes on a budget
nope the 4-port is PCI 66mhz, http://www.addonics.com/products/host_controller/adsa4r5.asp taken from their parts lists so 15 drives will be all running through that 133Mbytes/sec they will see a bottleneck. Me I would get a 3 slot pcie 4x and use two 8 port sata controllers and save the other for a dual port 10gige card I however don't agree with your 60MB/s on gige I have seen up to 85MB/s with a decent TOE and quick disks, take this from multiple hosts (multiple nics) and you will see bottleneck issues with the disks... On Thu, Sep 3, 2009 at 12:40 PM, Jake Anderson ya...@vapourforge.comwrote: On 03/09/09 10:37, Mark Walkom wrote: I was thinking the same, but I reckon because they are just backing up/archiving data it wouldn't be too bad. ie They aren't looking for huge performance, just huge, cheap storage. 2009/9/3 Morgan Storeym...@morganstorey.com I know I am a geek but that is hot. I am wondering if they see any throughput issues with the sata backplanes and pci sata cards. The backplanes are probably fine, each sata cable is good for ~300mbytes/sec most physical disks couldn't hope to hit that. lesse what their maximum xfer rate is. each drive can hit 103mb/sec average (better than I thought) each sata channel will max out at 250mbyte/sec so they are going to be loosing some bandwidth there. their backplanes take 5 disks, so a potential bandwidth of well call it 500mbyte/second so 50% is out the window there actual bandwidth per 5 disks is going to be 250mbyte they have 9 of these channels for a total bandwith available of 2250mbyte/sec (2 gigabytes a second, that'll rip some dvds fast) standard PCI tops out at 133mbyte/sec so thats out ;- it looks like they are using PCI-E SATA cards the mbo they have and the cards they are using indicate they have 3x of something like this http://www.syba.com/index.php?controller=Productaction=InfoId=861 which maxes out at 250mbyte/sec per (1x PCI-E 1x lane) and one 4 port card which if it comes from that mob must be a PCI by the look of things. but I'll assume that its PCI-E and at least 4 lanes. so the total xfer rate is 1750mbyte/sec (or 883 if they are using the PCI card) Vs the total possible xfer rate of 4500 they aren't doing *too* badly given that on a gigabit ethernet connection you are lucky to push 30mbyte/sec (or 60 if you tweak it), I think its not going to be a big issue ;- If they wanted more oomph their best bet would be to put 2x 16 port PCI-E 16x cards into a motherboard that supported it (most decent SLI motherboards will do that) better still one with 4x pci-E 16 slots so that you can put some 10gigE cards in as well something like http://www.aria.co.uk/Products/Components/Motherboards/Socket+AM3+(AMD)/MSI+790FX-GD70+AM3+Motherboard+with+4+x+PCIe+x+16+?productId=36604http://www.aria.co.uk/Products/Components/Motherboards/Socket+AM3+%28AMD%29/MSI+790FX-GD70+AM3+Motherboard+with+4+x+PCIe+x+16+?productId=36604 say (but with intel of course ;-) That should net you (assuming you use dual port 10gig-E nics) an xfer rate out of the box of around 2400mbytes/sec almost fast enough to spy on teh entirez intarwebz!! -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
