Brian, I have a similar question: why does transfer from a local filesystem to SequenceFile takes so long (about 1 second per Meg)? Thank you, Mark
On Tue, Feb 10, 2009 at 4:46 PM, Brian Bockelman <[email protected]>wrote: > > On Feb 10, 2009, at 4:10 PM, Wasim Bari wrote: > > Hi, >> Could someone help me to find some real Figures (transfer rate) about >> Hadoop File transfer from local filesystem to HDFS, S3 etc and among >> Storage Systems (HDFS to S3 etc) >> >> Thanks, >> >> Wasim >> > > What are you looking for? Maximum possible transfer rate? Maximum > possible transfer rate per client? Generally, if you're using the Java > client, transfer rate to/from HDFS is limited by the hardware you have and > the network connection (if you have 1Gbps per client). > > I could give you a graph showing a peak of 9Gbps from our Hadoop instance > to the WAN, but that's not very interesting if you don't have a 10Gbps > pipe... > > Brian > >
