On Feb 10, 2009, at 4:53 PM, Mark Kerzner wrote:
Brian, I have a similar question: why does transfer from a local
filesystem
to SequenceFile takes so long (about 1 second per Meg)?
Hey Mark,
I saw your question about speed the other day ... unfortunately, I
didn't have any specific advice so I stayed quiet :)
In a correctly configured cluster, performance is mostly limited by
available hardware. If it's obvious that performance is well below
hardware limits (such as in your case), it's usually (a) you're not
generating files fast enough or (b) something is configured wrong.
Have you just tried hadoop fs -put .... for some large file hanging
around locally? If that doesn't go more than 5MB/s or so (when your
hardware can obviously do such a rate), then there's probably a
configuration issue.
Brian
Thank you,
Mark
On Tue, Feb 10, 2009 at 4:46 PM, Brian Bockelman
<[email protected]>wrote:
On Feb 10, 2009, at 4:10 PM, Wasim Bari wrote:
Hi,
Could someone help me to find some real Figures (transfer rate)
about
Hadoop File transfer from local filesystem to HDFS, S3 etc and
among
Storage Systems (HDFS to S3 etc)
Thanks,
Wasim
What are you looking for? Maximum possible transfer rate? Maximum
possible transfer rate per client? Generally, if you're using the
Java
client, transfer rate to/from HDFS is limited by the hardware you
have and
the network connection (if you have 1Gbps per client).
I could give you a graph showing a peak of 9Gbps from our Hadoop
instance
to the WAN, but that's not very interesting if you don't have a
10Gbps
pipe...
Brian
