On Feb 10, 2009, at 11:09 PM, Mark Kerzner wrote:
Brian, large files using command-line hadoop go fast, so it is
something
about my computer or network. I won't worry about this now,
especially in
light of Amit reporting fast writes and reads.
You're creating files using SequenceFile, right? It might be that the
creation of the sequence file is the portion which is slow, not the
network I/O.
I don't have much knowledge about optimization of SequenceFile
creation. I assume that you'll want to start by tweaking compression
on and off. Additionally, Jeff (I think) pointed to a Hadoop Archive
file, which also might be an alternative for your system. I don't
know enough to give you a set of pros and cons, just enough to mention
it as an alternative to experiment with.
Sorry I'm not useful here...
Brian
Mark
On Tue, Feb 10, 2009 at 5:00 PM, Brian Bockelman
<[email protected]>wrote:
On Feb 10, 2009, at 4:53 PM, Mark Kerzner wrote:
Brian, I have a similar question: why does transfer from a local
filesystem
to SequenceFile takes so long (about 1 second per Meg)?
Hey Mark,
I saw your question about speed the other day ... unfortunately, I
didn't
have any specific advice so I stayed quiet :)
In a correctly configured cluster, performance is mostly limited by
available hardware. If it's obvious that performance is well below
hardware
limits (such as in your case), it's usually (a) you're not
generating files
fast enough or (b) something is configured wrong.
Have you just tried hadoop fs -put .... for some large file hanging
around
locally? If that doesn't go more than 5MB/s or so (when your
hardware can
obviously do such a rate), then there's probably a configuration
issue.
Brian
Thank you,
Mark
On Tue, Feb 10, 2009 at 4:46 PM, Brian Bockelman <[email protected]
wrote:
On Feb 10, 2009, at 4:10 PM, Wasim Bari wrote:
Hi,
Could someone help me to find some real Figures (transfer rate)
about
Hadoop File transfer from local filesystem to HDFS, S3 etc and
among
Storage Systems (HDFS to S3 etc)
Thanks,
Wasim
What are you looking for? Maximum possible transfer rate? Maximum
possible transfer rate per client? Generally, if you're using
the Java
client, transfer rate to/from HDFS is limited by the hardware you
have
and
the network connection (if you have 1Gbps per client).
I could give you a graph showing a peak of 9Gbps from our Hadoop
instance
to the WAN, but that's not very interesting if you don't have a
10Gbps
pipe...
Brian
