When you run terasort, pass -Dmapred.reduce.tasks=4 and see how that
goes for you. See this old thread for info:
http://mail-archives.apache.org/mod_mbox/hadoop-common-user/200906.mbox/%3ccbbf4b570906300617ma4505f5o2aa1b9fb87b31...@mail.gmail.com%3E
-Todd
On Fri, Feb 25, 2011 at 4:45 PM, Jeffrey
Hi Todd,
Thanks for the quick response.
I added true to dfs.replication, but I still get just one output
copy. Can hadoop apps overwrite the replication level even with this parameter?
I tried increasing mapred.tasktracker.reduce.tasks.maximum from 1 to 4, but
that didn't make any difference:
Hi Jeff,
The output of terasort has replication level 1 by default. This is so
it goes faster with the default settings and makes for more impressive
benchmark results :)
The reason you see it all on one machine is probably that you're
running with one reducer. Try configuring your terasort to use
I'm a newbie to hadoop and HDFS. I'm seeing odd behavior in HDFS that I hope
somebody can clear up for me. I'm running hadoop version 0.20.1+169.127 from
the cloudera distro on 4 identical nodes, each with 4 cpus and 100GB disk
space. Replication is set to 2.
I run:
hadoop jar /usr/lib/hado
Maybe grep for
2011-02-25 18:47:05,564 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Decommission complete for
node 102.1.1.1:50010
In the namenode log to see if decommission is completed?
I remember a similar problem was reported just a few days ago ( in attachment)
by James Litt
For an example how it is done, look at FsShell#copyToLocal() and its
internal implementation. It uses FileUtil#copy() method to do this copying.
On Fri, Feb 25, 2011 at 5:08 AM, Alessandro Binhara wrote:
> How to copy a file from a HDS to local file system with a JAVA API ?
>
> where i can find
Use this API
http://hadoop.apache.org/common/docs/current/api/index.html?org/apache/hadoop/fs/FileSystem.html
The code is pretty straightforward.
-Ayon
From: Alessandro Binhara
To: hdfs-user@hadoop.apache.org
Sent: Fri, February 25, 2011 5:08:57 AM
Subject:
Hello,
On Fri, Feb 25, 2011 at 6:38 PM, Alessandro Binhara wrote:
> How to copy a file from a HDS to local file system with a JAVA API ?
> where i can find a documentation and example about it?
Have a look at the FileSystem API
(http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop
How to copy a file from a HDS to local file system with a JAVA API ?
where i can find a documentation and example about it?
thanks
