Re: Concatenating files on HDFS

[Brian Bockelman]

Does

http://hadoop.apache.org/common/docs/current/hdfs_shell.html#getmerge

Help you any?





Brian

On Aug 27, 2009, at 3:50 AM, Andreas Kostyrka wrote:

Note that ">>" has nothing to do with file systems - it's a metacharacter in your shell. Do note that "append" isn't supported in HDFS in current releases; there's quite an effort going on to get it in for 0.21.0. I'd guess that will come out in the October-November timeframe, but someone can probably correct me on that.

Actually, for many use cases it's enough to keep a directory where all
parts of a given "logical" file are kept:

-) for input in hadoop jobs you just specify the directory.
-) if you need it in one piece externally, you can cat the whole
directory into one file.

Hence in my experience one does not need to concat files inside HDFS
usually.

Andreas


Am Donnerstag, den 27.08.2009, 11:04 +0530 schrieb Ankur Goel:

HDFS files are write once so you cannot append to them (at the moment). What you can do is copy your local file to HDFS dir containing the same file you want to append to. Once that is done you can run a simple (Identity Mapper & Identity Reducer) mapreduce job with input

as this directory and number of reducers = 1.

----- Original Message -----
From: "Turner Kunkel" <thkun...@gmail.com>
To: core-u...@hadoop.apache.org

Subject: Concatenating files on HDFS

Is there any way to concatenate/append a local file to a file on HDFS
without copying down the HDFS file locally first?

I tried:
bin/hadoop dfs -cat file:///[local file] >> hdfs://[hdfs file]
But it just tries to look for hdfs://[hdfs file] as a local file,
since I suppose the dfs -cat command doesn't support the >> operator.

Thanks.

Sent: Wednesday, August 26, 2009 10:02:41 PM GMT +05:30 Chennai, Kolkata, Mumbai, New Delhi

smime.p7s
Description: S/MIME cryptographic signature