Pig Output

[Aaron Griffith]

Using PigStorage() my pig script output gets put into partial files on the 
hadoop
file system.

When I use the copyToLocal fuction from Hadoop it creates a local directory with
all the partial files.

Is there a way to copy the partial files from hadoop into a single local file?

Thanks

Re: Pig Output

[Bejoy Ks]

Hi Aaron
 Instead of copyFromLocal use getmerge. It would do your job. The
syntax for CLI is
hadoop fs -getmerge source dir in hdfs/pig output dir lfs destn
dir/xyz.txt


Hope it helps!...

Regards
Bejoy.K.S

On Tue, Dec 6, 2011 at 1:57 AM, Aaron Griffith
aaron.c.griff...@gmail.comwrote:

 Using PigStorage() my pig script output gets put into partial files on the
 hadoop
 file system.

 When I use the copyToLocal fuction from Hadoop it creates a local
 directory with
 all the partial files.

 Is there a way to copy the partial files from hadoop into a single local
 file?

 Thanks

Re: Pig Output

[Russell Jurney]

hadoop dfs cat /my/path/*  single_file

Russell Jurney
twitter.com/rjurney
russell.jur...@gmail.com
datasyndrome.com

On Dec 5, 2011, at 12:30 PM, Aaron Griffith aaron.c.griff...@gmail.com wrote:

 Using PigStorage() my pig script output gets put into partial files on the 
 hadoop
 file system.

 When I use the copyToLocal fuction from Hadoop it creates a local directory 
 with
 all the partial files.

 Is there a way to copy the partial files from hadoop into a single local file?

 Thanks