Please help me on this. I need it very urgently Regards, Jagaran
----- Forwarded Message ---- From: jagaran das <jagaran_...@yahoo.co.in> To: common-user@hadoop.apache.org Sent: Thu, 16 June, 2011 9:51:51 PM Subject: Re: HDFS File Appending URGENT Thanks a lot Xiabo. I have tried with the below code in HDFS version 0.20.20 and it worked. Is it not stable yet? public class HadoopFileWriter { public static void main (String [] args) throws Exception{ try{ URI uri = new URI("hdfs://localhost:9000/Users/jagarandas/Work-Assignment/Analytics/analytics-poc/hadoop-0.20.203.0/data/test.dat"); Path pt=new Path(uri); FileSystem fs = FileSystem.get(new Configuration()); BufferedWriter br; if(fs.isFile(pt)){ br=new BufferedWriter(new OutputStreamWriter(fs.append(pt))); br.newLine(); }else{ br=new BufferedWriter(new OutputStreamWriter(fs.create(pt,true))); } String line = args[0]; System.out.println(line); br.write(line); br.close(); }catch(Exception e){ e.printStackTrace(); System.out.println("File not found"); } } } Thanks a lot for your help. Regards, Jagaran ________________________________ From: Xiaobo Gu <guxiaobo1...@gmail.com> To: common-user@hadoop.apache.org Sent: Thu, 16 June, 2011 8:01:14 PM Subject: Re: HDFS File Appending URGENT You can merge multiple files into a new one, there is no means to append to a existing file. On Fri, Jun 17, 2011 at 10:29 AM, jagaran das <jagaran_...@yahoo.co.in> wrote: > Is the hadoop version Hadoop 0.20.203.0 API > > That means still the hadoop files in HDFS version 0.20.20 are immutable? > And there is no means we can append to an existing file in HDFS? > > We need to do this urgently as we have do set up the pipeline accordingly in > production? > > Regards, > Jagaran > > > > ________________________________ > From: Xiaobo Gu <guxiaobo1...@gmail.com> > To: common-user@hadoop.apache.org > Sent: Thu, 16 June, 2011 6:26:45 PM > Subject: Re: HDFS File Appending > > please refer to FileUtil.CopyMerge > > On Fri, Jun 17, 2011 at 8:33 AM, jagaran das <jagaran_...@yahoo.co.in> wrote: >> Hi, >> >> We have a requirement where >> >> There would be huge number of small files to be pushed to hdfs and then use >>pig >> to do analysis. >> To get around the classic "Small File Issue" we merge the files and push a >> bigger file in to HDFS. >> But we are loosing time in this merging process of our pipeline. >> >> But If we can directly append to an existing file in HDFS we can save this >> "Merging Files" time. >> >> Can you please suggest if there a newer stable version of Hadoop where can go >> for appending ? >> >> Thanks and Regards, >> Jagaran >