Hi Manoj >From my limited knowledge on file appends in hdfs , i have seen more recommendations to use sync() in the latest releases than using append(). Let us wait for some commiter to authoritatively comment on 'the production readiness of append()' . :)
Regards Bejoy KS On Mon, Sep 10, 2012 at 11:03 AM, Manoj Babu <manoj...@gmail.com> wrote: > Thank you Bejoy. > > Does file append is production stable? > > > Cheers! > Manoj. > > > > On Sun, Sep 9, 2012 at 10:19 PM, Bejoy KS <bejoy.had...@gmail.com> wrote: > >> ** >> Hi Manoj >> >> You can load daily logs into a individual directories in hdfs and process >> them daily. Keep those results in hdfs or hbase or dbs etc. Every day do >> the processing, get the results and aggregate the same with the previously >> aggregated results till date. >> >> Regards >> Bejoy KS >> >> Sent from handheld, please excuse typos. >> ------------------------------ >> *From: * Manoj Babu <manoj...@gmail.com> >> *Date: *Sun, 9 Sep 2012 21:28:54 +0530 >> *To: *<mapreduce-user@hadoop.apache.org> >> *ReplyTo: * mapreduce-user@hadoop.apache.org >> *Subject: *Reg: parsing all files & file append >> >> Hi All, >> >> I have two questions, providing info on it will be helpful. >> >> 1, I am using hadoop to analyze and to find top n search term metric's >> from logs. >> If any new log file is added to HDFS then again we are running the job to >> find the metrics. >> Daily we will be getting log files and we are parsing the whole file and >> getting the metric's. >> All the log file's are parsed daily to get the latest metric's is there >> any way is there any way to avoid this? >> >> 2, Does file append is production stable? >> >> Cheers! >> Manoj. >> >> >
