Re: Synchronization among Mappers in map-reduce task

2014-08-12 Thread Wangda Tan
Hi Saurabh, >> am not sure making overwrite=false , will solve the problem. As per java doc by making overwrite=false , it will throw an exception if the file already exists. So, for all the remaining mappers it will throw an exception. You can catch the exception and wait. >> Can you please refe

Re: Synchronization among Mappers in map-reduce task

2014-08-12 Thread saurabh jain
Hi Wangda , I am not sure making overwrite=false , will solve the problem. As per java doc by making overwrite=false , it will throw an exception if the file already exists. So, for all the remaining mappers it will throw an exception. Also I am very new to ZK and have very basic knowledge of it

Re: Synchronization among Mappers in map-reduce task

2014-08-12 Thread Wangda Tan
Hi Saurabh, It's an interesting topic, >> So , here is the question , is it possible to make sure that when one of the mapper tasks is writing to a file , other should wait until the first one is finished. ? I read that all the mappers task don't interact with each other A simple way to do this i

Synchronization among Mappers in map-reduce task

2014-08-11 Thread saurabh jain
Hi Folks , I have been writing a map-reduce application where I am having an input file containing records and every field in the record is separated by some delimiter. In addition to this user will also provide a list of columns that he wants to lookup in a master properties file (stored in HDFS