This comment (from http://stackoverflow.com/questions/12504690/how-to-run-hadoop-multithread-way-in-single-jvm ) may also be relevant:
"Hadoop purposely does not run more than one task at the same time in one JVM for isolation purposes. And in stand-alone (local) mode, only one JVM is ever used. If you want to make use of your four cores, you should run in pseudo-distributed mode, and increase the max number of concurrent tasks to four. You can do this with the mapred.tasktracker.map.tasks.maximum and mapred.tasktracker.reduce.tasks.maximum properties." On Thu, Aug 29, 2013 at 8:49 PM, Harsh J <ha...@cloudera.com> wrote: > Map tasks run in parallel spawned JVMs, so are isolated from one > another at runtime. Use of static functions shouldn't affect you > generally. > > Default Map I/O is single-threaded. If you plan to use > multiple-threads, use MultiThreadedMapper for proper thread-safety. > > On Thu, Aug 29, 2013 at 8:15 PM, Pavan Sudheendra <pavan0...@gmail.com> > wrote: > > No, I had written a huge Map Reduce program which talks with hbase and > does > > a lot of computing using it as a source as well as sink.. One of my > > colleague saw my code and saw that I had used a lot of static function > > instead of making use of proper OOP concepts.. He was telling me that it > > shouldn't be the way I should go about doing it.. But my code works > fine.. > > So, was wondering will I face any problem in the future because of this.. > > That's all. > > > > Regards, > > Pavan > > > > On Aug 29, 2013 8:11 PM, "Adam Muise" <amu...@hortonworks.com> wrote: > >> > >> Mappers don't communicate with each other in traditional MapReduce. If > you > >> need something more MPI-ish then look to MPI over YARN or write your own > >> YARN app. > >> > >> If you need multi-threading within the mapper then it is up to you as > the > >> java developer to make it thread safe. Use the concurrent libraries like > >> anything else and Bob's your uncle. Having overly-complicated mappers > can be > >> difficult to manage however and it kind of misses the mark for MapReduce > >> problems. > >> > >> Maybe if you expand on your use case a bit someone here can provide > >> specific advice. > >> > >> > >> On Thu, Aug 29, 2013 at 10:33 AM, Pavan Sudheendra <pavan0...@gmail.com > > > >> wrote: > >>> > >>> Hi all, > >>> > >>> Is hadoop thread safe? Do mappers make use of threads in any chance? A > >>> little bit of information on the way they execute in parallel would > help me > >>> out. Thanks. > >>> > >>> Regards, > >>> Pavan > >> > >> > >> > >> > >> -- > >> > >> > >> Adam Muise > >> Solution Engineer > >> Hortonworks > >> amu...@hortonworks.com > >> 416-417-4037 > >> > >> Hortonworks - Develops, Distributes and Supports Enterprise Apache > Hadoop. > >> > >> Hortonworks Virtual Sandbox > >> > >> Hadoop: Disruptive Possibilities by Jeff Needham > >> > >> CONFIDENTIALITY NOTICE > >> NOTICE: This message is intended for the use of the individual or entity > >> to which it is addressed and may contain information that is > confidential, > >> privileged and exempt from disclosure under applicable law. If the > reader of > >> this message is not the intended recipient, you are hereby notified > that any > >> printing, copying, dissemination, distribution, disclosure or > forwarding of > >> this communication is strictly prohibited. If you have received this > >> communication in error, please contact the sender immediately and > delete it > >> from your system. Thank You. > > > > -- > Harsh J >
