static, pinned items persist across jvm reuse. On Sat, Apr 25, 2009 at 6:44 AM, amit handa <amha...@gmail.com> wrote:
> Thanks Aaron, > > The processing libs that we use, which take time to load are all c++ based > .so libs. > Can i invoke it from JVM during the configure stage of the mapper and keep > it running as you suggested ? > Can you point me to some documentation regarding the same ? > > Regards, > Amit > > On Sat, Apr 25, 2009 at 1:42 PM, Aaron Kimball <aa...@cloudera.com> wrote: > > > Amit, > > > > This can be made to work with Hadoop. Basically, in your mapper's > > "configure" stage it would do the heavy load-in process, then it would > > process your individual work items as records during the actual "map" > > stage. > > A map task can be comprised of many records, so you'll be fine here. > > > > If you use Hadoop 0.19 or 0.20, you can also enable JVM reuse, where > > multiple map tasks are performed serially in the same JVM instance. In > this > > case, the first task in the JVM would do the heavy load-in process into > > static fields or other globally-accessible items; subsequent tasks could > > recognize that the system state is already initialized and would not need > > to > > repeat it. > > > > The number of mapper/reducer tasks that run in parallel on a given node > can > > be configured with a simple setting; setting this to 6 will work just > fine. > > The capacity / fairshare schedulers are not what you need here -- their > > main > > function is to ensure that multiple jobs (separate sets of tasks) can all > > make progress simultaneously by sharing cluster resources across jobs > > rather > > than running jobs in a FIFO fashion. > > > > - Aaron > > > > On Sat, Apr 25, 2009 at 2:36 PM, amit handa <amha...@gmail.com> wrote: > > > > > Hi, > > > > > > We are planning to use hadoop for some very expensive and long running > > > processing tasks. > > > The computing nodes that we plan to use are very heavy in terms of CPU > > and > > > memory requirement e.g one process instance takes almost 100% CPU (1 > > core) > > > and around 300 -400 MB of RAM. > > > The first time the process loads it can take around 1-1:30 minutes but > > > after > > > that we can provide the data to process and it takes few seconds to > > > process. > > > Can I model it on hadoop ? > > > Can I have my processes pre-loaded on the task processing machines and > > the > > > data be provided by hadoop? This will save the 1-1:30 minutes of intial > > > load > > > time that it would otherwise take for each task. > > > I want to run a number of these processes in parallel based on the > > > machines > > > capacity (e.g 6 instances on a 8 cpu box) or using capacity scheduler. > > > > > > Please let me know if this is possible or any pointers to how it can be > > > done > > > ? > > > > > > Thanks, > > > Amit > > > > > > -- Alpha Chapters of my book on Hadoop are available http://www.apress.com/book/view/9781430219422
