> Ok I can understand your point - but I am sure that some people have been > trying to use map-reduce programming model to do CFD, or any other > scientific computing. > Any experience in this area from the list ?
I know of one project that assumes it has an entire Hadoop cluster, and generates the hostnames in the Mapper and uses those host lists in the Reducer to launch an MPI job. They do it because it provides a higher efficiency for doing very small data transfers. The alternative was doing a long chain of map/reduce jobs that have very small outputs from each phase. I wouldn't recommend using MPI under map/reduce in general, since it involves making a lot of assumptions about your application. In particular, to avoid from killing your cluster your shouldn't use checkpoints in your application and just rerun the application from the beginning on failures. That implies that the application can't run very long (upper bound of probably 30 minutes on 2000 nodes). That said, if you want to run other styles of applications, you really want a two level scheduler. Where the first level scheduler allocates nodes (or partial nodes) to jobs (or frameworks). Effectively, that is what Hadoop On Demand (HOD) was doing with Torque, but I suspect there will be a more performant solution than HOD with in the next year. -- Owen
