Under YARN (branch-2, branch-0.23, and trunk) the speculative execution decision is pluggable, and can be replaced by a user. If you could come up with a better solution to speculative execution that would be great. We have known for a while that it is not very good (most of the time we run a speculative task it is just wasted). In branch-2 we have a new version that we think is better (more conservative), but I am not sure how much of a study has been done on exactly how much better it is or what else can be done to make it even better. I would look at adding your ideas into that plugin and not so much of using the config to turn speculation on or off dynamically, because there are some map/reduce applications that abuse map/reduce somewhat and will not run correctly if speculative execution is enabled.
--Bobby Evans On 9/13/12 1:08 AM, "Suresh S" <suresh...@gmail.com> wrote: >Hello Sir/Madam.... > > I am doing PhD. I am interested to do research in Hadoop for publishing >paper. >I know little bit about speculative execution of slow tasks. I know it is >possible to enable or disable speculative execution. > >But, > Is there any idea published already for dynamically enable or disable >the speculative execution depending on application, cluster load and other >run time parameters? > > Is it worth to do research in this direction? > > Is this contribution is worth to publish a conference or journal >paper(s)? > >*Regards* >*S.Suresh,* >*Research Scholar,* >*Department of Computer Applications,* >*National Institute of Technology,* >*Tiruchirappalli - 620015.* >*India* >*Mobile: +91-9941506562*
