Hi, I am trying to implement Spark like generic framework on top of the YARN. This <http://blog.madhukaraphatak.com/custom-mesos-executor-scala/> blog post depicts the abstraction like Task where you can write any generic function and ship it as a serialized object to run on containers.
The following is the steps I am following as of now 1. Users write a generic function and passes it to the framework 2. That task is serialized and written into HDFS 3. Then that file is added to the local resources of Executor which is a Twill Runnable 4. Executor will load the file from hdfs, deserializes the task and runs it. The problem I am facing is if I reuse the same class for loading task dependent files, Twill is treating like same runnable and it only runs the last task. So only option I have is to generate new classes for each task which is not feasible or use anonymous classes which will be created on the fly. I am coding this in Scala. The reason I am trying use task abstraction is it allows me to support multiple scheduler like Mesos and YARN. Hope it makes sense. On Sat, Dec 20, 2014 at 3:18 AM, Terence Yim <[email protected]> wrote: > Looks like there is some issue with the mail server. I am resending it. > > On Thu, Dec 18, 2014 at 12:18 PM, Terence Yim <[email protected]> wrote: > > Hi, > > > > Currently Twill only supports top level or static inner class for > > implementation of TwillRunnable as it gives a simpler view on how to > > reconstruct the instance in the cluster (like what Hadoop MR does). > > > > Do you have a specific use-case in mind that needs anonymous inner > > class? We are willing to help on that. > > > > Terence > > > > On Thu, Dec 18, 2014 at 7:30 AM, madhu phatak <[email protected]> > wrote: > >> Hi, > >> I am new to twill project. I was trying to implement a functionality > using > >> anonymous class which extends AbstractTwillRunnable. But it seems like > >> Twill heavily uses classnames in various contexts. I am getting mal > >> functional class name or null pointer exception. > >> > >> Is there anyway I can create and run anonymous runnables? > >> > >> -- > >> Regards, > >> Madhukara Phatak > >> http://www.madhukaraphatak.com > -- Regards, Madhukara Phatak http://www.madhukaraphatak.com
