Hi Martin, You are right that the twill API was designed to be as neutral as possible and not to be specific for YARN. In fact, there are some interests to create a Mesos runner in twill. Would you mind sharing a bit more about the cluster management infrastructure your company is using ? Is it completely in house or is it based on some open source project?
Terence Sent from my iPhone > On Feb 6, 2017, at 12:44 PM, Martin Serrano <mar...@attivio.com> wrote: > > Devs, > > I've got an initial prototype going on this project. During this process I > noticed that much of what I needed to do ended up duplicating code that is in > the twill-yarn project. I'm curious if the community would be open to > refactoring that moved some of this code such that it could be shared by > different clustering control implementations. For instance, much of the code > in YarnTwillPreparer, YarnContainerMain, YarnServiceMain, etc is not really > yarn specific. > > Thoughts? > > Thanks, > Martin > >> On 02/01/2017 04:13 PM, Martin Serrano wrote: >> Hey Devs, >> >> I've been evangelizing Twill at here at my company. We have a for several >> years had a basic clustering solution for running our system and its >> components across multiple hosts. As we've worked to migrate the platform >> to Hadoop and YARN, we have started to move our cluster control to be >> Yarn-based. And my goal is to make it Twill-based. However as we do this >> work we anticipate the need to support customers that do not have and do not >> want to operate Hadoop infrastructure. >> >> I'm working on an architecture which would use Twill for command and control >> for all of our services. For customers that did not want the full >> enterprise capabilities that come with Hadoop we could continue to offer our >> basic clustering support (albeit with reduced capabilities) by plugging our >> clustering solution into Twill. One of the aspects of Twill that interested >> me from the start was that the control API was abstracted from YARN. >> >> A couple questions: >> >> 1) Do any plugins for other clustering backends already exist? Open source? >> Commercial? >> >> 2) If we were to do this, would the Twill community be interested in a >> donation of this code to the Twill project? >> >> I recall that Henry and I had a conversation at the last Apache Big Data >> that there had been talk of plugins for other clustering backends but I >> don't know if anything ever came of that. >> >> Cheers, >> Martin Serrano >> >
