Andreas, to me, Twill is a library, a convenience library, that one can use to write Yarn apps. Hoya aims to provide a general framework using which one can take existing apps (HBase/Accumulo to start with), and make them run well in a Yarn cluster, without intruding at all into the App internals. The goal is to have minimal, ideally zero changes, in the App itself. The only glue between the App and Hoya is a Hoya interface that the App needs to implement for it to be deployable/manageable by Hoya.
Although, one could argue that Hoya can be written using Twill libraries (which is something we should pursue as part of long/medium-term collaboration between the two projects), but I'd argue that the goals of the two projects are different - Twill would continue to make Yarn app developers' lives easier, while Hoya is a tool that could deploy distributed-frameworks easily in a Yarn cluster, and be able to later do basic management (as is talked about in the github docs). Things like dynamic configuration patching for the applications like HBase to easily run in a Yarn cluster, security issues, failure handling and defining a model for reacting to failures, being able to store some state about applications to facilitate better application restart behavior in a Yarn cluster, etc. would be in the purview of Hoya. Management frameworks could use Hoya as a tool as well to talk to Yarn and do application start/stop/shrink/expand an instance of an application (e.g. HBase) cluster. On Thu, Jan 16, 2014 at 4:38 PM, Andreas Neumann <a...@apache.org> wrote: > I do see a lot of value in all the features of Hoya, and I am not trying to > discredit it. Yet I do think that most of these features are actually > already in Twill or would be great additions to Twill, and sooner or later > will be implemented in Twill, implying even greater overlap. > > I am not sure whether I follow the distinction between porting an existing > app and developing a new app, if both require similar abstractions... And > after all, porting an existing application is one way to start developing a > Yarn application. > > Anyway, I just wanted to point out the overlap and offer collaboration. If > the community thinks that the two projects are different beasts, that's > fine with me. > > > On Wed, Jan 15, 2014 at 10:40 AM, Josh Elser <josh.el...@gmail.com> wrote: > >> I wanted to weigh in on some of Steve's thoughts, I'm actually really >> excited about being able to leverage Hoya inside of Accumulo itself. >> >> We presently have a few system tests that rely on manual set up, which can >> be frustrating to deal with on a beefy boxes (running multiple Accumulo >> procs on a single host). Hoya drastically reduces the amount of effort it >> takes to run these distributed tests in a 'closer to real' environment. >> On Jan 15, 2014 8:36 AM, "Steve Loughran" <ste...@hortonworks.com> wrote: >> >> > On 15 January 2014 02:13, Andreas Neumann <a...@apache.org> wrote: >> > >> > > I see. So is Hoya limited to HBase and Accumulo? Or is it open for any >> > > other type of existing application? If so, won't it have some common >> > > abstraction that is shared by all of them? That is where I see the >> > > similarity with Twill. >> > > >> > > >> > it started off as pure hbase, but now has the notion of a provider which >> > has a client-side and server side >> > >> > client side >> > -helps set up the template JSON file to describe the cluster (e.g. adds >> > default values), >> > -patches the configuration directory supplied at creation time with >> > whatever options it wants (e.g links up fs.default.name & ZK settings in >> > hbase-site.xml) >> > -does preflight validation of cluster options >> > -can also add its own .tar.gz to the resources of the AM (or any other >> > resources) >> > >> > server side >> > -runs in the AM and sets up everything needed to run instances of a role >> > (e.g. HBase master, Accumulo GC, ..) >> > -can run code in the AM to help set things up (e.g. Accumulo provider >> > service runs bin/accumulo init if needed) >> > -TODO: liveness monitoring >> > >> > the requirements of an app to be deployable are >> > >> > >> https://github.com/hortonworks/hoya/blob/master/src/site/markdown/app_needs.md >> > >> > >> > >> > > Whereas, if it is a separate effort for each existing application, say >> > > HBase, then what is the end goal for Hoya when it comes out of >> > incubation? >> > > To merge it back into HBase proper? >> > > >> > > >> > Now that it supports >1 application, it can't go into HBase. The >> individual >> > provider services can (their implementation classes are worked out via >> the >> > configuration.XML). >> > >> > But as we use the accumulo and HBase apps for testing, its really good to >> > have them both in the hoya project right now -a project that builds >> > downstream of both and needs >> > to be given the Hadoop filesystem paths to .tar.gz files of each app. >> > >> > There's nothing to stop 3rd party apps joining in, indeed, one thing I'd >> > like someone to do is actually have Hoya deploy SmartFrog .tar.gz files >> and >> > pass down configurations that deploy applications -for example >> jetty-based >> > web servers. That'd take an intern working at HP Labs over the summer, >> > maybe >> > >> > -steve >> > >> > -- >> > CONFIDENTIALITY NOTICE >> > NOTICE: This message is intended for the use of the individual or entity >> to >> > which it is addressed and may contain information that is confidential, >> > privileged and exempt from disclosure under applicable law. If the reader >> > of this message is not the intended recipient, you are hereby notified >> that >> > any printing, copying, dissemination, distribution, disclosure or >> > forwarding of this communication is strictly prohibited. If you have >> > received this communication in error, please contact the sender >> immediately >> > and delete it from your system. Thank You. >> > >> -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. --------------------------------------------------------------------- To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
