Hi Larry, security is a very interesting and challenging topic... and right now, Twill does not do anything particular for security, other than propagating the user's Hadoop tokens to the containers.
I recently had a discussion with one of my coworkers about authentication of containers: Suppose I want to run a distributed framework in YARN, something like Storm or an MPI cluster, which will bring up a bunch of containers. Twill allows each container to make itself discoverable, but right now, clients of the discovery service get little guarantees about the authenticity of the service. How can we make sure that containers authenticate each other and with the outside in ways similar to the Hadoop daemons (name node and data node, etc.), Data protection also makes a lot of sense, for example, in a Storm topology, you may want to encrypt (or at least sign) the data packets that flow between the containers. And rather than building that into every application, having this capability built into the platform would add a lot of convenience. So yes, I think you are on a very good track. -Andreas. On Mon, Mar 10, 2014 at 11:43 AM, larry mccay <[email protected]> wrote: > Hi Andreas - > > I am investigating whether it would make sense to add certain security > based features into an application framework for Yarn. > > Without an understanding of how one builds typical Yarn applications, it's > hard to discern where and whether it would be appropriate to add hooks for > things like data protection through encryption, etc. > > Perhaps, it would even make sense to write a data protection service based > on Twill that other applications can interact with in order to have their > data decrypted on demand? > > Just trying to get a feel for what things should go into a framework or > library and what things should go into the application itself. > > Does that make sense? > > thanks, > > --larry > > > > > On Mon, Mar 10, 2014 at 2:33 PM, Andreas Neumann <[email protected]> wrote: > > > Hi Larry, > > > > unfortunately, we don't have a lot of examples yet. > > > > My company's product (Continuuity Reactor) makes extensive use of Twill > > (but that is not open-source...). For example, to run a real-time stream > > processing engine similar to Storm, and also to run elastically scalable > > web services. > > > > Can you be a little more specific about what you would like to find out, > > and what kind of service you are planning to develop? > > Will be happy to help you get started or answer any questions. > > > > -Andreas. > > > > > > On Sun, Mar 9, 2014 at 9:44 AM, larry mccay <[email protected]> > wrote: > > > > > Hello apache-twillers - > > > > > > I am looking for some example uses of Twill - preferably a bit more > > > advanced than the EchoServer example. > > > > > > My immediate interest is for extending the framework to provide a new > > core > > > service to be available to all Twill based applications. So, I'd like > to > > > see a more comprehensive example. > > > > > > thanks, > > > > > > --larry > > > > > >
