On Wed, Mar 8, 2017 at 12:42 PM, jay vyas <[email protected]> wrote: > Hi bigtop: > > Since i work on kubernetes now adays, every once in a while I like to drop > an email to let you guys know what I've been up to, and how bigtop might > play a role in the new containerized world we live in. > > These are just some ideas to cross fertilize the community, not official > bigtop proposals, although I'm happy to discuss this further if anyone is > interested in running bigtop components in a containerized data cetner > environment.
I'm very curious where this could lead us. Interestingly enough, I've offered Google and CNCF similar ideas for collaboration at ODPi level, but those guys didn't really bite. > I think if anyone ever wanted to run bigtop in a containerized way, it > might be interesting to select components we care about, and create a > version of bigtop that deployed as "cloud native". Rather then > pontificating on merits, ill just discuss 2 ways this might look like if it > ever was done: > > 1) Deployment as an idiomatic containerized service. > > In order to do this, we basically could do something like: > > - Publish yaml files as part of a bigtop release. > - Define stateful sets for the things we care about, like spark masters, or > ignite, or your hadoop nodes. > - Deploy and scale up/down as needed. > > In kube, this is done via PetSets for performance or statefulness, but WLOG > the above instructions could possibly work in other places, i.e. like > docker swarm or something. Interesting! I've followed Kubernetes development somewhat, but at a pretty high level. What's a good intro (presentation, etc.) into those latest stateful capabilities and what's a good way to play with those hands-on? > 2) Deployment by simulating "nodes" (anti-pattern but might be a good > gateway to accomodate migration. > > Another way to do this same thing might be to simply deploy a set of JVM > containers in a cluster, which could serve as a "bigtop base image" (we do > this now with the bigtop tool chain, sorta), in fact, we already publish > containers for this iirc. This would break the containerized idiom that > is most popular (microservices), but might be an interesting staging ground > for people to deploy static hadoop clusters in a containerized microsystem > environment. I guess my biggest question is this: how do you see Kubernetes interact with things like Puppet, etc? Note that even our Juju integration is done via Puppet code today. Will this be the same for Kubernetes? Thanks, Roman.
