Hi bigtop: Since i work on kubernetes now adays, every once in a while I like to drop an email to let you guys know what I've been up to, and how bigtop might play a role in the new containerized world we live in.
These are just some ideas to cross fertilize the community, not official bigtop proposals, although I'm happy to discuss this further if anyone is interested in running bigtop components in a containerized data cetner environment. I think if anyone ever wanted to run bigtop in a containerized way, it might be interesting to select components we care about, and create a version of bigtop that deployed as "cloud native". Rather then pontificating on merits, ill just discuss 2 ways this might look like if it ever was done: 1) Deployment as an idiomatic containerized service. In order to do this, we basically could do something like: - Publish yaml files as part of a bigtop release. - Define stateful sets for the things we care about, like spark masters, or ignite, or your hadoop nodes. - Deploy and scale up/down as needed. In kube, this is done via PetSets for performance or statefulness, but WLOG the above instructions could possibly work in other places, i.e. like docker swarm or something. 2) Deployment by simulating "nodes" (anti-pattern but might be a good gateway to accomodate migration. Another way to do this same thing might be to simply deploy a set of JVM containers in a cluster, which could serve as a "bigtop base image" (we do this now with the bigtop tool chain, sorta), in fact, we already publish containers for this iirc. This would break the containerized idiom that is most popular (microservices), but might be an interesting staging ground for people to deploy static hadoop clusters in a containerized microsystem environment. -- jay vyas
