Thanks Jay for keeping us up to date!
We are heavily using kubernetes at [current job] and it's definitely
solving a lot of problems.
I would love to see some deeper integration between Apache Bigtop and
Kubernetes.
Thanks,
Bruno
On 03/08/2017 12:42 PM, jay vyas wrote:
Hi bigtop:
Since i work on kubernetes now adays, every once in a while I like to drop
an email to let you guys know what I've been up to, and how bigtop might
play a role in the new containerized world we live in.
These are just some ideas to cross fertilize the community, not official
bigtop proposals, although I'm happy to discuss this further if anyone is
interested in running bigtop components in a containerized data cetner
environment.
I think if anyone ever wanted to run bigtop in a containerized way, it
might be interesting to select components we care about, and create a
version of bigtop that deployed as "cloud native". Rather then
pontificating on merits, ill just discuss 2 ways this might look like if it
ever was done:
1) Deployment as an idiomatic containerized service.
In order to do this, we basically could do something like:
- Publish yaml files as part of a bigtop release.
- Define stateful sets for the things we care about, like spark masters, or
ignite, or your hadoop nodes.
- Deploy and scale up/down as needed.
In kube, this is done via PetSets for performance or statefulness, but WLOG
the above instructions could possibly work in other places, i.e. like
docker swarm or something.
2) Deployment by simulating "nodes" (anti-pattern but might be a good
gateway to accomodate migration.
Another way to do this same thing might be to simply deploy a set of JVM
containers in a cluster, which could serve as a "bigtop base image" (we do
this now with the bigtop tool chain, sorta), in fact, we already publish
containers for this iirc. This would break the containerized idiom that
is most popular (microservices), but might be an interesting staging ground
for people to deploy static hadoop clusters in a containerized microsystem
environment.
