Re: [DISCUSS] Kubernetes Orchestration for ZK, HDFS, and HBase

[Mallikarjun]

Hi Nick,

I agree with your thought that there is an increasing reliance on
kubernetes, more so for complex workloads like hbase deployments because of
unavailability of reliable automation frameworks outside of k8s.

But I have a slightly different view in terms of how to achieve it. When I
was exploring what are the possibilities such as kustomize or helm or
operator. I found it can get pretty complex in terms of writing extensible
deployment manifest (for different kinds of deployments) with tools like
kustomize or helm. Here is our attempt to conterairise hbase with operator
--> https://github.com/flipkart-incubator/hbase-k8s-operator

---
Mallikarjun

On Mon, Mar 13, 2023 at 3:58 PM Nick Dimiduk  wrote:

> Heya team,
>
> Over here at $dayjob, we have an increasing reliance on Kubernetes for
> both development and production workloads. Our tools are maturing and
> we're hoping that they might be of interest to the wider community.
> I'd like to see if there's community interest in receiving some/any of
> them as a contribution. I think we'll also need a plan from ASF Infra
> that makes kubernetes available to us as a project.


> We have implemented a basic stack of tools for orchestrating ZK + HDFS
> + HBase on Kubernetes. We use this for running a small local dev
> cluster via MiniKube/KIND ; for ITBLL on smallish distributed clusters
> in a public cloud ; and in production for running clusters of ~100
> Data Nodes/Region Servers in a public cloud. There was an earlier
> discussion about using our donation of test hardware for running more
> thorough tests in our CI, but one of the limiting factors is full
> cluster deployment. I hope that the community might be interested in
> receiving this tooling as a foundation for more rigorous correctness
> and maybe even performance tests in the open. Furthermore, perhaps the
> wider community has interest in an Apache licensed cluster
> orchestration tool for other uses.
>
> Now for some details: The implementation is built on Kustomize, so
> it's fundamentally transparent resource specification with yaml
> patches for composability; this is in contrast to a solution using
> templates with defined capabilities and interfaces. There is no
> operator ; it's all coordinated via init/bootstrap containers, shell
> scripts, shared volumes for state, &c. For now.


> Such a donation will amount to a code drop, which will have its
> challenges. I'm motivated via internal processes to carve it into
> smaller pieces, and I think that will benefit community review as
> well. Perhaps this approach could be used to make the contribution via
> a feature branch.
>
> Is there community interest in adding such a capability to our
> maintained responsibilities? I'd hope that we have several volunteers
> to work with me through the contribution process, and who are
> reasonably confident that they'll be able to help maintain such a
> capability going forward. We'll also need someone who can work with
> Infra to get us access to Kubernetes cluster(s), via whatever means.
>
> What do you think?
>
> Thanks,
> Nick & the HBase team at Apple

Re: [DISCUSS] Kubernetes Orchestration for ZK, HDFS, and HBase

[Lars Francke]

Hi Nick,

I do not mean to derail your mail so I'll keep mine short: Yes, I
think testing & infrastructure on Kubernetes would be worthwhile and I
thank you for the offer.
We're happy to take a look and would try to review any incoming
contributions depending on how large/digestable they are :)

We[1] are developing our own operator[2] and during that mission have
learned how prone Kubernetes is to bit rot so it'd be great if your
team would continue helping out.
I'm happy to go into details on why we did what we did but that'd be a
separate thread.

Cheers,
Lars

[1] 
[2] 

On Tue, Mar 14, 2023 at 3:23 PM Mallikarjun  wrote:
>
> Hi Nick,
>
> I agree with your thought that there is an increasing reliance on
> kubernetes, more so for complex workloads like hbase deployments because of
> unavailability of reliable automation frameworks outside of k8s.
>
> But I have a slightly different view in terms of how to achieve it. When I
> was exploring what are the possibilities such as kustomize or helm or
> operator. I found it can get pretty complex in terms of writing extensible
> deployment manifest (for different kinds of deployments) with tools like
> kustomize or helm. Here is our attempt to conterairise hbase with operator
> --> https://github.com/flipkart-incubator/hbase-k8s-operator
>
> ---
> Mallikarjun
>
> On Mon, Mar 13, 2023 at 3:58 PM Nick Dimiduk  wrote:
>
> > Heya team,
> >
> > Over here at $dayjob, we have an increasing reliance on Kubernetes for
> > both development and production workloads. Our tools are maturing and
> > we're hoping that they might be of interest to the wider community.
> > I'd like to see if there's community interest in receiving some/any of
> > them as a contribution. I think we'll also need a plan from ASF Infra
> > that makes kubernetes available to us as a project.
>
>
> > We have implemented a basic stack of tools for orchestrating ZK + HDFS
> > + HBase on Kubernetes. We use this for running a small local dev
> > cluster via MiniKube/KIND ; for ITBLL on smallish distributed clusters
> > in a public cloud ; and in production for running clusters of ~100
> > Data Nodes/Region Servers in a public cloud. There was an earlier
> > discussion about using our donation of test hardware for running more
> > thorough tests in our CI, but one of the limiting factors is full
> > cluster deployment. I hope that the community might be interested in
> > receiving this tooling as a foundation for more rigorous correctness
> > and maybe even performance tests in the open. Furthermore, perhaps the
> > wider community has interest in an Apache licensed cluster
> > orchestration tool for other uses.
> >
> > Now for some details: The implementation is built on Kustomize, so
> > it's fundamentally transparent resource specification with yaml
> > patches for composability; this is in contrast to a solution using
> > templates with defined capabilities and interfaces. There is no
> > operator ; it's all coordinated via init/bootstrap containers, shell
> > scripts, shared volumes for state, &c. For now.
>
>
> > Such a donation will amount to a code drop, which will have its
> > challenges. I'm motivated via internal processes to carve it into
> > smaller pieces, and I think that will benefit community review as
> > well. Perhaps this approach could be used to make the contribution via
> > a feature branch.
> >
> > Is there community interest in adding such a capability to our
> > maintained responsibilities? I'd hope that we have several volunteers
> > to work with me through the contribution process, and who are
> > reasonably confident that they'll be able to help maintain such a
> > capability going forward. We'll also need someone who can work with
> > Infra to get us access to Kubernetes cluster(s), via whatever means.
> >
> > What do you think?
> >
> > Thanks,
> > Nick & the HBase team at Apple

Re: [DISCUSS] Kubernetes Orchestration for ZK, HDFS, and HBase

[Nick Dimiduk]

Thank you both for your comments. I'm happy to hear that there is some
interest in this pursuit.

Before we get too deep into details of the implementation, I'm concerned
that we have support from ASF Infra. Can we install KIND or MiniKube on
Jenkins worker hosts? Can we get a full cluster provisioned in some
environment, 3PC or otherwise? How can we manage permissions and resource
access? Do we want this to be confined to CI, or will we grant permissions
to committers as well?

On Tue, Mar 14, 2023 at 6:18 PM Lars Francke  wrote:

> Hi Nick,
>
> I do not mean to derail your mail so I'll keep mine short: Yes, I
> think testing & infrastructure on Kubernetes would be worthwhile and I
> thank you for the offer.
> We're happy to take a look and would try to review any incoming
> contributions depending on how large/digestable they are :)
>
> We[1] are developing our own operator[2] and during that mission have
> learned how prone Kubernetes is to bit rot so it'd be great if your
> team would continue helping out.
> I'm happy to go into details on why we did what we did but that'd be a
> separate thread.
>
> Cheers,
> Lars
>
> [1] 
> [2] 
>
> On Tue, Mar 14, 2023 at 3:23 PM Mallikarjun 
> wrote:
> >
> > Hi Nick,
> >
> > I agree with your thought that there is an increasing reliance on
> > kubernetes, more so for complex workloads like hbase deployments because
> of
> > unavailability of reliable automation frameworks outside of k8s.
> >
> > But I have a slightly different view in terms of how to achieve it. When
> I
> > was exploring what are the possibilities such as kustomize or helm or
> > operator. I found it can get pretty complex in terms of writing
> extensible
> > deployment manifest (for different kinds of deployments) with tools like
> > kustomize or helm. Here is our attempt to conterairise hbase with
> operator
> > --> https://github.com/flipkart-incubator/hbase-k8s-operator
> >
> > ---
> > Mallikarjun
> >
> > On Mon, Mar 13, 2023 at 3:58 PM Nick Dimiduk 
> wrote:
> >
> > > Heya team,
> > >
> > > Over here at $dayjob, we have an increasing reliance on Kubernetes for
> > > both development and production workloads. Our tools are maturing and
> > > we're hoping that they might be of interest to the wider community.
> > > I'd like to see if there's community interest in receiving some/any of
> > > them as a contribution. I think we'll also need a plan from ASF Infra
> > > that makes kubernetes available to us as a project.
> >
> >
> > > We have implemented a basic stack of tools for orchestrating ZK + HDFS
> > > + HBase on Kubernetes. We use this for running a small local dev
> > > cluster via MiniKube/KIND ; for ITBLL on smallish distributed clusters
> > > in a public cloud ; and in production for running clusters of ~100
> > > Data Nodes/Region Servers in a public cloud. There was an earlier
> > > discussion about using our donation of test hardware for running more
> > > thorough tests in our CI, but one of the limiting factors is full
> > > cluster deployment. I hope that the community might be interested in
> > > receiving this tooling as a foundation for more rigorous correctness
> > > and maybe even performance tests in the open. Furthermore, perhaps the
> > > wider community has interest in an Apache licensed cluster
> > > orchestration tool for other uses.
> > >
> > > Now for some details: The implementation is built on Kustomize, so
> > > it's fundamentally transparent resource specification with yaml
> > > patches for composability; this is in contrast to a solution using
> > > templates with defined capabilities and interfaces. There is no
> > > operator ; it's all coordinated via init/bootstrap containers, shell
> > > scripts, shared volumes for state, &c. For now.
> >
> >
> > > Such a donation will amount to a code drop, which will have its
> > > challenges. I'm motivated via internal processes to carve it into
> > > smaller pieces, and I think that will benefit community review as
> > > well. Perhaps this approach could be used to make the contribution via
> > > a feature branch.
> > >
> > > Is there community interest in adding such a capability to our
> > > maintained responsibilities? I'd hope that we have several volunteers
> > > to work with me through the contribution process, and who are
> > > reasonably confident that they'll be able to help maintain such a
> > > capability going forward. We'll also need someone who can work with
> > > Infra to get us access to Kubernetes cluster(s), via whatever means.
> > >
> > > What do you think?
> > >
> > > Thanks,
> > > Nick & the HBase team at Apple
>

Re: [DISCUSS] Kubernetes Orchestration for ZK, HDFS, and HBase

[Tak Lon (Stephen) Wu]

Hi Nick,

Other than your concerns on the Apache infrastructure, our end are
interests and discussed internally about how HBase could be deployed in
K8s, but would it be possible to share any architecture diagram or design
documentation?

Although moving ZK, HDFS, HBase sounds like as simple as using Kustomize,
we have few questions on the details.

1. After deploying in K8s, did you introduce any change in HBase to support
multi-tenancy other than using AD / Kerberous ?
2. Did you guys face any challenge after HDFS deploying in K8s? e.g.
Network topology (hostname/IP problem) or performance issue?

That's a bit detail and we can take above question offline if we have a
JIRA or other share documentation as well.

Thanks,
Stephen



On Fri, Mar 17, 2023 at 5:10 AM Nick Dimiduk  wrote:

> Thank you both for your comments. I'm happy to hear that there is some
> interest in this pursuit.
>
> Before we get too deep into details of the implementation, I'm concerned
> that we have support from ASF Infra. Can we install KIND or MiniKube on
> Jenkins worker hosts? Can we get a full cluster provisioned in some
> environment, 3PC or otherwise? How can we manage permissions and resource
> access? Do we want this to be confined to CI, or will we grant permissions
> to committers as well?
>
> On Tue, Mar 14, 2023 at 6:18 PM Lars Francke 
> wrote:
>
> > Hi Nick,
> >
> > I do not mean to derail your mail so I'll keep mine short: Yes, I
> > think testing & infrastructure on Kubernetes would be worthwhile and I
> > thank you for the offer.
> > We're happy to take a look and would try to review any incoming
> > contributions depending on how large/digestable they are :)
> >
> > We[1] are developing our own operator[2] and during that mission have
> > learned how prone Kubernetes is to bit rot so it'd be great if your
> > team would continue helping out.
> > I'm happy to go into details on why we did what we did but that'd be a
> > separate thread.
> >
> > Cheers,
> > Lars
> >
> > [1] 
> > [2] 
> >
> > On Tue, Mar 14, 2023 at 3:23 PM Mallikarjun 
> > wrote:
> > >
> > > Hi Nick,
> > >
> > > I agree with your thought that there is an increasing reliance on
> > > kubernetes, more so for complex workloads like hbase deployments
> because
> > of
> > > unavailability of reliable automation frameworks outside of k8s.
> > >
> > > But I have a slightly different view in terms of how to achieve it.
> When
> > I
> > > was exploring what are the possibilities such as kustomize or helm or
> > > operator. I found it can get pretty complex in terms of writing
> > extensible
> > > deployment manifest (for different kinds of deployments) with tools
> like
> > > kustomize or helm. Here is our attempt to conterairise hbase with
> > operator
> > > --> https://github.com/flipkart-incubator/hbase-k8s-operator
> > >
> > > ---
> > > Mallikarjun
> > >
> > > On Mon, Mar 13, 2023 at 3:58 PM Nick Dimiduk 
> > wrote:
> > >
> > > > Heya team,
> > > >
> > > > Over here at $dayjob, we have an increasing reliance on Kubernetes
> for
> > > > both development and production workloads. Our tools are maturing and
> > > > we're hoping that they might be of interest to the wider community.
> > > > I'd like to see if there's community interest in receiving some/any
> of
> > > > them as a contribution. I think we'll also need a plan from ASF Infra
> > > > that makes kubernetes available to us as a project.
> > >
> > >
> > > > We have implemented a basic stack of tools for orchestrating ZK +
> HDFS
> > > > + HBase on Kubernetes. We use this for running a small local dev
> > > > cluster via MiniKube/KIND ; for ITBLL on smallish distributed
> clusters
> > > > in a public cloud ; and in production for running clusters of ~100
> > > > Data Nodes/Region Servers in a public cloud. There was an earlier
> > > > discussion about using our donation of test hardware for running more
> > > > thorough tests in our CI, but one of the limiting factors is full
> > > > cluster deployment. I hope that the community might be interested in
> > > > receiving this tooling as a foundation for more rigorous correctness
> > > > and maybe even performance tests in the open. Furthermore, perhaps
> the
> > > > wider community has interest in an Apache licensed cluster
> > > > orchestration tool for other uses.
> > > >
> > > > Now for some details: The implementation is built on Kustomize, so
> > > > it's fundamentally transparent resource specification with yaml
> > > > patches for composability; this is in contrast to a solution using
> > > > templates with defined capabilities and interfaces. There is no
> > > > operator ; it's all coordinated via init/bootstrap containers, shell
> > > > scripts, shared volumes for state, &c. For now.
> > >
> > >
> > > > Such a donation will amount to a code drop, which will have its
> > > > challenges. I'm motivated via internal processes to carve it into
> > > > smaller pieces, and

Re: [DISCUSS] Kubernetes Orchestration for ZK, HDFS, and HBase

[Andrew Purtell]

We have also completed a migration to kubernetes (more specifically, EKS)
at $dayjob. Our basic deployment stack is helm, terraform, and spinnaker,
so we took a different approach in implementation. I think we would be
happy to review and offer advice/opinion/suggestion on points of
commonality, though.

Then like Stephen asked, as we moved our stack onto k8s we did encounter a
few issues, like inappropriate caching of DNS resolutions in Hadoop and
HBase. The issues I know of in that regard have been fixed in open source
as long as you are running the latest --- Hadoop 3.3, HBase 2.5, Zookeeper
3.7 or 3.8, etc. But there will be I think common cause and interest in
addressing any remaining challenges that may reveal themselves once we have
KIND or Minicube running on ASF infrastructure and the HBase stack deployed
on top of it.


On Fri, Mar 17, 2023 at 9:52 AM Tak Lon (Stephen) Wu 
wrote:

> Hi Nick,
>
> Other than your concerns on the Apache infrastructure, our end are
> interests and discussed internally about how HBase could be deployed in
> K8s, but would it be possible to share any architecture diagram or design
> documentation?
>
> Although moving ZK, HDFS, HBase sounds like as simple as using Kustomize,
> we have few questions on the details.
>
> 1. After deploying in K8s, did you introduce any change in HBase to support
> multi-tenancy other than using AD / Kerberous ?
> 2. Did you guys face any challenge after HDFS deploying in K8s? e.g.
> Network topology (hostname/IP problem) or performance issue?
>
> That's a bit detail and we can take above question offline if we have a
> JIRA or other share documentation as well.
>
> Thanks,
> Stephen
>
>
>
> On Fri, Mar 17, 2023 at 5:10 AM Nick Dimiduk  wrote:
>
> > Thank you both for your comments. I'm happy to hear that there is some
> > interest in this pursuit.
> >
> > Before we get too deep into details of the implementation, I'm concerned
> > that we have support from ASF Infra. Can we install KIND or MiniKube on
> > Jenkins worker hosts? Can we get a full cluster provisioned in some
> > environment, 3PC or otherwise? How can we manage permissions and resource
> > access? Do we want this to be confined to CI, or will we grant
> permissions
> > to committers as well?
> >
> > On Tue, Mar 14, 2023 at 6:18 PM Lars Francke 
> > wrote:
> >
> > > Hi Nick,
> > >
> > > I do not mean to derail your mail so I'll keep mine short: Yes, I
> > > think testing & infrastructure on Kubernetes would be worthwhile and I
> > > thank you for the offer.
> > > We're happy to take a look and would try to review any incoming
> > > contributions depending on how large/digestable they are :)
> > >
> > > We[1] are developing our own operator[2] and during that mission have
> > > learned how prone Kubernetes is to bit rot so it'd be great if your
> > > team would continue helping out.
> > > I'm happy to go into details on why we did what we did but that'd be a
> > > separate thread.
> > >
> > > Cheers,
> > > Lars
> > >
> > > [1] 
> > > [2] 
> > >
> > > On Tue, Mar 14, 2023 at 3:23 PM Mallikarjun 
> > > wrote:
> > > >
> > > > Hi Nick,
> > > >
> > > > I agree with your thought that there is an increasing reliance on
> > > > kubernetes, more so for complex workloads like hbase deployments
> > because
> > > of
> > > > unavailability of reliable automation frameworks outside of k8s.
> > > >
> > > > But I have a slightly different view in terms of how to achieve it.
> > When
> > > I
> > > > was exploring what are the possibilities such as kustomize or helm or
> > > > operator. I found it can get pretty complex in terms of writing
> > > extensible
> > > > deployment manifest (for different kinds of deployments) with tools
> > like
> > > > kustomize or helm. Here is our attempt to conterairise hbase with
> > > operator
> > > > --> https://github.com/flipkart-incubator/hbase-k8s-operator
> > > >
> > > > ---
> > > > Mallikarjun
> > > >
> > > > On Mon, Mar 13, 2023 at 3:58 PM Nick Dimiduk 
> > > wrote:
> > > >
> > > > > Heya team,
> > > > >
> > > > > Over here at $dayjob, we have an increasing reliance on Kubernetes
> > for
> > > > > both development and production workloads. Our tools are maturing
> and
> > > > > we're hoping that they might be of interest to the wider community.
> > > > > I'd like to see if there's community interest in receiving some/any
> > of
> > > > > them as a contribution. I think we'll also need a plan from ASF
> Infra
> > > > > that makes kubernetes available to us as a project.
> > > >
> > > >
> > > > > We have implemented a basic stack of tools for orchestrating ZK +
> > HDFS
> > > > > + HBase on Kubernetes. We use this for running a small local dev
> > > > > cluster via MiniKube/KIND ; for ITBLL on smallish distributed
> > clusters
> > > > > in a public cloud ; and in production for running clusters of ~100
> > > > > Data Nodes/Region Servers in a public cloud. There was an earlier
> > 

Re: [DISCUSS] Kubernetes Orchestration for ZK, HDFS, and HBase

[Nick Dimiduk]

Indeed, as with Andrew's comments, we have encountered several "usual
suspects" in terms of network issues. Many of our fixes have either landed
upstream or are waiting in PRs that need to be merged. I'm sure there are
lingering issues, and I'm sure that new issues will arise as we run on more
Kubernetes implementations. For now though, things seem solid enough. Indeed,
we are on a stack of ZooKeeper 3.8, Hadoop 3.3, HBase 2.5. We've also
implemented a basic Chaos Monkey Cluster Manager for interacting with pods,
and re-cast ITBLL with the Map tasks as a Kubernetes Job and the Verify
task as a Spark application. In short, I'd like to provide a set of
primitives that can give us a lot of confidence in the functionality of the
systems in question.

We have not attempted to tackle kerberos. It seems to me like a bad
technology choice for a containerized environment, but I'll be the first to
admit my ignorance in this area and I'm quite prepared to be wrong. There
have been some promising experiments using Cert Manager, but they should be
vetted by a wider audience. I hope that such an effort could lead to a
review and abstraction of how authentication and authorization are handled
in Hadoop generally.

On Fri, Mar 17, 2023 at 6:51 PM Andrew Purtell  wrote:

> We have also completed a migration to kubernetes (more specifically, EKS)
> at $dayjob. Our basic deployment stack is helm, terraform, and spinnaker,
> so we took a different approach in implementation. I think we would be
> happy to review and offer advice/opinion/suggestion on points of
> commonality, though.
>
> Then like Stephen asked, as we moved our stack onto k8s we did encounter a
> few issues, like inappropriate caching of DNS resolutions in Hadoop and
> HBase. The issues I know of in that regard have been fixed in open source
> as long as you are running the latest --- Hadoop 3.3, HBase 2.5, Zookeeper
> 3.7 or 3.8, etc. But there will be I think common cause and interest in
> addressing any remaining challenges that may reveal themselves once we have
> KIND or Minicube running on ASF infrastructure and the HBase stack deployed
> on top of it.
>
>
> On Fri, Mar 17, 2023 at 9:52 AM Tak Lon (Stephen) Wu 
> wrote:
>
> > Hi Nick,
> >
> > Other than your concerns on the Apache infrastructure, our end are
> > interests and discussed internally about how HBase could be deployed in
> > K8s, but would it be possible to share any architecture diagram or design
> > documentation?
> >
> > Although moving ZK, HDFS, HBase sounds like as simple as using Kustomize,
> > we have few questions on the details.
> >
> > 1. After deploying in K8s, did you introduce any change in HBase to
> support
> > multi-tenancy other than using AD / Kerberous ?
> > 2. Did you guys face any challenge after HDFS deploying in K8s? e.g.
> > Network topology (hostname/IP problem) or performance issue?
> >
> > That's a bit detail and we can take above question offline if we have a
> > JIRA or other share documentation as well.
> >
> > Thanks,
> > Stephen
> >
> >
> >
> > On Fri, Mar 17, 2023 at 5:10 AM Nick Dimiduk 
> wrote:
> >
> > > Thank you both for your comments. I'm happy to hear that there is some
> > > interest in this pursuit.
> > >
> > > Before we get too deep into details of the implementation, I'm
> concerned
> > > that we have support from ASF Infra. Can we install KIND or MiniKube on
> > > Jenkins worker hosts? Can we get a full cluster provisioned in some
> > > environment, 3PC or otherwise? How can we manage permissions and
> resource
> > > access? Do we want this to be confined to CI, or will we grant
> > permissions
> > > to committers as well?
> > >
> > > On Tue, Mar 14, 2023 at 6:18 PM Lars Francke 
> > > wrote:
> > >
> > > > Hi Nick,
> > > >
> > > > I do not mean to derail your mail so I'll keep mine short: Yes, I
> > > > think testing & infrastructure on Kubernetes would be worthwhile and
> I
> > > > thank you for the offer.
> > > > We're happy to take a look and would try to review any incoming
> > > > contributions depending on how large/digestable they are :)
> > > >
> > > > We[1] are developing our own operator[2] and during that mission have
> > > > learned how prone Kubernetes is to bit rot so it'd be great if your
> > > > team would continue helping out.
> > > > I'm happy to go into details on why we did what we did but that'd be
> a
> > > > separate thread.
> > > >
> > > > Cheers,
> > > > Lars
> > > >
> > > > [1] 
> > > > [2] 
> > > >
> > > > On Tue, Mar 14, 2023 at 3:23 PM Mallikarjun <
> mallik.v.ar...@gmail.com>
> > > > wrote:
> > > > >
> > > > > Hi Nick,
> > > > >
> > > > > I agree with your thought that there is an increasing reliance on
> > > > > kubernetes, more so for complex workloads like hbase deployments
> > > because
> > > > of
> > > > > unavailability of reliable automation frameworks outside of k8s.
> > > > >
> > > > > But I have a slightly different view in terms of 

Re: [DISCUSS] Kubernetes Orchestration for ZK, HDFS, and HBase

[Andrew Purtell]

We have implemented deployments secured with Kerberos. It is no less painful on 
kubernetes. 

Our team at $dayjob does not operate or manage the Kerberos service itself, I 
suppose that shifts the pain. I could not give advice on how best to operate 
krb on k8s beyond “don’t do it”.

Consider an encrypted overlay network for transport security instead, perhaps 
based on istio or envoy. HBase does have a nifty new mTLS based RPC in branch-2 
that does need to make its way into a released version first but would be what 
I’d point to as a starting point for native transport security if that is 
desired, but this only addresses the HBase layer, you would still need to 
secure the dependencies (like hdfs), and tackling this at the network layer 
instead would be simpler. This will not help you if you will have an 
adversarial relationship with your service clients and require more than simple 
auth, though. Security is going to be something each user must architect for 
themselves but I think we could maybe design and offer some configuration 
primitives to aid them. I know how to do that with helm but not kustomize but 
could look into it. 

> On Mar 22, 2023, at 4:46 AM, Nick Dimiduk  wrote:
> 
> Indeed, as with Andrew's comments, we have encountered several "usual
> suspects" in terms of network issues. Many of our fixes have either landed
> upstream or are waiting in PRs that need to be merged. I'm sure there are
> lingering issues, and I'm sure that new issues will arise as we run on more
> Kubernetes implementations. For now though, things seem solid enough. Indeed,
> we are on a stack of ZooKeeper 3.8, Hadoop 3.3, HBase 2.5. We've also
> implemented a basic Chaos Monkey Cluster Manager for interacting with pods,
> and re-cast ITBLL with the Map tasks as a Kubernetes Job and the Verify
> task as a Spark application. In short, I'd like to provide a set of
> primitives that can give us a lot of confidence in the functionality of the
> systems in question.
> 
> We have not attempted to tackle kerberos. It seems to me like a bad
> technology choice for a containerized environment, but I'll be the first to
> admit my ignorance in this area and I'm quite prepared to be wrong. There
> have been some promising experiments using Cert Manager, but they should be
> vetted by a wider audience. I hope that such an effort could lead to a
> review and abstraction of how authentication and authorization are handled
> in Hadoop generally.
> 
>> On Fri, Mar 17, 2023 at 6:51 PM Andrew Purtell  wrote:
>> 
>> We have also completed a migration to kubernetes (more specifically, EKS)
>> at $dayjob. Our basic deployment stack is helm, terraform, and spinnaker,
>> so we took a different approach in implementation. I think we would be
>> happy to review and offer advice/opinion/suggestion on points of
>> commonality, though.
>> 
>> Then like Stephen asked, as we moved our stack onto k8s we did encounter a
>> few issues, like inappropriate caching of DNS resolutions in Hadoop and
>> HBase. The issues I know of in that regard have been fixed in open source
>> as long as you are running the latest --- Hadoop 3.3, HBase 2.5, Zookeeper
>> 3.7 or 3.8, etc. But there will be I think common cause and interest in
>> addressing any remaining challenges that may reveal themselves once we have
>> KIND or Minicube running on ASF infrastructure and the HBase stack deployed
>> on top of it.
>> 
>> 
>> On Fri, Mar 17, 2023 at 9:52 AM Tak Lon (Stephen) Wu 
>> wrote:
>> 
>>> Hi Nick,
>>> 
>>> Other than your concerns on the Apache infrastructure, our end are
>>> interests and discussed internally about how HBase could be deployed in
>>> K8s, but would it be possible to share any architecture diagram or design
>>> documentation?
>>> 
>>> Although moving ZK, HDFS, HBase sounds like as simple as using Kustomize,
>>> we have few questions on the details.
>>> 
>>> 1. After deploying in K8s, did you introduce any change in HBase to
>> support
>>> multi-tenancy other than using AD / Kerberous ?
>>> 2. Did you guys face any challenge after HDFS deploying in K8s? e.g.
>>> Network topology (hostname/IP problem) or performance issue?
>>> 
>>> That's a bit detail and we can take above question offline if we have a
>>> JIRA or other share documentation as well.
>>> 
>>> Thanks,
>>> Stephen
>>> 
>>> 
>>> 
>>> On Fri, Mar 17, 2023 at 5:10 AM Nick Dimiduk 
>> wrote:
>>> 
 Thank you both for your comments. I'm happy to hear that there is some
 interest in this pursuit.
 
 Before we get too deep into details of the implementation, I'm
>> concerned
 that we have support from ASF Infra. Can we install KIND or MiniKube on
 Jenkins worker hosts? Can we get a full cluster provisioned in some
 environment, 3PC or otherwise? How can we manage permissions and
>> resource
 access? Do we want this to be confined to CI, or will we grant
>>> permissions
 to committers as well?
 
 On Tue, Mar 14, 2023 at 6:18 PM Lars Francke 
 w

Re: [DISCUSS] Kubernetes Orchestration for ZK, HDFS, and HBase

[Nick Dimiduk]

Heya team,

I've created a feature ticket [0] from which this contribution can hang.
I've created an associated release version [1] and feature branch [2]
against which we can target PRs while things take shape. I've published my
initial extraction of this feature as a whole for your review [3] -- take a
look at the big picture there. For each commit on that branch, I've created
a sub-task on HBASE-27827. Probably reviewers will find other items we need
to peel off as sub-tasks. I'll start turning each of these commits into PRs
suitable for the Apache repo and your perusal.

I think we're getting due for the 1.3 release of Operator Tools, so I
expect this will land in 1.4.

Thanks,
Nick

[0]: https://issues.apache.org/jira/browse/HBASE-27827
[1]: https://issues.apache.org/jira/browse/HBASE/fixforversion/12353199
[2]:
https://github.com/apache/hbase-operator-tools/tree/HBASE-27827-kubernetes-deployment
[3]:
https://github.com/apache/hbase-operator-tools/compare/HBASE-27827-kubernetes-deployment...ndimiduk:hbase-operator-tools:HBASE-27827-kubernetes-deployment



On Mon, Mar 13, 2023 at 11:28 AM Nick Dimiduk  wrote:

> Heya team,
>
> Over here at $dayjob, we have an increasing reliance on Kubernetes for
> both development and production workloads. Our tools are maturing and
> we're hoping that they might be of interest to the wider community.
> I'd like to see if there's community interest in receiving some/any of
> them as a contribution. I think we'll also need a plan from ASF Infra
> that makes kubernetes available to us as a project.
>
> We have implemented a basic stack of tools for orchestrating ZK + HDFS
> + HBase on Kubernetes. We use this for running a small local dev
> cluster via MiniKube/KIND ; for ITBLL on smallish distributed clusters
> in a public cloud ; and in production for running clusters of ~100
> Data Nodes/Region Servers in a public cloud. There was an earlier
> discussion about using our donation of test hardware for running more
> thorough tests in our CI, but one of the limiting factors is full
> cluster deployment. I hope that the community might be interested in
> receiving this tooling as a foundation for more rigorous correctness
> and maybe even performance tests in the open. Furthermore, perhaps the
> wider community has interest in an Apache licensed cluster
> orchestration tool for other uses.
>
> Now for some details: The implementation is built on Kustomize, so
> it's fundamentally transparent resource specification with yaml
> patches for composability; this is in contrast to a solution using
> templates with defined capabilities and interfaces. There is no
> operator ; it's all coordinated via init/bootstrap containers, shell
> scripts, shared volumes for state, &c. For now.
>
> Such a donation will amount to a code drop, which will have its
> challenges. I'm motivated via internal processes to carve it into
> smaller pieces, and I think that will benefit community review as
> well. Perhaps this approach could be used to make the contribution via
> a feature branch.
>
> Is there community interest in adding such a capability to our
> maintained responsibilities? I'd hope that we have several volunteers
> to work with me through the contribution process, and who are
> reasonably confident that they'll be able to help maintain such a
> capability going forward. We'll also need someone who can work with
> Infra to get us access to Kubernetes cluster(s), via whatever means.
>
> What do you think?
>
> Thanks,
> Nick & the HBase team at Apple
>

Re: [DISCUSS] Kubernetes Orchestration for ZK, HDFS, and HBase

[Nick Dimiduk]

Heya team,

I have created individual pull-requests for each of the major functional
pieces outlined in the initial branch. These await review.

I've now started working integrating the test harness into the maven build.
After a brief detour for a Yetus plugin, I'm now looking instead at maven
integration via exec-maven-plugin. I'm also investigating how to pull the
container image build up into maven as well.

Thanks,
Nic,

On Thu, May 4, 2023 at 2:42 PM Nick Dimiduk  wrote:

> Heya team,
>
> I've created a feature ticket [0] from which this contribution can hang.
> I've created an associated release version [1] and feature branch [2]
> against which we can target PRs while things take shape. I've published my
> initial extraction of this feature as a whole for your review [3] -- take a
> look at the big picture there. For each commit on that branch, I've created
> a sub-task on HBASE-27827. Probably reviewers will find other items we need
> to peel off as sub-tasks. I'll start turning each of these commits into PRs
> suitable for the Apache repo and your perusal.
>
> I think we're getting due for the 1.3 release of Operator Tools, so I
> expect this will land in 1.4.
>
> Thanks,
> Nick
>
> [0]: https://issues.apache.org/jira/browse/HBASE-27827
> [1]: https://issues.apache.org/jira/browse/HBASE/fixforversion/12353199
> [2]:
> https://github.com/apache/hbase-operator-tools/tree/HBASE-27827-kubernetes-deployment
> [3]:
> https://github.com/apache/hbase-operator-tools/compare/HBASE-27827-kubernetes-deployment...ndimiduk:hbase-operator-tools:HBASE-27827-kubernetes-deployment
>
>
>
> On Mon, Mar 13, 2023 at 11:28 AM Nick Dimiduk  wrote:
>
>> Heya team,
>>
>> Over here at $dayjob, we have an increasing reliance on Kubernetes for
>> both development and production workloads. Our tools are maturing and
>> we're hoping that they might be of interest to the wider community.
>> I'd like to see if there's community interest in receiving some/any of
>> them as a contribution. I think we'll also need a plan from ASF Infra
>> that makes kubernetes available to us as a project.
>>
>> We have implemented a basic stack of tools for orchestrating ZK + HDFS
>> + HBase on Kubernetes. We use this for running a small local dev
>> cluster via MiniKube/KIND ; for ITBLL on smallish distributed clusters
>> in a public cloud ; and in production for running clusters of ~100
>> Data Nodes/Region Servers in a public cloud. There was an earlier
>> discussion about using our donation of test hardware for running more
>> thorough tests in our CI, but one of the limiting factors is full
>> cluster deployment. I hope that the community might be interested in
>> receiving this tooling as a foundation for more rigorous correctness
>> and maybe even performance tests in the open. Furthermore, perhaps the
>> wider community has interest in an Apache licensed cluster
>> orchestration tool for other uses.
>>
>> Now for some details: The implementation is built on Kustomize, so
>> it's fundamentally transparent resource specification with yaml
>> patches for composability; this is in contrast to a solution using
>> templates with defined capabilities and interfaces. There is no
>> operator ; it's all coordinated via init/bootstrap containers, shell
>> scripts, shared volumes for state, &c. For now.
>>
>> Such a donation will amount to a code drop, which will have its
>> challenges. I'm motivated via internal processes to carve it into
>> smaller pieces, and I think that will benefit community review as
>> well. Perhaps this approach could be used to make the contribution via
>> a feature branch.
>>
>> Is there community interest in adding such a capability to our
>> maintained responsibilities? I'd hope that we have several volunteers
>> to work with me through the contribution process, and who are
>> reasonably confident that they'll be able to help maintain such a
>> capability going forward. We'll also need someone who can work with
>> Infra to get us access to Kubernetes cluster(s), via whatever means.
>>
>> What do you think?
>>
>> Thanks,
>> Nick & the HBase team at Apple
>>
>

Re: [DISCUSS] Kubernetes Orchestration for ZK, HDFS, and HBase

[Nick Dimiduk]

Heya team,

I have rebuilt one of the early PRs so that the docker image build pieces
are integrated with the maven build. If this is acceptable to the
reviewers, I'll go forward with integrating the other images and
kustomize/kuttl tests in the same way.

Please take a look.

Thanks,
Nick

https://github.com/apache/hbase-operator-tools/pull/118

On Fri, May 12, 2023 at 4:19 PM Nick Dimiduk  wrote:

> Heya team,
>
> I have created individual pull-requests for each of the major functional
> pieces outlined in the initial branch. These await review.
>
> I've now started working integrating the test harness into the maven
> build. After a brief detour for a Yetus plugin, I'm now looking instead at
> maven integration via exec-maven-plugin. I'm also investigating how to pull
> the container image build up into maven as well.
>
> Thanks,
> Nic,
>
> On Thu, May 4, 2023 at 2:42 PM Nick Dimiduk  wrote:
>
>> Heya team,
>>
>> I've created a feature ticket [0] from which this contribution can hang.
>> I've created an associated release version [1] and feature branch [2]
>> against which we can target PRs while things take shape. I've published my
>> initial extraction of this feature as a whole for your review [3] -- take a
>> look at the big picture there. For each commit on that branch, I've created
>> a sub-task on HBASE-27827. Probably reviewers will find other items we need
>> to peel off as sub-tasks. I'll start turning each of these commits into PRs
>> suitable for the Apache repo and your perusal.
>>
>> I think we're getting due for the 1.3 release of Operator Tools, so I
>> expect this will land in 1.4.
>>
>> Thanks,
>> Nick
>>
>> [0]: https://issues.apache.org/jira/browse/HBASE-27827
>> [1]: https://issues.apache.org/jira/browse/HBASE/fixforversion/12353199
>> [2]:
>> https://github.com/apache/hbase-operator-tools/tree/HBASE-27827-kubernetes-deployment
>> [3]:
>> https://github.com/apache/hbase-operator-tools/compare/HBASE-27827-kubernetes-deployment...ndimiduk:hbase-operator-tools:HBASE-27827-kubernetes-deployment
>>
>>
>>
>> On Mon, Mar 13, 2023 at 11:28 AM Nick Dimiduk 
>> wrote:
>>
>>> Heya team,
>>>
>>> Over here at $dayjob, we have an increasing reliance on Kubernetes for
>>> both development and production workloads. Our tools are maturing and
>>> we're hoping that they might be of interest to the wider community.
>>> I'd like to see if there's community interest in receiving some/any of
>>> them as a contribution. I think we'll also need a plan from ASF Infra
>>> that makes kubernetes available to us as a project.
>>>
>>> We have implemented a basic stack of tools for orchestrating ZK + HDFS
>>> + HBase on Kubernetes. We use this for running a small local dev
>>> cluster via MiniKube/KIND ; for ITBLL on smallish distributed clusters
>>> in a public cloud ; and in production for running clusters of ~100
>>> Data Nodes/Region Servers in a public cloud. There was an earlier
>>> discussion about using our donation of test hardware for running more
>>> thorough tests in our CI, but one of the limiting factors is full
>>> cluster deployment. I hope that the community might be interested in
>>> receiving this tooling as a foundation for more rigorous correctness
>>> and maybe even performance tests in the open. Furthermore, perhaps the
>>> wider community has interest in an Apache licensed cluster
>>> orchestration tool for other uses.
>>>
>>> Now for some details: The implementation is built on Kustomize, so
>>> it's fundamentally transparent resource specification with yaml
>>> patches for composability; this is in contrast to a solution using
>>> templates with defined capabilities and interfaces. There is no
>>> operator ; it's all coordinated via init/bootstrap containers, shell
>>> scripts, shared volumes for state, &c. For now.
>>>
>>> Such a donation will amount to a code drop, which will have its
>>> challenges. I'm motivated via internal processes to carve it into
>>> smaller pieces, and I think that will benefit community review as
>>> well. Perhaps this approach could be used to make the contribution via
>>> a feature branch.
>>>
>>> Is there community interest in adding such a capability to our
>>> maintained responsibilities? I'd hope that we have several volunteers
>>> to work with me through the contribution process, and who are
>>> reasonably confident that they'll be able to help maintain such a
>>> capability going forward. We'll also need someone who can work with
>>> Infra to get us access to Kubernetes cluster(s), via whatever means.
>>>
>>> What do you think?
>>>
>>> Thanks,
>>> Nick & the HBase team at Apple
>>>
>>

Re: [DISCUSS] Kubernetes Orchestration for ZK, HDFS, and HBase

[TAK-LON WU]

Sorry that I’m on vacation and will be back online after 06/06 , but thanks
for putting the PR out and I believe someone on our side will review it . (
or when I come back I will review them)

-Stephen

On Mon, May 22, 2023 at 6:08 PM Nick Dimiduk  wrote:

> Heya team,
>
> I have rebuilt one of the early PRs so that the docker image build pieces
> are integrated with the maven build. If this is acceptable to the
> reviewers, I'll go forward with integrating the other images and
> kustomize/kuttl tests in the same way.
>
> Please take a look.
>
> Thanks,
> Nick
>
> https://github.com/apache/hbase-operator-tools/pull/118
>
> On Fri, May 12, 2023 at 4:19 PM Nick Dimiduk  wrote:
>
> > Heya team,
> >
> > I have created individual pull-requests for each of the major functional
> > pieces outlined in the initial branch. These await review.
> >
> > I've now started working integrating the test harness into the maven
> > build. After a brief detour for a Yetus plugin, I'm now looking instead
> at
> > maven integration via exec-maven-plugin. I'm also investigating how to
> pull
> > the container image build up into maven as well.
> >
> > Thanks,
> > Nic,
> >
> > On Thu, May 4, 2023 at 2:42 PM Nick Dimiduk  wrote:
> >
> >> Heya team,
> >>
> >> I've created a feature ticket [0] from which this contribution can hang.
> >> I've created an associated release version [1] and feature branch [2]
> >> against which we can target PRs while things take shape. I've published
> my
> >> initial extraction of this feature as a whole for your review [3] --
> take a
> >> look at the big picture there. For each commit on that branch, I've
> created
> >> a sub-task on HBASE-27827. Probably reviewers will find other items we
> need
> >> to peel off as sub-tasks. I'll start turning each of these commits into
> PRs
> >> suitable for the Apache repo and your perusal.
> >>
> >> I think we're getting due for the 1.3 release of Operator Tools, so I
> >> expect this will land in 1.4.
> >>
> >> Thanks,
> >> Nick
> >>
> >> [0]: https://issues.apache.org/jira/browse/HBASE-27827
> >> [1]: https://issues.apache.org/jira/browse/HBASE/fixforversion/12353199
> >> [2]:
> >>
> https://github.com/apache/hbase-operator-tools/tree/HBASE-27827-kubernetes-deployment
> >> [3]:
> >>
> https://github.com/apache/hbase-operator-tools/compare/HBASE-27827-kubernetes-deployment...ndimiduk:hbase-operator-tools:HBASE-27827-kubernetes-deployment
> >>
> >>
> >>
> >> On Mon, Mar 13, 2023 at 11:28 AM Nick Dimiduk 
> >> wrote:
> >>
> >>> Heya team,
> >>>
> >>> Over here at $dayjob, we have an increasing reliance on Kubernetes for
> >>> both development and production workloads. Our tools are maturing and
> >>> we're hoping that they might be of interest to the wider community.
> >>> I'd like to see if there's community interest in receiving some/any of
> >>> them as a contribution. I think we'll also need a plan from ASF Infra
> >>> that makes kubernetes available to us as a project.
> >>>
> >>> We have implemented a basic stack of tools for orchestrating ZK + HDFS
> >>> + HBase on Kubernetes. We use this for running a small local dev
> >>> cluster via MiniKube/KIND ; for ITBLL on smallish distributed clusters
> >>> in a public cloud ; and in production for running clusters of ~100
> >>> Data Nodes/Region Servers in a public cloud. There was an earlier
> >>> discussion about using our donation of test hardware for running more
> >>> thorough tests in our CI, but one of the limiting factors is full
> >>> cluster deployment. I hope that the community might be interested in
> >>> receiving this tooling as a foundation for more rigorous correctness
> >>> and maybe even performance tests in the open. Furthermore, perhaps the
> >>> wider community has interest in an Apache licensed cluster
> >>> orchestration tool for other uses.
> >>>
> >>> Now for some details: The implementation is built on Kustomize, so
> >>> it's fundamentally transparent resource specification with yaml
> >>> patches for composability; this is in contrast to a solution using
> >>> templates with defined capabilities and interfaces. There is no
> >>> operator ; it's all coordinated via init/bootstrap containers, shell
> >>> scripts, shared volumes for state, &c. For now.
> >>>
> >>> Such a donation will amount to a code drop, which will have its
> >>> challenges. I'm motivated via internal processes to carve it into
> >>> smaller pieces, and I think that will benefit community review as
> >>> well. Perhaps this approach could be used to make the contribution via
> >>> a feature branch.
> >>>
> >>> Is there community interest in adding such a capability to our
> >>> maintained responsibilities? I'd hope that we have several volunteers
> >>> to work with me through the contribution process, and who are
> >>> reasonably confident that they'll be able to help maintain such a
> >>> capability going forward. We'll also need someone who can work with
> >>> Infra to get us access to Kubernetes cluster(s), via what

Re: [DISCUSS] Kubernetes Orchestration for ZK, HDFS, and HBase

[Nick Dimiduk]

I went ahead and rebuilt the Hadoop image module in the same style. I
rebased the zookeeper-single and hdfs kustimize implementations onto the
same structure. So, PR’s #118, #119, #120, and #121 are all in this style.
I don’t have a place for running integration tests, but unit tests are now
running in Jenkins.

I appreciate any attention you can provide.

Thanks,
Nick

On Mon, 22 May 2023 at 16:02, TAK-LON WU  wrote:

> Sorry that I’m on vacation and will be back online after 06/06 , but thanks
> for putting the PR out and I believe someone on our side will review it . (
> or when I come back I will review them)
>
> -Stephen
>
> On Mon, May 22, 2023 at 6:08 PM Nick Dimiduk  wrote:
>
> > Heya team,
> >
> > I have rebuilt one of the early PRs so that the docker image build pieces
> > are integrated with the maven build. If this is acceptable to the
> > reviewers, I'll go forward with integrating the other images and
> > kustomize/kuttl tests in the same way.
> >
> > Please take a look.
> >
> > Thanks,
> > Nick
> >
> > https://github.com/apache/hbase-operator-tools/pull/118
> >
> > On Fri, May 12, 2023 at 4:19 PM Nick Dimiduk 
> wrote:
> >
> > > Heya team,
> > >
> > > I have created individual pull-requests for each of the major
> functional
> > > pieces outlined in the initial branch. These await review.
> > >
> > > I've now started working integrating the test harness into the maven
> > > build. After a brief detour for a Yetus plugin, I'm now looking instead
> > at
> > > maven integration via exec-maven-plugin. I'm also investigating how to
> > pull
> > > the container image build up into maven as well.
> > >
> > > Thanks,
> > > Nic,
> > >
> > > On Thu, May 4, 2023 at 2:42 PM Nick Dimiduk 
> wrote:
> > >
> > >> Heya team,
> > >>
> > >> I've created a feature ticket [0] from which this contribution can
> hang.
> > >> I've created an associated release version [1] and feature branch [2]
> > >> against which we can target PRs while things take shape. I've
> published
> > my
> > >> initial extraction of this feature as a whole for your review [3] --
> > take a
> > >> look at the big picture there. For each commit on that branch, I've
> > created
> > >> a sub-task on HBASE-27827. Probably reviewers will find other items we
> > need
> > >> to peel off as sub-tasks. I'll start turning each of these commits
> into
> > PRs
> > >> suitable for the Apache repo and your perusal.
> > >>
> > >> I think we're getting due for the 1.3 release of Operator Tools, so I
> > >> expect this will land in 1.4.
> > >>
> > >> Thanks,
> > >> Nick
> > >>
> > >> [0]: https://issues.apache.org/jira/browse/HBASE-27827
> > >> [1]:
> https://issues.apache.org/jira/browse/HBASE/fixforversion/12353199
> > >> [2]:
> > >>
> >
> https://github.com/apache/hbase-operator-tools/tree/HBASE-27827-kubernetes-deployment
> > >> [3]:
> > >>
> >
> https://github.com/apache/hbase-operator-tools/compare/HBASE-27827-kubernetes-deployment...ndimiduk:hbase-operator-tools:HBASE-27827-kubernetes-deployment
> > >>
> > >>
> > >>
> > >> On Mon, Mar 13, 2023 at 11:28 AM Nick Dimiduk 
> > >> wrote:
> > >>
> > >>> Heya team,
> > >>>
> > >>> Over here at $dayjob, we have an increasing reliance on Kubernetes
> for
> > >>> both development and production workloads. Our tools are maturing and
> > >>> we're hoping that they might be of interest to the wider community.
> > >>> I'd like to see if there's community interest in receiving some/any
> of
> > >>> them as a contribution. I think we'll also need a plan from ASF Infra
> > >>> that makes kubernetes available to us as a project.
> > >>>
> > >>> We have implemented a basic stack of tools for orchestrating ZK +
> HDFS
> > >>> + HBase on Kubernetes. We use this for running a small local dev
> > >>> cluster via MiniKube/KIND ; for ITBLL on smallish distributed
> clusters
> > >>> in a public cloud ; and in production for running clusters of ~100
> > >>> Data Nodes/Region Servers in a public cloud. There was an earlier
> > >>> discussion about using our donation of test hardware for running more
> > >>> thorough tests in our CI, but one of the limiting factors is full
> > >>> cluster deployment. I hope that the community might be interested in
> > >>> receiving this tooling as a foundation for more rigorous correctness
> > >>> and maybe even performance tests in the open. Furthermore, perhaps
> the
> > >>> wider community has interest in an Apache licensed cluster
> > >>> orchestration tool for other uses.
> > >>>
> > >>> Now for some details: The implementation is built on Kustomize, so
> > >>> it's fundamentally transparent resource specification with yaml
> > >>> patches for composability; this is in contrast to a solution using
> > >>> templates with defined capabilities and interfaces. There is no
> > >>> operator ; it's all coordinated via init/bootstrap containers, shell
> > >>> scripts, shared volumes for state, &c. For now.
> > >>>
> > >>> Such a donation will amount to a code drop, which will have its
> > >>> cha