Ashish,
Thank you for doing this writeup and starting the discussion around
namespaces and multi-tenancy.
We have implemented a namespace solution on top of Kafka trunk and our
motivation to do so matches your description in KIP-37 to a large extent.
But there are some differences in our approach. The main difference is that
we are not exposing namespaces to clients. Our goal is to provide a
multi-tenant interface where each tenant has a view of Kafka that looks and
feels like a single-user Kafka cluster. This is a quick summary of our
approach. We are waiting for the SASL implementation in Kafka to complete
this work and will be happy to share the code once we have integrated with
SASL.
*Current implementation:*
*Goal*: Provide a secure multi-tenant Kafka service that works with
out-of-the-box Kafka 0.9.0.0 clients. We have a requirement to isolate
tenants so that tenants do not have any awareness of other tenants.
*Implementation*:
- We use a modified version of kafka.server.KafkaApis.scala which
enables a pluggable interceptor for requests and responses.
- Our interceptor plugin adds a tenant-specific prefix to topics and
consumer groups in requests and removes the prefix in responses. At the
moment, we are using valid topic/group characters in prefixes so that no
other changes are required in Kafka.
- Tenants are identified and authenticated based on their security
principal. We are using clientid to hold an API key temporarily while we
wait for SASL implementation in Kafka.
- External clients which connect using SSL+SASL only see the topics
within their namespace. They have no access to Zookeeper.
- Our internal clients use PLAINTEXT and do see all topics and also have
access to Zookeeper.
*Longer term goals:*
- Even though we started our prototype with hierarchical topics, we
decided against it for our initial release to avoid changes to clients. But
for the longer term, it will definitely be useful to use hierarchical
topics to provide namespaces. In particular, we would like the option to
use different encryption keys to encrypt data at rest for different
tenants. If/when hierarchical topics are supported in Kafka, we would use
the first level of the hierarchy as a tenant prefix.
- We are currently not using quotas for tenants, but we do have a
requirement to support quotas.
I think we can make your proposed solution work for us. But I wasn't clear
on why namespaces would be a preferred approach to full-fledged
hierarchical topics. There is no mention of consumer groups which also
requires namespaces to support multi-tenancy. We prefer to have
multi-tenant applications which aren't aware of their namespace as we have
today, but if that is something specific to our environment and the general
consensus is to make tenants namespace-aware, we may be able to work around
it.
Thoughts?
Regards,
Rajini
On Sat, Oct 10, 2015 at 10:03 AM, Magnus Edenhill <mag...@edenhill.se>
wrote:
> Good write-up Ashish.
>
> Looking at one of the rejected alternatives got me thinking:
> "Modify request/ response formats to take namespace specifically.
>
> Solves the issue of delimiting string required in proposed approach and the
> issue with existing regex consumers.
> This definitely is the cleanest approach. However, will require lots of API
> and protocol changes. This is listed in rejected alternatives, but we
> should totally consider this as an option."
>
>
> I think we could actually achieve this without any protocol change at all
> by moving the namespace token to the common request header's ClientId
> field.
> E.g.: .. RequestHeader { ..., ClientId = "myclient@mynamespace", ... }
>
> That would provide the desired per-request namespacing and in theory the
> existing clients dont even need to be updated as long as the clientId is
> configurable (which it should be).
>
>
> My two cents,
> Magnus
>
>
> 2015-10-10 3:32 GMT+02:00 Ashish Singh <asi...@cloudera.com>:
>
> > Hey Guys,
> >
> > I just created KIP-37 for adding namespaces to Kafka.
> >
> > KIP-37
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-37+-+Add+Namespaces+to+Kafka
> > >
> > tracks the proposal.
> >
> > The idea is to make Kafka support multi-tenancy via namespaces.
> >
> > Feedback and comments are welcome.
> >
> > --
> >
> > Regards,
> > Ashish
> >
>
