Ashish, Thank you for doing this writeup and starting the discussion around namespaces and multi-tenancy.
We have implemented a namespace solution on top of Kafka trunk and our motivation to do so matches your description in KIP-37 to a large extent. But there are some differences in our approach. The main difference is that we are not exposing namespaces to clients. Our goal is to provide a multi-tenant interface where each tenant has a view of Kafka that looks and feels like a single-user Kafka cluster. This is a quick summary of our approach. We are waiting for the SASL implementation in Kafka to complete this work and will be happy to share the code once we have integrated with SASL. *Current implementation:* *Goal*: Provide a secure multi-tenant Kafka service that works with out-of-the-box Kafka 0.9.0.0 clients. We have a requirement to isolate tenants so that tenants do not have any awareness of other tenants. *Implementation*: - We use a modified version of kafka.server.KafkaApis.scala which enables a pluggable interceptor for requests and responses. - Our interceptor plugin adds a tenant-specific prefix to topics and consumer groups in requests and removes the prefix in responses. At the moment, we are using valid topic/group characters in prefixes so that no other changes are required in Kafka. - Tenants are identified and authenticated based on their security principal. We are using clientid to hold an API key temporarily while we wait for SASL implementation in Kafka. - External clients which connect using SSL+SASL only see the topics within their namespace. They have no access to Zookeeper. - Our internal clients use PLAINTEXT and do see all topics and also have access to Zookeeper. *Longer term goals:* - Even though we started our prototype with hierarchical topics, we decided against it for our initial release to avoid changes to clients. But for the longer term, it will definitely be useful to use hierarchical topics to provide namespaces. In particular, we would like the option to use different encryption keys to encrypt data at rest for different tenants. If/when hierarchical topics are supported in Kafka, we would use the first level of the hierarchy as a tenant prefix. - We are currently not using quotas for tenants, but we do have a requirement to support quotas. I think we can make your proposed solution work for us. But I wasn't clear on why namespaces would be a preferred approach to full-fledged hierarchical topics. There is no mention of consumer groups which also requires namespaces to support multi-tenancy. We prefer to have multi-tenant applications which aren't aware of their namespace as we have today, but if that is something specific to our environment and the general consensus is to make tenants namespace-aware, we may be able to work around it. Thoughts? Regards, Rajini On Sat, Oct 10, 2015 at 10:03 AM, Magnus Edenhill <mag...@edenhill.se> wrote: > Good write-up Ashish. > > Looking at one of the rejected alternatives got me thinking: > "Modify request/ response formats to take namespace specifically. > > Solves the issue of delimiting string required in proposed approach and the > issue with existing regex consumers. > This definitely is the cleanest approach. However, will require lots of API > and protocol changes. This is listed in rejected alternatives, but we > should totally consider this as an option." > > > I think we could actually achieve this without any protocol change at all > by moving the namespace token to the common request header's ClientId > field. > E.g.: .. RequestHeader { ..., ClientId = "myclient@mynamespace", ... } > > That would provide the desired per-request namespacing and in theory the > existing clients dont even need to be updated as long as the clientId is > configurable (which it should be). > > > My two cents, > Magnus > > > 2015-10-10 3:32 GMT+02:00 Ashish Singh <asi...@cloudera.com>: > > > Hey Guys, > > > > I just created KIP-37 for adding namespaces to Kafka. > > > > KIP-37 > > < > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-37+-+Add+Namespaces+to+Kafka > > > > > tracks the proposal. > > > > The idea is to make Kafka support multi-tenancy via namespaces. > > > > Feedback and comments are welcome. > > > > -- > > > > Regards, > > Ashish > > >