Rajini, thanks for the review. Answers inline.
On Mon, Oct 12, 2015 at 8:01 AM, Rajini Sivaram <
rajinisiva...@googlemail.com> wrote:
> Ashish,
>
> Thank you for doing this writeup and starting the discussion around
> namespaces and multi-tenancy.
>
> We have implemented a namespace solution on top of Kafka trunk and our
> motivation to do so matches your description in KIP-37 to a large extent.
> But there are some differences in our approach. The main difference is that
> we are not exposing namespaces to clients. Our goal is to provide a
> multi-tenant interface where each tenant has a view of Kafka that looks and
> feels like a single-user Kafka cluster. This is a quick summary of our
> approach. We are waiting for the SASL implementation in Kafka to complete
> this work and will be happy to share the code once we have integrated with
> SASL.
>
As Jay later suggested, and I agree, clients should be able to talk to any
namespace as long as they have the permission to access that namespace and
the namespace exists.
I think below is a client should be able to do.
1. Get a list of namespaces it has permissions to.
2. Get a list of topics in namespaces it has permissions to.
3. Produce/ Consume to any topic in any namespace as long as that namespace
exists, the client has permissions to the namespace and the topic.
I think as long as we satisfy the above mentioned reqs, your requirement
should by met. Let me know if I am missing something.
>
> *Current implementation:*
>
> *Goal*: Provide a secure multi-tenant Kafka service that works with
> out-of-the-box Kafka 0.9.0.0 clients. We have a requirement to isolate
> tenants so that tenants do not have any awareness of other tenants.
>
I think one the advantages of current proposal is that existing and
previous clients will also be able to work seamlessly with the proposed
changes.
>
> *Implementation*:
>
> - We use a modified version of kafka.server.KafkaApis.scala which
> enables a pluggable interceptor for requests and responses.
> - Our interceptor plugin adds a tenant-specific prefix to topics and
> consumer groups in requests and removes the prefix in responses. At the
> moment, we are using valid topic/group characters in prefixes so that no
> other changes are required in Kafka.
> - Tenants are identified and authenticated based on their security
> principal. We are using clientid to hold an API key temporarily while we
> wait for SASL implementation in Kafka.
> - External clients which connect using SSL+SASL only see the topics
> within their namespace. They have no access to Zookeeper.
> - Our internal clients use PLAINTEXT and do see all topics and also have
> access to Zookeeper.
>
> *Longer term goals:*
>
> - Even though we started our prototype with hierarchical topics, we
> decided against it for our initial release to avoid changes to clients.
> But
> for the longer term, it will definitely be useful to use hierarchical
> topics to provide namespaces. In particular, we would like the option to
> use different encryption keys to encrypt data at rest for different
> tenants. If/when hierarchical topics are supported in Kafka, we would
> use
> the first level of the hierarchy as a tenant prefix.
> - We are currently not using quotas for tenants, but we do have a
> requirement to support quotas.
>
> I think we can make your proposed solution work for us. But I wasn't clear
> on why namespaces would be a preferred approach to full-fledged
> hierarchical topics.
What do you mean by full-fledged hierarchical topics? I think full-fledged
hierarchical topics is a way to implement namespace. Let me know if I am
missing something.
> There is no mention of consumer groups which also
> requires namespaces to support multi-tenancy.
Not sure what you mean by namespaces for consumer groups. As long as
consumers in consumer group have permissions to interact with a namespace,
they should be able to work they work right now. Let me know if I am
missing something.
We prefer to have
> multi-tenant applications which aren't aware of their namespace as we have
> today, but if that is something specific to our environment and the general
> consensus is to make tenants namespace-aware, we may be able to work around
> it.
>
> Thoughts?
>
>
> Regards,
>
> Rajini
>
> On Sat, Oct 10, 2015 at 10:03 AM, Magnus Edenhill <mag...@edenhill.se>
> wrote:
>
> > Good write-up Ashish.
> >
> > Looking at one of the rejected alternatives got me thinking:
> > "Modify request/ response formats to take namespace specifically.
> >
> > Solves the issue of delimiting string required in proposed approach and
> the
> > issue with existing regex consumers.
> > This definitely is the cleanest approach. However, will require lots of
> API
> > and protocol changes. This is listed in rejected alternatives, but we
> > should totally consider this as an option."
> >
> >
> > I think we could actually achieve this without any protocol change at all
> > by moving the namespace token to the common request header's ClientId
> > field.
> > E.g.: .. RequestHeader { ..., ClientId = "myclient@mynamespace", ... }
> >
> > That would provide the desired per-request namespacing and in theory the
> > existing clients dont even need to be updated as long as the clientId is
> > configurable (which it should be).
> >
> >
> > My two cents,
> > Magnus
> >
> >
> > 2015-10-10 3:32 GMT+02:00 Ashish Singh <asi...@cloudera.com>:
> >
> > > Hey Guys,
> > >
> > > I just created KIP-37 for adding namespaces to Kafka.
> > >
> > > KIP-37
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-37+-+Add+Namespaces+to+Kafka
> > > >
> > > tracks the proposal.
> > >
> > > The idea is to make Kafka support multi-tenancy via namespaces.
> > >
> > > Feedback and comments are welcome.
> > >
> > > --
> > >
> > > Regards,
> > > Ashish
> > >
> >
>
--
Regards,
Ashish
