Great. I definitely would strongly favor carrying over user's intuition from FS unless we think we need a very different model. The minor details like the seperator and namespace term will help with that.
Follow-up question, say I have a layout like /chicago-datacenter/user-events/pageviews Can I subscribe to /chicago-datacenter/user-events to get the full firehose of user events from chicago? Can I subscribe to /*/user-events to get user events originating from all datacenters? (Assuming, for now, that these are all in the same cluster...) Also, just to confirm, it sounds from the proposal like config overrides would become fully hierarchical so you can override config at any directory point. This will add complexity in implementation but I think will likely be much more operator friendly. There are about a thousand details to discuss in terms of how this would impact the metadata request, various zk entries, and various other aspects, but probably it makes sense to first agree on how we would want it to work and then start to dive into how to implement that. -Jay On Mon, Oct 12, 2015 at 5:28 PM, Ashish Singh <asi...@cloudera.com> wrote: > Hey Jay, thanks for reviewing the proposal. Answers inline. > > On Mon, Oct 12, 2015 at 10:53 AM, Jay Kreps <j...@confluent.io> wrote: > > > Hey guys, > > > > I think this is an important feature and one we've talked about for a > > while. I really think trying to invent a new nomenclature is going to > make > > it hard for people to understand, though. As such I recommend we call > > namespaces "directories" and denote them with '/'--this will make the > > feature 1000x more understandable to people. > > Essentially you are suggesting two things here. > 1. Use "Directory" instead of "Namespace" as it is more intuitive. I agree. > 2. Make '/' as delimiter instead of ':'. Fine with me and I agree if we > call these directories, '/' is the way to go. > > I think we should inheret the > > semantics of normal unix fs in so far as it makes sense. > > > > In this approach we get rid of topics entirely, instead we really just > have > > partitions which are the equivalent of a file and retain their numeric > > names, and the existing topic concept is just the first directory level > but > > we generalize to allow arbitrarily many more levels of nesting. This > allows > > categorization of data, such as /datacenter1/user-events/page-views/3 and > > you can subscribe, apply configs or permissions at any level of the > > hierarchy. > > > +1. This actually requires just a minor change to existing proposal, i.e., > "some:namespace:topic" becomes "some/namespace/topic". > > > > > I'm actually not 100% such what the semantics of accessing data in > > differing namespaces is in the current proposal, maybe you can clarify > > Ashish? > > I will add more info to KIP on this, however I think a client should be > able to access data in any namespace as long as following conditions are > satisfied. > > 1. Namespace, the client is trying to access, exists. > 2. The client has sufficient permissions on the namespace for type of > operation the client is trying to perform on a topic within that namespace. > 3. The client has sufficient permissions on the topic for type of operation > the client is trying to perform on that topic. > > If we choose to go with what you suggested earlier that just have hierarchy > of directories, then step 3 will actually be covered in step 2. > > In the current proposal, consumers will subscribe to a topic in a namespace > by specifying <namespace>:<topic> as the topic name. They can subscribe to > topics from multiple namespaces. > > Let me know if I totally missed your question. > > Since the point of Kafka is sharing data I think it is really > > important that the grouping be just for > convenience/permissions/config/etc > > and that it remain possible to access multiple directories/namespaces > from > > the same client. > > > Totally agree with you. > > > > > -Jay > > > > On Fri, Oct 9, 2015 at 6:32 PM, Ashish Singh <asi...@cloudera.com> > wrote: > > > > > Hey Guys, > > > > > > I just created KIP-37 for adding namespaces to Kafka. > > > > > > KIP-37 > > > < > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-37+-+Add+Namespaces+to+Kafka > > > > > > > tracks the proposal. > > > > > > The idea is to make Kafka support multi-tenancy via namespaces. > > > > > > Feedback and comments are welcome. > > > > > > -- > > > > > > Regards, > > > Ashish > > > > > > > > > -- > > Regards, > Ashish >
