Okay this is similar to what I think we have talked about before. Let me elaborate on the idea that I think has been floating around--it's pretty similar with a few differences.
I think what you are calling the "default namespace" is basically what I would call the "current working directory" with paths not beginning with '/' being interpreted relative to this directory as in the fs. One thing you have to work out is what levels in this hierarchy you can actually subscribe to. I think you are assuming only what we currently consider a "topic", i.e. the first level of directories but not the partitions or parent dirs, would be subscribable. If you think about it, though, that constraint is a bit arbitrary. I'd propose instead the semantics that: - Subscribing to /a/b/c/0 means subscribing to the 0th partition of topic "c" in directory /a/b - Subscribing to /a/b/c means subscribing to all partitions in topic/directory "c" - Subscribing to /a/b means subscribing to all partitions in all topics/subdirectories under a/b recursively Effectively the concept of topics goes away entirely--you just have partitions/logs and directories. In this respect rather than adding new concepts this new feature would actually just generalizes what we have (which I think is a good thing). -Jay On Mon, Oct 12, 2015 at 6:24 PM, Ashish Singh <asi...@cloudera.com> wrote: > On Mon, Oct 12, 2015 at 5:42 PM, Jay Kreps <j...@confluent.io> wrote: > > > Great. I definitely would strongly favor carrying over user's intuition > > from FS unless we think we need a very different model. The minor details > > like the seperator and namespace term will help with that. > > > > Follow-up question, say I have a layout like > > /chicago-datacenter/user-events/pageviews > > Can I subscribe to > > /chicago-datacenter/user-events > > > Yes, however they will have need a regex like > /chicago-datacenter/user-events/* > > > to get the full firehose of user events from chicago? Can I subscribe to > > /*/user-events > > to get user events originating from all datacenters? > > > Yes, however they will have need a regex like > /chicago-datacenter/user-events/* > Yes > > > > > (Assuming, for now, that these are all in the same cluster...) > > > > Also, just to confirm, it sounds from the proposal like config overrides > > would become fully hierarchical so you can override config at any > directory > > point. This will add complexity in implementation but I think will likely > > be much more operator friendly. > > > Yes, that is the idea. > > > > > There are about a thousand details to discuss in terms of how this would > > impact the metadata request, various zk entries, and various other > aspects, > > but probably it makes sense to first agree on how we would want it to > work > > and then start to dive into how to implement that. > > > Agreed. > > > > > -Jay > > > > On Mon, Oct 12, 2015 at 5:28 PM, Ashish Singh <asi...@cloudera.com> > wrote: > > > > > Hey Jay, thanks for reviewing the proposal. Answers inline. > > > > > > On Mon, Oct 12, 2015 at 10:53 AM, Jay Kreps <j...@confluent.io> wrote: > > > > > > > Hey guys, > > > > > > > > I think this is an important feature and one we've talked about for a > > > > while. I really think trying to invent a new nomenclature is going to > > > make > > > > it hard for people to understand, though. As such I recommend we call > > > > namespaces "directories" and denote them with '/'--this will make the > > > > feature 1000x more understandable to people. > > > > > > Essentially you are suggesting two things here. > > > 1. Use "Directory" instead of "Namespace" as it is more intuitive. I > > agree. > > > 2. Make '/' as delimiter instead of ':'. Fine with me and I agree if we > > > call these directories, '/' is the way to go. > > > > > > I think we should inheret the > > > > semantics of normal unix fs in so far as it makes sense. > > > > > > > > In this approach we get rid of topics entirely, instead we really > just > > > have > > > > partitions which are the equivalent of a file and retain their > numeric > > > > names, and the existing topic concept is just the first directory > level > > > but > > > > we generalize to allow arbitrarily many more levels of nesting. This > > > allows > > > > categorization of data, such as /datacenter1/user-events/page-views/3 > > and > > > > you can subscribe, apply configs or permissions at any level of the > > > > hierarchy. > > > > > > > +1. This actually requires just a minor change to existing proposal, > > i.e., > > > "some:namespace:topic" becomes "some/namespace/topic". > > > > > > > > > > > I'm actually not 100% such what the semantics of accessing data in > > > > differing namespaces is in the current proposal, maybe you can > clarify > > > > Ashish? > > > > > > I will add more info to KIP on this, however I think a client should be > > > able to access data in any namespace as long as following conditions > are > > > satisfied. > > > > > > 1. Namespace, the client is trying to access, exists. > > > 2. The client has sufficient permissions on the namespace for type of > > > operation the client is trying to perform on a topic within that > > namespace. > > > 3. The client has sufficient permissions on the topic for type of > > operation > > > the client is trying to perform on that topic. > > > > > > If we choose to go with what you suggested earlier that just have > > hierarchy > > > of directories, then step 3 will actually be covered in step 2. > > > > > > In the current proposal, consumers will subscribe to a topic in a > > namespace > > > by specifying <namespace>:<topic> as the topic name. They can subscribe > > to > > > topics from multiple namespaces. > > > > > > Let me know if I totally missed your question. > > > > > > Since the point of Kafka is sharing data I think it is really > > > > important that the grouping be just for > > > convenience/permissions/config/etc > > > > and that it remain possible to access multiple directories/namespaces > > > from > > > > the same client. > > > > > > > Totally agree with you. > > > > > > > > > > > -Jay > > > > > > > > On Fri, Oct 9, 2015 at 6:32 PM, Ashish Singh <asi...@cloudera.com> > > > wrote: > > > > > > > > > Hey Guys, > > > > > > > > > > I just created KIP-37 for adding namespaces to Kafka. > > > > > > > > > > KIP-37 > > > > > < > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-37+-+Add+Namespaces+to+Kafka > > > > > > > > > > > tracks the proposal. > > > > > > > > > > The idea is to make Kafka support multi-tenancy via namespaces. > > > > > > > > > > Feedback and comments are welcome. > > > > > > > > > > -- > > > > > > > > > > Regards, > > > > > Ashish > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Regards, > > > Ashish > > > > > > > > > -- > > Regards, > Ashish >
