On Thu, Aug 18, 2016 at 2:30 AM, Sijie Guo <[email protected]> wrote: > Jon, > > Sorry for late response. This is a very good question. Comments in line. > > Sijie > > On Monday, August 15, 2016, Jon Derrick <[email protected]> > wrote: > > > Hello all, > > > > I read the distributed log code closely. I found that the DL namespace > is a > > flat namespace. There will be a potential issue if there are a lot of > > streams created under a same namespace. I am very curious what are the > > thoughts behind that. Here are some questions: > > > > - How many streams that a namespace can support? > > > The maximum number of streams we have had for a single namespace is more > than 30k. But yup, you are right. It is limited by the number of children > that a znode can have. > > > > > > > It seems to be bound with > > the limitation on the number of children that a zookeeper znode can have. > > What's the maximum number of logs do you guys have? > > - Why not choose a tree representation? Then it might be easier to > organize > > streams. For example, if I want to use multiple dl streams as > partitions, I > > can just easily organize them together under same znode. > > > We don't want to DL to focus on partitions. We let applications decide how > to partition. So we choose a simple way to start. However, I don't think it > is necessary to be just a flat namespace. You probably already noticed that > there is another namespace implementation to support hierarchy. > > If you do like to support filesystem like namespace, I would suggest adding > a namespace type on metadata binding. So it can support different types of > namespaces. Does that meet your requirements? >
+1 for supporting different types of namespaces. I want to organize a kafka topic in following format: namespace/topic/partitions : storing all the partitions namespace/topic/partitions/N : storing the given partition `N` namespace/topic/subscriptions : storing all the subscriptions namespace/topic/subscriptions/S : storing the information of subscription `S` both `namespace/topic/partitions/N` and `namespace/topic/subscriptions/S` are DL streams. So it would make me easier to manage the streams if I can customize namespace layout. - KN > > > > - Also if it is a tree-like namespace, it might be easier to implement a > > filesystem over the streams. Each file can be backed by one dl stream. In > > that way, I can also use DL as long term storage. > > > > Any thoughts? Appreciate your comments. > > > > > > -- > > - jderrick > > >
