Hi, Tayler, That's a good question. As your pointed out, a large number of topics will put stress on local file directory and ZK. Maybe you can do a bit testing first to see what breaks with a large number of topics. After that, we can look into what needs to be fixed. Making the directory structure hierarchical is a possibility.
Thanks, Jun On Fri, Jul 22, 2011 at 1:23 PM, Taylor Gautier <[email protected]> wrote: > Hi. > > I am thinking to use kafka to send/receive messages for a large number of > topics - order of 100k - 1M. > > It seems that the directory structure used for topics will probably not > work > for this usage. Also, I'm not sure if the in-memory data structures might > suffer - and also it may be problematic for zookeeper. > > One thought I have is to modify the directory structure to be a tree of > directories. Not sure what if anything might need to be done to in-memory > structures or zookeeper info. > > Any thoughts? >
