There was an interesting discussion over in the kafka mailing list that might give you more ideas Roger. Although they don't mention anything about the number of partitions when doing so, anyways maybe it helps.
Renato M. [1] https://www.mail-archive.com/users@kafka.apache.org/msg11976.html 2015-03-19 5:43 GMT+01:00 Roger Hoover <roger.hoo...@gmail.com>: > Thanks, guys. I was also playing around with including partition count and > even the partition key in the topic name. My thought was that topics may > have the same data and number of partitions but only differ by partition > key. After a while, the naming does get crazy (too long and ugly). We > really need a topic metatdata store. > > On Wed, Mar 18, 2015 at 6:21 PM, Chinmay Soman <chinmay.cere...@gmail.com> > wrote: > > > Yeah ! It does seem a bit hackish - but I think this approach promises > less > > config/operation errors. > > > > Although I think some of these checks can be built within Samza - > assuming > > Kafka has a metadata store in the near future - the Samza container can > > validate the #topics against this store. > > > > On Wed, Mar 18, 2015 at 6:16 PM, Chris Riccomini <criccom...@apache.org> > > wrote: > > > > > Hey Chinmay, > > > > > > Cool, this is good feedback. I didn't think I was *that* crazy. :) > > > > > > Cheers, > > > Chris > > > > > > On Wed, Mar 18, 2015 at 6:10 PM, Chinmay Soman < > > chinmay.cere...@gmail.com> > > > wrote: > > > > > > > Thats what we're doing as well - appending partition count to the > kafka > > > > topic name. This actually helps keep track of the #partitions for > each > > > > topic (since Kafka doesn't have a Metadata store yet). > > > > > > > > In case of topic expansion - we actually just resort to creating a > new > > > > topic. Although that is an overhead - the thought process is that > this > > > will > > > > minimize operational errors. Also, this is necessary to do in case > > we're > > > > doing some kind of joins. > > > > > > > > > > > > On Wed, Mar 18, 2015 at 5:59 PM, Jakob Homan <jgho...@gmail.com> > > wrote: > > > > > > > > > On 18 March 2015 at 17:48, Chris Riccomini <criccom...@apache.org> > > > > wrote: > > > > > > One thing I haven't seen, but might be relevant, is including > > > partition > > > > > > counts in the topic. > > > > > > > > > > Yeah, but then if you change the partition count later on, you've > got > > > > > incorrect information forever. Or you need to create a new stream, > > > > > which might be a nice forcing function to make sure your join isn't > > > > > screwed up. There'd need to be something somewhere to enforce that > > > > > though. > > > > > > > > > > > > > > > > > > > > > -- > > > > Thanks and regards > > > > > > > > Chinmay Soman > > > > > > > > > > > > > > > -- > > Thanks and regards > > > > Chinmay Soman > > >
