Renato, Thanks for the link. Some interesting suggests there as well.
On Thu, Mar 19, 2015 at 2:28 AM, Renato Marroquín Mogrovejo < renatoj.marroq...@gmail.com> wrote: > There was an interesting discussion over in the kafka mailing list that > might give you more ideas Roger. > Although they don't mention anything about the number of partitions when > doing so, anyways maybe it helps. > > > Renato M. > > [1] https://www.mail-archive.com/users@kafka.apache.org/msg11976.html > > 2015-03-19 5:43 GMT+01:00 Roger Hoover <roger.hoo...@gmail.com>: > > > Thanks, guys. I was also playing around with including partition count > and > > even the partition key in the topic name. My thought was that topics > may > > have the same data and number of partitions but only differ by partition > > key. After a while, the naming does get crazy (too long and ugly). We > > really need a topic metatdata store. > > > > On Wed, Mar 18, 2015 at 6:21 PM, Chinmay Soman < > chinmay.cere...@gmail.com> > > wrote: > > > > > Yeah ! It does seem a bit hackish - but I think this approach promises > > less > > > config/operation errors. > > > > > > Although I think some of these checks can be built within Samza - > > assuming > > > Kafka has a metadata store in the near future - the Samza container can > > > validate the #topics against this store. > > > > > > On Wed, Mar 18, 2015 at 6:16 PM, Chris Riccomini < > criccom...@apache.org> > > > wrote: > > > > > > > Hey Chinmay, > > > > > > > > Cool, this is good feedback. I didn't think I was *that* crazy. :) > > > > > > > > Cheers, > > > > Chris > > > > > > > > On Wed, Mar 18, 2015 at 6:10 PM, Chinmay Soman < > > > chinmay.cere...@gmail.com> > > > > wrote: > > > > > > > > > Thats what we're doing as well - appending partition count to the > > kafka > > > > > topic name. This actually helps keep track of the #partitions for > > each > > > > > topic (since Kafka doesn't have a Metadata store yet). > > > > > > > > > > In case of topic expansion - we actually just resort to creating a > > new > > > > > topic. Although that is an overhead - the thought process is that > > this > > > > will > > > > > minimize operational errors. Also, this is necessary to do in case > > > we're > > > > > doing some kind of joins. > > > > > > > > > > > > > > > On Wed, Mar 18, 2015 at 5:59 PM, Jakob Homan <jgho...@gmail.com> > > > wrote: > > > > > > > > > > > On 18 March 2015 at 17:48, Chris Riccomini < > criccom...@apache.org> > > > > > wrote: > > > > > > > One thing I haven't seen, but might be relevant, is including > > > > partition > > > > > > > counts in the topic. > > > > > > > > > > > > Yeah, but then if you change the partition count later on, you've > > got > > > > > > incorrect information forever. Or you need to create a new > stream, > > > > > > which might be a nice forcing function to make sure your join > isn't > > > > > > screwed up. There'd need to be something somewhere to enforce > that > > > > > > though. > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Thanks and regards > > > > > > > > > > Chinmay Soman > > > > > > > > > > > > > > > > > > > > > -- > > > Thanks and regards > > > > > > Chinmay Soman > > > > > >
