One tactic that might be worth exploring is to rely on the message key to facilitate this.
It would require engineering careful functions for the key which hashes to the partitions for your topic(s). It would also mean that your consumers for the topic would be evaluating the key and discarding messages that aren’t relevant. The only other option I can think of if you are using the high-level API would be finer-grained topics. Best Regards, -Jonathan On Aug 18, 2014, at 9:14 AM, Josh J <joshjd...@gmail.com> wrote: > Is it possible to modify and use the high level consumer so that I can > ignore processing certain partitions? > > > On Mon, Aug 18, 2014 at 5:07 PM, Sharninder <sharnin...@gmail.com> wrote: > >> On Mon, Aug 18, 2014 at 7:27 PM, Josh J <joshjd...@gmail.com> wrote: >> >>>> You can see an example of using the SimpleConsumer here >>> < >>> >>> >> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example >>>> >>> >>> Any suggestions on where in the code to modify the high level producer to >>> support reading from specific partitions ? >>> >>> >> High level producer? I'm assuming you meant to write the high level >> consumer, in which case it isn't possible. The link above, which has an >> example for reading messages off a specific partition, is for the Simple >> consumer, which ironically, is more complex than the high level consumer. >> >> In short, if you have a usecase where you want to read from a specific >> partition, you will need to implement a simple consumer. >> >> -- >> Sharninder >> >> Josh >>> . >>> >>> On Thu, Aug 14, 2014 at 4:27 PM, Neha Narkhede <neha.narkh...@gmail.com> >>> wrote: >>> >>>> You can see an example of using the SimpleConsumer here >>>> < >>>> >>> >> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example >>>>> >>>> . >>>> >>>> >>>> >>>> On Thu, Aug 14, 2014 at 3:23 AM, Sharninder <sharnin...@gmail.com> >>> wrote: >>>> >>>>> Implement the low level "Simple Consumer". >>>>> >>>>> -- >>>>> Sharninder >>>>> >>>>> >>>>> >>>>> On Thu, Aug 14, 2014 at 2:16 PM, Josh J <joshjd...@gmail.com> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Suppose I have N partitions. I would like to have X different >>> consumer >>>>>> threads ( X < N) read from a specified set of partitions. How can I >>>>> achieve >>>>>> this? >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Josh >>>>>> >>>>> >>>> >>> >>
