Jiangjie, Seems I was misunderstood; KafakConsumer.poll after subscribing to topic via "KafkaConsumer.subscribe(String... topics)" does not work, "Unknown api code 11" error, even with both client and broker being latest 0.8.3/trunk. Will try to create a failing test and open bug report.
Kind regards, Stevo Slavic. On Wed, Jul 22, 2015 at 8:36 PM, Jiangjie Qin <j...@linkedin.com.invalid> wrote: > I don't think we have consumer coordinator in 0.8.2 brokers. So > KafkaConsumer in 0.8.3 will only be able to subscribe to partitions > explicitly. Subscribing to a topic won't work with 0.8.2 brokers. > > Jiangjie (Becket) Qin > > On Wed, Jul 22, 2015 at 4:26 AM, Stevo Slavić <ssla...@gmail.com> wrote: > > > I'm getting "Unknown api code 11" even when both client and server are > > 0.8.3/trunk, when "KafkaConsumer.subscribe(String... topics)" is used. > > > > Bug? > > > > Kind regards, > > Stevo Slavic. > > > > On Fri, Apr 24, 2015 at 7:13 PM, Neha Narkhede <n...@confluent.io> > wrote: > > > > > Yes, I was clearly confused :-) > > > > > > On Fri, Apr 24, 2015 at 9:37 AM, Sean Lydon <lydon.s...@gmail.com> > > wrote: > > > > > > > Thanks for the responses. Ewen is correct that I am referring to the > > > > *new* consumer (org.apache.kafka.clients.consumer.KafkaConsumer). > > > > > > > > I am extending the consumer to allow my applications more control > over > > > > committed offsets. I really want to get away from zookeeper (so > using > > > > the offset storage), and re-balancing is something I haven't really > > > > needed to tackle in an automated/seamless way. Either way, I'll hold > > > > off going further down this road until there is more interest. > > > > > > > > @Gwen > > > > I set up a single consumer without partition.assignment.strategy or > > > > rebalance.callback.class. I was unable to subscribe to just a topic > > > > ("Unknown api code 11" on broker), but I could subscribe to a > > > > topicpartition. This makes sense as I would need to handle > re-balance > > > > outside the consumer. Things functioned as expected (well I have an > > > > additional minor fix to code from KAFKA-2121), and the only > exceptions > > > > on broker were due to closing consumers (which I have become > > > > accustomed to). My tests are specific to my extended version of the > > > > consumer, but they basically do a little writing and reading with > > > > different serde classes with application controlled commits (similar > > > > to onSuccess and onFailure after each record, but with tolerance for > > > > out of order acknowledgements). > > > > > > > > If you are interested, here is the patch of the hack against trunk. > > > > > > > > On Thu, Apr 23, 2015 at 10:27 PM, Ewen Cheslack-Postava > > > > <e...@confluent.io> wrote: > > > > > @Neha I think you're mixing up the 0.8.1/0.8.2 updates and the > > > > 0.8.2/0.8.3 > > > > > that's being discussed here? > > > > > > > > > > I think the original question was about using the *new* consumer > > > > ("clients > > > > > consumer") with 0.8.2. Gwen's right, it will use features not even > > > > > implemented in the broker in trunk yet, let alone the 0.8.2. > > > > > > > > > > I don't think the "enable.commit.downgrade" type option, or > > supporting > > > > the > > > > > old protocol with the new consumer at all, makes much sense. You'd > > end > > > up > > > > > with some weird hybrid of simple and high-level consumers -- you > > could > > > > use > > > > > offset storage, but you'd have to manage rebalancing yourself since > > > none > > > > of > > > > > the coordinator support would be there. > > > > > > > > > > > > > > > On Thu, Apr 23, 2015 at 9:22 PM, Neha Narkhede <n...@confluent.io> > > > > wrote: > > > > > > > > > >> My understanding is that ideally the 0.8.3 consumer should work > with > > > an > > > > >> 0.8.2 broker if the offset commit config was set to "zookeeper". > > > > >> > > > > >> The only thing that might not work is offset commit to Kafka, > which > > > > makes > > > > >> sense since the 0.8.2 broker does not support Kafka based offset > > > > >> management. > > > > >> > > > > >> If we broke all kinds of offset commits, then it seems like a > > > > regression, > > > > >> no? > > > > >> > > > > >> On Thu, Apr 23, 2015 at 7:26 PM, Gwen Shapira < > > gshap...@cloudera.com> > > > > >> wrote: > > > > >> > > > > >> > I didn't think 0.8.3 consumer will ever be able to talk to 0.8.2 > > > > >> > broker... there are some essential pieces that are missing in > > 0.8.2 > > > > >> > (Coordinator, Heartbeat, etc). > > > > >> > Maybe I'm missing something. It will be nice if this will work > :) > > > > >> > > > > > >> > Mind sharing what / how you tested? Were there no errors in > broker > > > > >> > logs after your fix? > > > > >> > > > > > >> > On Thu, Apr 23, 2015 at 5:37 PM, Sean Lydon < > lydon.s...@gmail.com > > > > > > > >> wrote: > > > > >> > > Currently the clients consumer (trunk) sends offset commit > > > requests > > > > of > > > > >> > > version 2. The 0.8.2 brokers fail to handle this particular > > > request > > > > >> > > with a: > > > > >> > > > > > > >> > > java.lang.AssertionError: assertion failed: Version 2 is > invalid > > > for > > > > >> > > OffsetCommitRequest. Valid versions are 0 or 1. > > > > >> > > > > > > >> > > I was able to make this work via a forceful downgrade of this > > > > >> > > particular request, but I would like some feedback on whether > a > > > > >> > > "enable.commit.downgrade" configuration would be a tolerable > > > method > > > > to > > > > >> > > allow 0.8.3 consumers to interact with 0.8.2 brokers. I'm > also > > > > >> > > interested in this even being a goal worth pursuing. > > > > >> > > > > > > >> > > Thanks, > > > > >> > > Sean > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> -- > > > > >> Thanks, > > > > >> Neha > > > > >> > > > > > > > > > > > > > > > > > > > > -- > > > > > Thanks, > > > > > Ewen > > > > > > > > > > > > > > > > -- > > > Thanks, > > > Neha > > > > > >