> On Aug. 16, 2016, 8:42 a.m., Balázs Donát Bessenyei wrote: > > flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java, > > line 301 > > <https://reviews.apache.org/r/51112/diff/1/?file=1474463#file1474463line301> > > > > Do you think we could use the Kafka Consumer Configs ( > > https://kafka.apache.org/08/configuration.html#consumerconfigs ) instead of > > the hard-coded 30000-s? > > Grant Henke wrote: > The new consumer doesn't have zookeeper configurations I can leverage. > What configurations should I use? > Tying it to some configuration isn't a bad idea. Though this client isn't > used for long and 30 seconds is pretty liberal. So I am not sure its > neccessary. > > Balázs Donát Bessenyei wrote: > According to https://flume.apache.org/FlumeUserGuide.html#kafka-channel , > there is an "Other Kafka Properties". If it's similar to > https://flume.apache.org/FlumeUserGuide.html#kafka-source , then it should > have "Other Kafka Consumer Properties" which links to > https://kafka.apache.org/08/configuration.html#consumerconfigs -> > zookeeper.session.timeout.ms and zookeeper.connection.timeout.ms. > > I might be mistaken, though > > Grant Henke wrote: > zookeeper.session.timeout.ms and zookeeper.connection.timeout.ms are used > in the old conusmer. The Kafka Channel now now uses the new consumer with > configs described here: > http://kafka.apache.org/documentation.html#newconsumerconfigs > > There is no existing config that I know of that would make sense to > leverage and I am not sure its worth exposing a config of its own for this > migration connection.
I updated the patch to move these value to private constants since I understand the desire to avoid magic values. I did not make them configurable because I can't think of a reason someone would want to change them, and I don't want to introduce a new config (public api) thats deprecated immediatly. - Grant ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51112/#review145846 ----------------------------------------------------------- On Aug. 16, 2016, 7:30 p.m., Grant Henke wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51112/ > ----------------------------------------------------------- > > (Updated Aug. 16, 2016, 7:30 p.m.) > > > Review request for Flume. > > > Bugs: FLUME-2972 > https://issues.apache.org/jira/browse/FLUME-2972 > > > Repository: flume-git > > > Description > ------- > > Offsets tracking the position in Kafka consumers change from using zookeeper > for offset storage to Kafka when moving from 0.8.x to 0.9.x. > FLUME-2823 makes the client change in the Kafka Channel but does not ensure > existing offsets get migrated in order to continue consuming where it left > off. > > This patch adds automated logic on startup to check if Kafka offsets exist, > if not and migration is enabled (by default) then the offsets from Zookeeper > are copied and committed to Kafka. > > > Diffs > ----- > > flume-ng-channels/flume-kafka-channel/pom.xml 587b4b4 > > flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java > 90e3288 > > flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannelConfiguration.java > ccf46d9 > > flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java > b63ac9b > flume-ng-doc/sphinx/FlumeUserGuide.rst fde9ff7 > > Diff: https://reviews.apache.org/r/51112/diff/ > > > Testing > ------- > > Unit tests so far. > > > Thanks, > > Grant Henke > >
