Created JIRA : https://issues.apache.org/jira/browse/APEXMALHAR-2459
On Wed, Mar 22, 2017 at 4:11 PM, Chaitanya Chebolu < [email protected]> wrote: > Hi All, > > I would like to propose the Kafka Input Operator using 0.10.0.0 > consumer APIs. > > Changes in 0.10.0.0 version of Kafka > ------------------------------------------------- > Message format in 0.10.0 has changed. Messages now include a time-stamp > field and compressed messages include relative offsets. > > Please refer the below link about the changes in Kafka 0.10.0.0 version: > https://kafka.apache.org/documentation/#upgrade_10_breaking. > > Design > --------- > I'm proposing the below design for new Kafka connectors by refactoring > the existing Kafka connector(0.9.* version) > 1) Split up the malhar-kafka to malhar-kafka-common, malhar-kafka_0.9, > malhar-kafka_0.10. Directory structure would be as follows: > malhar-kafka > |--malhar-kafka-common > |--malhar-kafka_0.9 > |--malhar-kafka_0.10 > If possible, will add the connectors for 0.8.x version. > 2) Convert the KafkaConsumerWrapper to AbstractKafkaConsumerWrapper by > adding the abstract methods over the consumer API's. > 3) "malhar-kafka-common" package consists of the abstract classes and by > default it builds over the 0.9.x version of Kafka. > 4) "malhar-kafka-0.*" consists of concrete classes and its corresponding > KafkaConsumerWrapper. > > Please share your thoughts on design. > > Regards, > Chaitanya > > -- > > *Chaitanya* > > Software Engineer > > E: [email protected] | Twitter: @chaithu1403 > > www.datatorrent.com | apex.apache.org > > > -- *Chaitanya* Software Engineer E: [email protected] | Twitter: @chaithu1403 www.datatorrent.com | apex.apache.org
