[ https://issues.apache.org/jira/browse/FLINK-7673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16177962#comment-16177962 ]
Tzu-Li (Gordon) Tai edited comment on FLINK-7673 at 9/23/17 8:23 PM: --------------------------------------------------------------------- Yes, it would definitely be nice to eventually have a ASL license-free Kinesis connector that we can publish. If we were to reverse engineer the functionality that KCL / KPL provides, the amount of work would vary across the consumer and producer: - For the consumer, we would only need to work out how to perform the record message aggregation. I think KCL uses protobuf to do this magic. It might simply be a matter of migrating some protobuf generated classes to the Kinesis connector. - For the producer, I previously already worked on a ASL-free version here: https://github.com/tzulitai/flink/tree/FLINK-3229-RemoveKclKplRework. Its functionality probably isn't on-par with the current producer which uses KPL and is mostly only WIP code, so we would need to look into that. Robert and I initially did not decide to continue with it because we wanted to wait and see if there may be some possibilities of license changes on the AWS side. Either way, I would be very happy to see some progress on this front. It would make our AWS users much more happier :) was (Author: tzulitai): Yes, it would definitely be nice to eventually have a ASL license-free Kinesis connector that we can publish. If we were to reverse engineer the functionality that KCL / KPL provides, the amount of work would vary across the consumer and producer: - For the consumer, we would only need to work out how to perform the record message aggregation. I think KCL uses protobuf to do this magic. It might simply be a matter of migrating some protobuf generated classes to the Kinesis connector. - For the producer, I previously already worked on a ASL-free version here: https://github.com/tzulitai/flink/tree/FLINK-3229-RemoveKclKplRework. Its functionality probably isn't on-par with the current producer which uses KPL, so we would need to look into that. Robert and I initially did not decide to use it because we wanted to wait and see if there may be some possibilities of license changes on the AWS side. Either way, I would be very happy to see some progress on this front. It would make our AWS users much more happier :) > Flink Kinesis connector - remove dependency on the ASL code > ----------------------------------------------------------- > > Key: FLINK-7673 > URL: https://issues.apache.org/jira/browse/FLINK-7673 > Project: Flink > Issue Type: Improvement > Components: Kinesis Connector > Reporter: Radoslaw Gruchalski > > Currently the Flink Kinesis connector depends on two artifacts which are > available under Amazon Software License code. The artifacts are: > {noformat} > <dependency> > <groupId>com.amazonaws</groupId> > <artifactId>amazon-kinesis-producer</artifactId> > <version>${aws.kinesis-kpl.version}</version> > </dependency> > <dependency> > <groupId>com.amazonaws</groupId> > <artifactId>amazon-kinesis-client</artifactId> > <version>${aws.kinesis-kcl.version}</version> > <!-- > We're excluding the below from the KCL since > we'll only be using the > > com.amazonaws.services.kinesis.clientlibrary.types.UserRecord class, which > will not need these dependencies. > --> > <exclusions> > <exclusion> > <groupId>com.amazonaws</groupId> > > <artifactId>aws-java-sdk-dynamodb</artifactId> > </exclusion> > <exclusion> > <groupId>com.amazonaws</groupId> > > <artifactId>aws-java-sdk-cloudwatch</artifactId> > </exclusion> > </exclusions> > </dependency> > {noformat} > This prevents the connector being published to Maven Central. I would like to > contribute to Flink Kinesis connector by replacing the ASL licensed code with > an implementation using AWS SDK for Java. For that, I'd like to understand > two things: > - what's the correct process for making the contribution (contributor > agreement, etc.) > - what's the minimum functionality the alternative implementation would have > to provide in order to get accepted > There are three classes in the connector's source requiring modification: > - org.apache.flink.streaming.connectors.kinesis.FlinkKinesisProducer > - org.apache.flink.streaming.connectors.kinesis.util.KinesisConfigUtil > - org.apache.flink.streaming.connectors.kinesis.internals.ShardConsumer > Thank you. -- This message was sent by Atlassian JIRA (v6.4.14#64029)