[
https://issues.apache.org/jira/browse/FLINK-4582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16664305#comment-16664305
]
Devin Thomson commented on FLINK-4582:
--------------------------------------
[~yxu-lyft] [~tzulitai]
Hey guys! Devin from Tinder Engineering here
(https://www.linkedin.com/in/devin-thomson-44a35651/).
I've been following this ticket for a couple of weeks now. We have a strong
need for this at Tinder so I went ahead and built it! Here's a PR:
[https://github.com/tinder-dthomson/flink/pull/3]
I am not a contributor so I had to fork flink. If you want to take a look and
let me know what you think, I'd be glad to contribute this back to the
community!
Also [~yxu-lyft] I don't mean to step on your toes here - if you have a better
solution I am of course happy to use that instead!
- Devin
> Allow FlinkKinesisConsumer to adapt for AWS DynamoDB Streams
> ------------------------------------------------------------
>
> Key: FLINK-4582
> URL: https://issues.apache.org/jira/browse/FLINK-4582
> Project: Flink
> Issue Type: New Feature
> Components: Kinesis Connector, Streaming Connectors
> Reporter: Tzu-Li (Gordon) Tai
> Assignee: Ying Xu
> Priority: Major
>
> AWS DynamoDB is a NoSQL database service that has a CDC-like (change data
> capture) feature called DynamoDB Streams
> (http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html),
> which is a stream feed of item-level table activities.
> The DynamoDB Streams shard abstraction follows that of Kinesis Streams with
> only a slight difference in resharding behaviours, so it is possible to build
> on the internals of our Flink Kinesis Consumer for an exactly-once DynamoDB
> Streams source.
> I propose an API something like this:
> {code}
> DataStream dynamoItemsCdc =
> FlinkKinesisConsumer.asDynamoDBStream(tableNames, schema, config)
> {code}
> The feature adds more connectivity to popular AWS services for Flink, and
> combining what Flink has for exactly-once semantics, out-of-core state
> backends, and queryable state with CDC can have very strong use cases. For
> this feature there should only be an extra dependency to the AWS Java SDK for
> DynamoDB, which has Apache License 2.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
