[
https://issues.apache.org/jira/browse/KUDU-3662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17986858#comment-17986858
]
ASF subversion and git services commented on KUDU-3662:
-------------------------------------------------------
Commit 4c43fdba56d453a13a15cd86c8c3ea7596d073dd in kudu's branch
refs/heads/master from Zoltan Chovan
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=4c43fdba5 ]
KUDU-3662 [3/n] Setup source and sink and tests
- Introduce CustomReplicationRowResultConverter to detect delete
rows on the source side
- Add CustomReplicationOperationMapper to map Flink Row to Kudu
upsert/delete operations
- Wire source and sink using the above in ReplicationEnvProvider
- Set up ReplicationTestBase with separate source and sink Kudu
test harnesses to test replication logic
- Add basic replication tests verifying row count and content equality
Change-Id: I3dd7122933b125b6d4c6dd5dde27a7e8c3f790d4
Reviewed-on: http://gerrit.cloudera.org:8080/23003
Tested-by: Kudu Jenkins
Reviewed-by: Marton Greber <[email protected]>
Reviewed-by: Zoltan Martonka <[email protected]>
> Flink based continuous replication
> ----------------------------------
>
> Key: KUDU-3662
> URL: https://issues.apache.org/jira/browse/KUDU-3662
> Project: Kudu
> Issue Type: New Feature
> Reporter: Marton Greber
> Priority: Major
>
> Goal:
> Implement a Flink job that continuously reads from one Kudu cluster and
> writes to a sink Kudu cluster.
> Prerequisites:
> Previously there existed only a Flink Kudu sink connector. With the release
> of flink-connector-kudu 2.0 we developed a source connector that has a
> continuous unbounded mode, that utilises diffscan to read from Kudu.
> (https://github.com/apache/flink-connector-kudu/pull/8)
> The above prerequisite is now available.
> Design:
> A high level design doc has been already sent out to the mailing list:
> https://docs.google.com/document/d/1oaAn_cOY7aKth0C6MbNXgKU3R-PYols-V4got-_gpDk/edit?usp=sharing
> Development:
> - The Flink based Kudu replication job would live in the Kudu java project.
> (similar how the backup and restore Spark job is)
> - We need to create a Flink job that utilises the Flink Kudu source and sink
> implementations.
> - Provide CLI interface to be able to pipe down all the necessary reader and
> writer configs.
> - Create a table initialiser that can re-create the source table schema and
> partitioning schema on the sink cluster, if desired. (this is a convenience
> feature, the provide easier setup)
> - The Flink Kudu source is at this time missing metrics. In order to avoid
> waiting another release cycle, we can just create a wrapped source inside our
> project. We will contribute those metrics back to the flink-connector-kudu
> repo, and then we can just remove this intermediary logic from the job.
> - Write unit and integration tests.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
