Hi folks, I'm glad to say that the RocketMQ-Flink module has been merged into master. https://github.com/apache/rocketmq-externals/tree/master/rocketmq-flink
Here is the state of rocketmq streaming integration: Module Status * RocketMQ-Storm *Available* * RocketMQ-Spark *Available*, *The improvements will be available soon*. * RocketMQ-Flink *Available* * RocketMQ-Serializer(including JSON, Apache Avro) *Available*, *Not merged yet*. And, some new modules on the road: * RocketMQ-SQL https://issues.apache.org/jira/browse/ROCKETMQ-357 * RocketMQ-Beam(Beam IO) Anybody who is interested in these tasks too, please join me. Let's fight together. Thanks, Xin Wang 2018-01-27 19:35 GMT+08:00 Xin Wang <data.xinw...@gmail.com>: > Hi devs, > > I'd like to update the state of RocketMQ Streaming Integration: > Now the task for RocketMQ-Flink integration is completed and patch is > available here: https://github.com/apache/rocketmq-externals/pull/45 > Following is the brief changelog: > > - RocketMQSource - The RocketMQSource is based on RocketMQ pull > consumer mode, and provides exactly once reliability guarantees when > checkpoints are enabled. > Otherwise, the source doesn't provide any reliability guarantees. > - RocketMQSink - The RocketMQSink provides at-least-once reliability > guarantees when checkpoints are enabled and withBatchFlushOnCheckpoint > (true) is set. > Otherwise, the sink reliability guarantees depends on rocketmq > producer's retry policy, for this case, the messages sending way is sync by > default, > but you can change it by invoking withAsync(true). > - KeyValueDeserializationSchema - The main API for deserializing topic > and tags is the org.apache.rocketmq.flink.common.serialization. > KeyValueDeserializationSchema interface. > rocketmq-flink includes general purpose KeyValueDeserializationSchema > implementations > called SimpleKeyValueDeserializationSchema. > - KeyValueSerializationSchema - The main API for serializing topic and > tags is the org.apache.rocketmq.flink.common.serialization. > KeyValueSerializationSchema interface. > rocketmq-flink includes general purpose KeyValueSerializationSchema > implementations > called SimpleKeyValueSerializationSchema. > - TopicSelector - The main API for selecting topic and tags is the > org.apache.rocketmq.flink.common.selector.TopicSelector interface. > rocketmq-flink includes general purpose TopicSelector implementations > called DefaultTopicSelector and SimpleTopicSelector. > - RocketMQFlinkExample - which receive messages from RocketMQ brokers > and send messages to broker after processing. > > Any comments are welcome. And anybody who is also interested in these > tasks, please join me. Let's fight together. > > Thanks, > Xin Wang > > > 2018-01-19 21:47 GMT+08:00 yukon <yu...@apache.org>: > >> Cool, thanks for your outstanding contribution, to help RocketMQ community >> integrate with other streaming platforms. >> >> On Fri, Jan 19, 2018 at 7:52 PM, Xin Wang <data.xinw...@gmail.com> wrote: >> >> > Hi devs, >> > >> > I have finished the improvements for integrating RocketMQ with Apache >> > Storm. >> > Main changes are as following: >> > >> > - Upgraded RocketMQ version to 4.2.0 which brings improvements and >> new >> > features like batch sending >> > - Imporved retry policy for RocketMQ consumer push mode to avoid data >> > loss in some scenes >> > - Batch sending supported for bolt and trident state >> > - Allow running several consumer instances in one process, that is to >> > say, different topics in one worker is possible >> > >> > PR: https://github.com/apache/storm/pull/2518 >> > >> > And I submit the `RocketMQ-Serializer` patch several days ago. This >> module >> > includes several serialization formats, especially Apache Avro which I >> > stated before[1]. >> > >> > - Raw String >> > - JSON >> > - Avro Generic >> > - Avro Specified >> > >> > PR: https://github.com/apache/rocketmq-externals/pull/42 >> > >> > Any comments for these PRs are welcome. BTW here is the state of >> rocketmq >> > streaming integration: >> > Module Status >> > * RocketMQ-Storm Patch Available >> > * RocketMQ-Spark To Refactor >> > * RocketMQ-Flink Patch Available Soon >> > * RocketMQ-Serializer Patch Available >> > >> > Anybody who is also interested in these tasks, please join me. Let's >> fight >> > together. >> > >> > >> > [1] https://issues.apache.org/jira/browse/ROCKETMQ-157 >> > >> > >> > Thanks, >> > >> > Xin Wang (vesense) >> > >> > > > > -- > Thanks, > Xin > -- Thanks, Xin
