Thanks, Xin! These plugins have greatly enriched the RocketMQ community, hope we could get more feedback from the community about these plugins.
Regards, yukon On Mon, Mar 26, 2018 at 10:44 AM, Xin Wang <data.xinw...@gmail.com> wrote: > Hi folks, > > I'm glad to say that the RocketMQ-Flink module has been merged into master. > https://github.com/apache/rocketmq-externals/tree/master/rocketmq-flink > > Here is the state of rocketmq streaming integration: > Module Status > * RocketMQ-Storm *Available* > * RocketMQ-Spark *Available*, *The improvements will be available soon*. > * RocketMQ-Flink *Available* > * RocketMQ-Serializer(including JSON, Apache Avro) *Available*, *Not > merged yet*. > > And, some new modules on the road: > * RocketMQ-SQL https://issues.apache.org/jira/browse/ROCKETMQ-357 > * RocketMQ-Beam(Beam IO) > > > Anybody who is interested in these tasks too, please join me. Let's fight > together. > > > Thanks, > Xin Wang > > > > 2018-01-27 19:35 GMT+08:00 Xin Wang <data.xinw...@gmail.com>: > > > Hi devs, > > > > I'd like to update the state of RocketMQ Streaming Integration: > > Now the task for RocketMQ-Flink integration is completed and patch is > > available here: https://github.com/apache/rocketmq-externals/pull/45 > > Following is the brief changelog: > > > > - RocketMQSource - The RocketMQSource is based on RocketMQ pull > > consumer mode, and provides exactly once reliability guarantees when > > checkpoints are enabled. > > Otherwise, the source doesn't provide any reliability guarantees. > > - RocketMQSink - The RocketMQSink provides at-least-once reliability > > guarantees when checkpoints are enabled and withBatchFlushOnCheckpoint > > (true) is set. > > Otherwise, the sink reliability guarantees depends on rocketmq > > producer's retry policy, for this case, the messages sending way is > sync by > > default, > > but you can change it by invoking withAsync(true). > > - KeyValueDeserializationSchema - The main API for deserializing topic > > and tags is the org.apache.rocketmq.flink.common.serialization. > > KeyValueDeserializationSchema interface. > > rocketmq-flink includes general purpose KeyValueDeserializationSchema > implementations > > called SimpleKeyValueDeserializationSchema. > > - KeyValueSerializationSchema - The main API for serializing topic and > > tags is the org.apache.rocketmq.flink.common.serialization. > > KeyValueSerializationSchema interface. > > rocketmq-flink includes general purpose KeyValueSerializationSchema > implementations > > called SimpleKeyValueSerializationSchema. > > - TopicSelector - The main API for selecting topic and tags is the > > org.apache.rocketmq.flink.common.selector.TopicSelector interface. > > rocketmq-flink includes general purpose TopicSelector implementations > > called DefaultTopicSelector and SimpleTopicSelector. > > - RocketMQFlinkExample - which receive messages from RocketMQ brokers > > and send messages to broker after processing. > > > > Any comments are welcome. And anybody who is also interested in these > > tasks, please join me. Let's fight together. > > > > Thanks, > > Xin Wang > > > > > > 2018-01-19 21:47 GMT+08:00 yukon <yu...@apache.org>: > > > >> Cool, thanks for your outstanding contribution, to help RocketMQ > community > >> integrate with other streaming platforms. > >> > >> On Fri, Jan 19, 2018 at 7:52 PM, Xin Wang <data.xinw...@gmail.com> > wrote: > >> > >> > Hi devs, > >> > > >> > I have finished the improvements for integrating RocketMQ with Apache > >> > Storm. > >> > Main changes are as following: > >> > > >> > - Upgraded RocketMQ version to 4.2.0 which brings improvements and > >> new > >> > features like batch sending > >> > - Imporved retry policy for RocketMQ consumer push mode to avoid > data > >> > loss in some scenes > >> > - Batch sending supported for bolt and trident state > >> > - Allow running several consumer instances in one process, that is > to > >> > say, different topics in one worker is possible > >> > > >> > PR: https://github.com/apache/storm/pull/2518 > >> > > >> > And I submit the `RocketMQ-Serializer` patch several days ago. This > >> module > >> > includes several serialization formats, especially Apache Avro which I > >> > stated before[1]. > >> > > >> > - Raw String > >> > - JSON > >> > - Avro Generic > >> > - Avro Specified > >> > > >> > PR: https://github.com/apache/rocketmq-externals/pull/42 > >> > > >> > Any comments for these PRs are welcome. BTW here is the state of > >> rocketmq > >> > streaming integration: > >> > Module Status > >> > * RocketMQ-Storm Patch Available > >> > * RocketMQ-Spark To Refactor > >> > * RocketMQ-Flink Patch Available Soon > >> > * RocketMQ-Serializer Patch Available > >> > > >> > Anybody who is also interested in these tasks, please join me. Let's > >> fight > >> > together. > >> > > >> > > >> > [1] https://issues.apache.org/jira/browse/ROCKETMQ-157 > >> > > >> > > >> > Thanks, > >> > > >> > Xin Wang (vesense) > >> > > >> > > > > > > > > -- > > Thanks, > > Xin > > > > > > -- > Thanks, > Xin >
