Thanks, Xin!
These plugins have greatly enriched the RocketMQ community, hope we could
get more feedback from the community about these plugins.
Regards,
yukon
On Mon, Mar 26, 2018 at 10:44 AM, Xin Wang <data.xinw...@gmail.com> wrote:
> Hi folks,
>
> I'm glad to say that the RocketMQ-Flink module has been merged into master.
> https://github.com/apache/rocketmq-externals/tree/master/rocketmq-flink
>
> Here is the state of rocketmq streaming integration:
>ModuleStatus
> * RocketMQ-Storm*Available*
> * RocketMQ-Spark*Available*, *The improvements will be available soon*.
> * RocketMQ-Flink *Available*
> * RocketMQ-Serializer(including JSON, Apache Avro)*Available*, *Not
> merged yet*.
>
> And, some new modules on the road:
> * RocketMQ-SQL https://issues.apache.org/jira/browse/ROCKETMQ-357
> * RocketMQ-Beam(Beam IO)
>
>
> Anybody who is interested in these tasks too, please join me. Let's fight
> together.
>
>
> Thanks,
> Xin Wang
>
>
>
> 2018-01-27 19:35 GMT+08:00 Xin Wang <data.xinw...@gmail.com>:
>
> > Hi devs,
> >
> > I'd like to update the state of RocketMQ Streaming Integration:
> > Now the task for RocketMQ-Flink integration is completed and patch is
> > available here: https://github.com/apache/rocketmq-externals/pull/45
> > Following is the brief changelog:
> >
> >- RocketMQSource - The RocketMQSource is based on RocketMQ pull
> >consumer mode, and provides exactly once reliability guarantees when
> >checkpoints are enabled.
> >Otherwise, the source doesn't provide any reliability guarantees.
> >- RocketMQSink - The RocketMQSink provides at-least-once reliability
> >guarantees when checkpoints are enabled and withBatchFlushOnCheckpoint
> >(true) is set.
> >Otherwise, the sink reliability guarantees depends on rocketmq
> >producer's retry policy, for this case, the messages sending way is
> sync by
> >default,
> >but you can change it by invoking withAsync(true).
> >- KeyValueDeserializationSchema - The main API for deserializing topic
> >and tags is the org.apache.rocketmq.flink.common.serialization.
> >KeyValueDeserializationSchema interface.
> >rocketmq-flink includes general purpose KeyValueDeserializationSchema
> implementations
> >called SimpleKeyValueDeserializationSchema.
> >- KeyValueSerializationSchema - The main API for serializing topic and
> >tags is the org.apache.rocketmq.flink.common.serialization.
> >KeyValueSerializationSchema interface.
> >rocketmq-flink includes general purpose KeyValueSerializationSchema
> implementations
> >called SimpleKeyValueSerializationSchema.
> >- TopicSelector - The main API for selecting topic and tags is the
> >org.apache.rocketmq.flink.common.selector.TopicSelector interface.
> >rocketmq-flink includes general purpose TopicSelector implementations
> >called DefaultTopicSelector and SimpleTopicSelector.
> >- RocketMQFlinkExample - which receive messages from RocketMQ brokers
> >and send messages to broker after processing.
> >
> > Any comments are welcome. And anybody who is also interested in these
> > tasks, please join me. Let's fight together.
> >
> > Thanks,
> > Xin Wang
> >
> >
> > 2018-01-19 21:47 GMT+08:00 yukon <yu...@apache.org>:
> >
> >> Cool, thanks for your outstanding contribution, to help RocketMQ
> community
> >> integrate with other streaming platforms.
> >>
> >> On Fri, Jan 19, 2018 at 7:52 PM, Xin Wang <data.xinw...@gmail.com>
> wrote:
> >>
> >> > Hi devs,
> >> >
> >> > I have finished the improvements for integrating RocketMQ with Apache
> >> > Storm.
> >> > Main changes are as following:
> >> >
> >> >- Upgraded RocketMQ version to 4.2.0 which brings improvements and
> >> new
> >> >features like batch sending
> >> >- Imporved retry policy for RocketMQ consumer push mode to avoid
> data
> >> >loss in some scenes
> >> >- Batch sending supported for bolt and trident state
> >> >- Allow running several consumer instances in one process, that is
> to
> >> >say, different topics in one worker is possible
> >> >
> >> > PR: https://github.com/apache/storm/pull/2518
> >> >
> >> > And I submit the `RocketMQ-Serializer` patch several days ago. This
> >> module
> >> > includes several serialization