Re: [VOTE]Release Apache RocketMQ APIs 2.0.0 RC1
+1 binding I checked: 1. The signed key can be found in the KEYs file 2. LICENSE is Apache V2, the notice of Protocol Buffers has been added to the notice of this release. 3. gpg verify ok for both source and binary releases. 4. sha512sum ok for both source and binary releases. Regards, yukon On Fri, Jul 15, 2022 at 11:13 PM aaron ai wrote: > Hello RocketMQ Community, > > This is the vote for 2.0.0 RC1 of Apache RocketMQ APIs. > > RocketMQ APIs use Protocol Buffers version 3 (proto3) as their Interface > Definition Language (IDL) to define the API interface and the structure of > the payload messages. > > The artifacts: > > https://dist.apache.org/repos/dist/dev/rocketmq/2.0.0-rc1 > > The staging repo: > > https://repository.apache.org/content/repositories/orgapacherocketmq-1091/ > > Git tag for the release: > > https://github.com/apache/rocketmq-apis/releases/tag/v2.0 > > https://github.com/apache/rocketmq-apis/releases/tag/rocketmq-proto-2.0.0 > > Hash for the release tag: > > 303962118171075c3a5a8c395822fecfdf492d6f > > Relate Notes: > > https://github.com/apache/rocketmq-apis/releases/tag/v2.0 > > https://github.com/apache/rocketmq-apis/releases/tag/rocketmq-proto-2.0.0 > > The artifacts have been signed with Key : > > 3A11FEBBD64C807A233BF1F27C46C79BD4D29011, which can be found in the keys > file: https://dist.apache.org/repos/dist/dev/rocketmq/KEYS > > Fill in the following: > > [+] check LICENSE, should be Apache V2 > > [+] check NOTICE, should have a notice for third-party dependency if > necessary > > [+] extract the zip and check if the source version is correct > > [+] verify the asc(PGP sign),SHA512 > > > > > > > > The vote will be open for at least 72 hours or until the necessary number > of votes are reached. > > Please vote accordingly: > > > > > [ ] +1 approve > > [ ] +0 no opinion > > [ ] -1 disapprove with the reason > > > > > Thanks > > The Apache RocketMQ Team >
[ANNOUNCE]New Committers of Apache RocketMQ: Lin Shen(shenlin) and Yangkun Ai(aaronai)
Hi Apache RocketMQ Community, The Project Management Committee (PMC) for Apache RocketMQ has invited Lin Shen (apache id: shenlin, github id: 2011shenlin) and Yangkun Ai (apache id: aaronai, github id: aaron-ai) to become committers, and we are pleased to announce that they have accepted. Congrats, guys :-) Best regards, Yukon
Re: [VOTE] RocketMQ-client-go 1.2.0 Release And Graduation
+1 On Fri, Jan 4, 2019 at 2:15 PM Kevin Wang wrote: > +1 Go! > > > > 在 2019年1月4日，下午1:15，Shannon 写道： > > > > Hello RocketMQ Community, > > > > This is the vote for the 1.2.0 release and the graduation of Apache > RocketMQ-client-go. > > > > Github repo: https://github.com/apache/rocketmq-client-go > > > > The version released this time is the first initial golang client of > rocketmq. It is based on the kernel of the CPP client and uses cgo to > encapsulate the API Implementation of C. You can import go package with go > get command. > > The current version provides the following functions: > > 1.support reliable synchronous sending of messages; > > 2.support reliable orderly sending of messages; > > 3.support reliable push consumption model; > > 4.support default cluster consumption; > > 5.support delayed messages; > > 6.support reliable pull consumption model; > > 7.support custom message properties; > > 8.support message Compression. > > 9.support oneway sending of messages; > > The vote will be open for at least 72 hours or until a necessary number > of votes are reached. > > > > Please vote accordingly: > > > > [ ] +1 approve > > [ ] +0 no opinion > > [ ] -1 disapprove with the reason > > > > Thanks, > > The Apache RocketMQ Team > >
Re: [ANNOUNCE] Congrats, CSoC Start !
Hi, Do you have any plan for the project? And we can use slack for communication. We are GMT+8. On Sun, Apr 29, 2018 at 9:32 AM, Ratnasingam Kasthuriraajan < kasthuriraaja...@gmail.com> wrote: > Hi Von, yukon, > > What would be the plan until May 14th? What would be the medium that we > will use for our chats during the project time? Do we have a daily scrum? > Also what about the time zones? I am in GMT+5.30 zone. > Thanks. > > Best Regards > R.Kasthuriraajan. > > > On Tue, Apr 24, 2018 at 7:51 AM, Von Gosling > wrote: > >> Hi, >> >> I am pleased to announce our 3 topics are selected by GSoC Mentor >> Groups[1]. I am also pleasure to mentor our students Sergio Esteves and >> Kasthuriraajan, our pmc member yukon will also help Mayar Mahmoud to finish >> the openwhisk integration. Let’s complete the topics with close teamwork. >> Any guys who want to join the topics, welcome discussion in the dev email >> list:-) >> >> >> >> [1] https://summerofcode.withgoogle.com/dashboard/projects/ >> all/?sp-search=rocketmq >> >> >> Best Regards, >> Von Gosling >> > > > > -- > R.Kasthuriraajan. > Undergraduate student | Department of Computer Science, > Faculty of Science | University of Jaffna. > > LinkedIn: https://www.linkedin.com/in/ratnasingam- > kasthuriraajan-2a6892121/ > <https://www.linkedin.com/in/ratnasingam-kasthuriraajan-2a6892121/> > stackoverflow: https://stackoverflow.com/users/8870269/kasthuriraajan?tab= > profile > GitHub: https://github.com/kasthuriraajan > Medium: https://medium.com/@Kasthuriraajan > >
Re: [ANNOUNCE] Congrats, CSoC Start !
Hi Guys, Congratulations on getting these GSoC projects, though that it's too late. Looking forward to working with you guys this summer. ``` I would like to ask why Yukon isn't listed in the mentors list assigned to my project? ``` I am, but the listed name is `Xinyu Zhou` which is my full name, while `Yukon` is my nickname, sorry for the confusion. ``` My sincere thanks to Yukon too for reviewing my proposal before apply. ``` It's my pleasure. Regards, yukon On Thu, Apr 26, 2018 at 4:42 PM, Von Gosling wrote: > Hi, > > GSoC guys, have you subscribed our dev and user email list, I have > received some unauthorized mails from your addresses. > > Best Regards, > Von Gosling > > 在 2018年4月24日，10:21，Von Gosling 写道： > > I am pleased to announce our 3 topics are selected by GSoC Mentor > Groups[1]. I am also pleasure to mentor our students Sergio Esteves and > Kasthuriraajan, our pmc member yukon will also help Mayar Mahmoud to finish > the openwhisk integration. Let’s complete the topics with close teamwork. > Any guys who want to join the topics, welcome discussion in the dev email > list:-) > > >
Re: The State of RocketMQ Streaming Integration
Thanks, Xin! These plugins have greatly enriched the RocketMQ community, hope we could get more feedback from the community about these plugins. Regards, yukon On Mon, Mar 26, 2018 at 10:44 AM, Xin Wang wrote: > Hi folks, > > I'm glad to say that the RocketMQ-Flink module has been merged into master. > https://github.com/apache/rocketmq-externals/tree/master/rocketmq-flink > > Here is the state of rocketmq streaming integration: >ModuleStatus > * RocketMQ-Storm*Available* > * RocketMQ-Spark*Available*, *The improvements will be available soon*. > * RocketMQ-Flink *Available* > * RocketMQ-Serializer(including JSON, Apache Avro)*Available*, *Not > merged yet*. > > And, some new modules on the road: > * RocketMQ-SQL https://issues.apache.org/jira/browse/ROCKETMQ-357 > * RocketMQ-Beam(Beam IO) > > > Anybody who is interested in these tasks too, please join me. Let's fight > together. > > > Thanks, > Xin Wang > > > > 2018-01-27 19:35 GMT+08:00 Xin Wang : > > > Hi devs, > > > > I'd like to update the state of RocketMQ Streaming Integration: > > Now the task for RocketMQ-Flink integration is completed and patch is > > available here: https://github.com/apache/rocketmq-externals/pull/45 > > Following is the brief changelog: > > > >- RocketMQSource - The RocketMQSource is based on RocketMQ pull > >consumer mode, and provides exactly once reliability guarantees when > >checkpoints are enabled. > >Otherwise, the source doesn't provide any reliability guarantees. > >- RocketMQSink - The RocketMQSink provides at-least-once reliability > >guarantees when checkpoints are enabled and withBatchFlushOnCheckpoint > >(true) is set. > >Otherwise, the sink reliability guarantees depends on rocketmq > >producer's retry policy, for this case, the messages sending way is > sync by > >default, > >but you can change it by invoking withAsync(true). > >- KeyValueDeserializationSchema - The main API for deserializing topic > >and tags is the org.apache.rocketmq.flink.common.serialization. > >KeyValueDeserializationSchema interface. > >rocketmq-flink includes general purpose KeyValueDeserializationSchema > implementations > >called SimpleKeyValueDeserializationSchema. > >- KeyValueSerializationSchema - The main API for serializing topic and > >tags is the org.apache.rocketmq.flink.common.serialization. > >KeyValueSerializationSchema interface. > >rocketmq-flink includes general purpose KeyValueSerializationSchema > implementations > >called SimpleKeyValueSerializationSchema. > >- TopicSelector - The main API for selecting topic and tags is the > >org.apache.rocketmq.flink.common.selector.TopicSelector interface. > >rocketmq-flink includes general purpose TopicSelector implementations > >called DefaultTopicSelector and SimpleTopicSelector. > > - RocketMQFlinkExample - which receive messages from RocketMQ brokers > >and send messages to broker after processing. > > > > Any comments are welcome. And anybody who is also interested in these > > tasks, please join me. Let's fight together. > > > > Thanks, > > Xin Wang > > > > > > 2018-01-19 21:47 GMT+08:00 yukon : > > > >> Cool, thanks for your outstanding contribution, to help RocketMQ > community > >> integrate with other streaming platforms. > >> > >> On Fri, Jan 19, 2018 at 7:52 PM, Xin Wang > wrote: > >> > >> > Hi devs, > >> > > >> > I have finished the improvements for integrating RocketMQ with Apache > >> > Storm. > >> > Main changes are as following: > >> > > >> >- Upgraded RocketMQ version to 4.2.0 which brings improvements and > >> new > >> >features like batch sending > >> >- Imporved retry policy for RocketMQ consumer push mode to avoid > data > >> >loss in some scenes > >> >- Batch sending supported for bolt and trident state > >> >- Allow running several consumer instances in one process, that is > to > >> >say, different topics in one worker is possible > >> > > >> > PR: https://github.com/apache/storm/pull/2518 > >> > > >> > And I submit the `RocketMQ-Serializer` patch several days ago. This > >> module > >> > includes several serialization formats, especially Apache Avro which I > >> > stated before[1]. > >> > &g
Re: 投稿 RocketMQ Master宕机后重启出现的严重问题分析
Hi,
非常欢迎撰写RocketMQ相关的技术文章，建议放到Google DOC上并共享出来方便大家进行Comment。
Regards,
yukon
2018-04-02 12:06 GMT+08:00 404828407 <404828...@qq.com>:
>
> RocketMQ Master宕机后重启出现的严重问题分析
>
> 作者：卢松
>
>
> 以前发现RocketMQ的master机器宕机时会出现很多诡异的问题，所以最近做了一系列实验来试图找到问题的原因，
> 也是想发现风险点。没想到发现了一个更严重的问题，下面会详细的说明下，另外出现问题的原因属于猜测，望高人看到后能指点下。
>
>
> *试验环境准备：*
>
> 1.RocketMQ版本：阿里的3.2.6版本
>
> 2.集群包含2组broker，每组broker都是1master + 1slave，都设置成SYNC_MASTER、ASYNC_FLUSH模式。
>
> 3.每个broker上已经存储过好几个commitlog文件，并且已经有commitlog文件删除过（数据过期了）。
>
>
> *测试步骤：*
>
> 1.尽量多的消费组、尽量多的消费者（10个以上）开始消费多个topic，消费者自始至终不停机。
>
> 2.在消费者消费时，突然把其中一组broker（姑且命名为BrokerA吧）的master断电（*注意是断电或者硬盘损坏、不是kill -9*）。
>
> 3.过一段时间（半小时左右），重启宕机的这个master。
>
>
> *观察现象，记录事后的现象及疑问：*
>
> 1.BrokerA的slave机器一直存活着，master宕机后，消费者新的消费进度会存在slave上面。
>
> 2.*宕机后，拉取BrokerA的master中的数据文件，文件没有损坏。*
>
> 3.重启master后，消费客户端会从很久以前的queue offset开始拉取消息，为啥会去取到很久以前的queueOffset？
>
> 4.重启master后，这个master出现大量如下异常，异常日志达到10分钟每G的量：
>
> {
>
> "timestamp": "2017-12-28T00:18:48.620+0800",
>
> "level": "WARN",
>
> "thread": "FlushConsumeQueueService",
>
> "logger": "RocketmqStoreError",
>
> "message": "findMapedFileByOffset offset not matched, request Offset:
> 0, index: -6, mapedFileSize: 600, mapedFiles count: 2, StackTrace:
> \n\tjava.lang.Thread.getStackTrace(Thread.java:
> 1567)\n\tcom.alibaba.rocketmq.common.UtilAll.currentStackTrace(UtilAll.
> java:59)\n\tcom.alibaba.rocketmq.store.MapedFileQueue.
> findMapedFileByOffset(MapedFileQueue.java:467)\n\
> tcom.alibaba.rocketmq.store.MapedFileQueue.commit(
> MapedFileQueue.java:442)\n\tcom.alibaba.rocketmq.store.
> ConsumeQueue.commit(ConsumeQueue.java:345)\n\tcom.alibaba.rocketmq.store.
> DefaultMessageStore$FlushConsumeQueueService.doFlush(DefaultMessageStore.
> java:1459)\n\tcom.alibaba.rocketmq.store.DefaultMessageStore$
> FlushConsumeQueueService.run(DefaultMessageStore.java:1479)
> \n\tjava.lang.Thread.run(Thread.java:722)"
>
> }
>
>
> 5.另外还有大量的这种异常，这类异常显示大量消费客户端去拉取很久以前的消息。
>
> {
>
> "timestamp": "2017-12-28T00:39:01.748+0800",
>
> "level": "WARN",
>
> "thread": "PullMessageThread_15",
>
> "logger": "RocketmqStoreError",
>
> "message": "findMapedFileByOffset offset not matched, request Offset:
> 19382879484, index: -7, mapedFileSize: 1073741824 <010%207374%201824>,
> mapedFiles count: 2, StackTrace: \n\tjava.lang.Thread.
> getStackTrace(Thread.java:1567)\n\tcom.alibaba.rocketmq.common.UtilAll.
> currentStackTrace(UtilAll.java:59)\n\tcom.alibaba.
> rocketmq.store.MapedFileQueue.findMapedFileByOffset(
> MapedFileQueue.java:467)\n\tcom.alibaba.rocketmq.store.
> CommitLog.getMessage(*CommitLog*.java:664)\n\tcom.alibaba.rocketmq.store.
> DefaultMessageStore.getMessage(*DefaultMessageStore*
> .java:546)\n\tcom.alibaba.rocketmq.broker.processor.*PullMessageProcessor*
> .processRequest(PullMessageProcessor.java:283)\n\tcom.alibaba.rocketmq.
> broker.processor.PullMessageProcessor.processRequest(
> *PullMessageProcessor*.java:84)\n\tcom.alibaba.rocketmq.remoting.netty.
> NettyRemotingAbstract$1.run(NettyRemotingAbstract.java:
> 172)\n\tjava.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:471)\n\tjava.util.concurrent.FutureTask$Sync.innerRun(
> FutureTask.java:334)\n\tjava.util.concurrent.FutureTask.
> run(FutureTask.java:166)\n\tjava.util.concurrent.
> ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\
> n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)\n\tjava.lang.Thread.run(Thread.java:722)"
>
> }
>
>
>
> 这种现象的原因分析过程：
>
> 首先第二类异常出在PullMessageProcessor的拉取消息过程中，消费客户端不停拉取消息时，
> PullMessageProcessor会调用DefaultMessageStore.getMessage()，而在getMessage()中，更具体的是在CommitLog的public
> SelectMapedBufferResult getMessage(final long offset, final int size) 方法中，
> 这个方法会调用MapedFileQueue的方法findMapedFileByOffset(final long offset, final
> boolean returnFirstOnNotFound)，具体看下图：
>
>
>
>
>
> 从代码上看，出现这个异常原因就是DefaultMessageStore.getMessage()中offsetPy拿到的值过小，
> 这个值实际上是从ConsumeQueue的文件中读到的单条消息的CommitlogOffset，
> 说明bufferConsumeQueue这个对象拿到的是老的数据。如下图：
>
>
>
> bufferConsumeQueue对象又是根据offset定位的，offset是每一个queue的queueOffset，
> 这个offset实际上是消费端拉取消息时传递过来的，它来自于两种地方，一是消费端自己拉取的消费进度，另外一种是消费端每次拉取消息，
> 拉取不到后拿到的nextBeginOffset参数。如下图：
>
>
>
> 所以根源就出在这个nextBeginOffset的计算逻辑上，为什么nextBeginOffset会计算出过小的值，
> 导致消费端一直在消费过小的queueOffset对应的消息？
>
>
> 我们先不去查这个问题的原因，就按照这个逻辑走下去，看看会发生什么情况。还是回到DefaultMessageStore.getMessage()中，当SelectMapedBufferResul
Re: 投稿 RocketMQ中新消费组上线时有什么风险？
Hi, 非常欢迎撰写RocketMQ相关的技术文章，建议放到Google DOC上并共享出来方便大家进行Comment。 Regards, yukon 2018-04-02 12:09 GMT+08:00 404828407 <404828...@qq.com>: > RocketMQ中新消费组上线时有什么风险？ > 作者：卢松 > > > > > 以前碰到过一个问题，一个新消费组上线时和我们期望的消费行为有偏差，今天专门研究下这个问题，看看是怎么回事。 > > > 在RocketMQ中，假如一个新消费组订阅了几个topic，按正常人或者正常业务的期望，新消费组应该从订阅topic的最后一个消息开始消费， > *但是实际情形不是如此，有时候新消费组会从这些topic的开头开始消费。这就是新消费组上线的风险点。* > > > 这对业务来说就有风险，因为消费端需要关心要不要处理以前的消息。或者以前的消息非常多，都还没有删除，消费端要处理多久才能处理完成。再或者， > 我的消费者处理不了以前的那些老的消息，处理时都出错，这该怎么办？ > > > 下面我们来分析下这个问题的来龙去脉，按照1,2,3,4...来说明前因后果及解决方案。 > > > *（一）消费端配置。*消费端集群消费时，消费端的默认配置是从topic的最后offset开始消费。 > 具体配置代码在DefaultMQPullConsumerImpl的consumeFromWhere()中： > > > > 而CONSUME_FROM_LAST_OFFSET的含义是“一个新的订阅组第一次启动从队列的最后位置开始消费”，RocketMQ > 3.2.6版本的代码注释中清晰的说明了，但是实际表现却不是如此。 > > > > *（二），消费端拉取消息的位置计算逻辑。*消费端拉取消息是按照topic下的queue来进行主动拉取的， > 最关键的是这个拉取的位置offset是怎么计算出来的。消费端拉取消息位置的触发点在RebalanceImpl的up > dateProcessQueueTableInRebalance()中，如图： > > > > 对于集群消费者，真正实现计算拉取消息位置的逻辑是在 > *RebalancePushImpl的computePullFromWhere()方法中，这个方法在消费客户端逻辑中是非常重要的*， > 经常出现问题都是这段逻辑导致的。虽然这段代码没有bug，但是要深刻理解这段代码才能避免各种问题的出现。以后的文章中，会经常提到这块的实现逻辑。 > > > > *（三）computePullFromWhere()的集群实现*。computePullFromWhere()是计算不同配置时的拉取offset， > 我们只关心CONSUME_FROM_LAST_OFFSET时的实现。如下图箭头处，首先去broker端拉取某个queue的消费进度信息： > > > > offsetStore.readOffset(mq, ReadOffsetType.READ_FROM_STORE) > 这段实际上执行的是RemoteBrokerOffsetStore的readOffset()， 如下图红框中的逻辑： > > > > 其中long brokerOffset = this.fetchConsumeOffsetFromBroker(mq)的实现如下图， > 主要是通过MQClientAPIImpl的queryConsumerOffset()去broker上拿某个queue的消费进度， > 发送的是requestCode是QUERY_CONSUMER_OFFSET的请求： > > > > *（四），broker计算consumerOffser的逻辑。*broker收到上面消费客户端的请求， > 是在ClientManageProcessor的queryConsumerOffset(ChannelHandlerContext ctx, > RemotingCommand request)中做处理的： > > > > 当一个新消费组上线时，会走到订阅组不存在的情况，然后计算当前queue的minOffset，而getMessageStore(). > checkInDiskByConsumeOffset( > > requestHeader.getTopic(),requestHeader.getQueueId(), 0) > > 一般都是返回false，这个方法含义是检查当前queue的第一条消息是否在磁盘中，而不在内存中。 > > > > *所以这个api的结果是，当minOffset=0时，返回offset=0，当minOffset>0时返回QUERY_NOT_FOUND结果，消费端拿到这个结果时会抛出MQBrokerException异常。* > > > broker源代码的注释中也写的很清楚了，“ > > > *订阅组不存在情况下，如果这个队列的消息最小Offset是0，则表示这个Topic上线时间不长，服务器堆积的数据也不多，那么这个订阅组就从0开始消费。尤其对于Topic队列数动态扩容时，必须要从0开始消费。* > > ” > > *（五），消费客户端处理broker返回的consumerOffset，查清问题根源。*RemoteBrokerOffsetStor > e的readOffset()中，*当minOffset为0时，这个方法返回0；当minOffset大于0时，这个方法返回-1* > 。中间的分析过程省略了，不然贴出来的代码会更多。如下图实现： > > > > 上述方法返回的值赋值给*lastOffset ，*继续往上回到 > *RebalancePushImpl的computePullFromWhere()中，*当*lastOffset = > 0时，会返回0；当lastOffset = -1时，将会返回queue的maxOffset。* > > > > *这段逻辑对新消费组的意义就是：* > *如果订阅的queue不是从0开始的（minOffset大于0，已经删除过数据了），那么消费端将从maxOffset开始消费，即从最新位置开始消费；如果订阅的queue是从0开始的（minOffset等于0，没有删除过数据），那么消费端将从0开始消费这个queue。* > > > *这就是风险点，和我们最初的期望，每次都从最后位置消费消息有偏差！！！* > > > *（六）RocketMQ为什么这么设计？设计是否合理？* > > > 我们先去查看下Apache RocketMQ 4.2.0中broker端的实现，因为主要实现点还是在broker端。 > 虽然计算消费位置offset的逻辑已经挪到了这个类中： > > > *org.apache.rocketmq.broker.processor.ConsumerManageProcessor，但是实现逻辑是没有任何改变的。* > > > 既然Apache版本都没有做任何改动，说明这不是个bug，就是这么设计的。下面我们来分析下 > *为什么一个queue的minOffset为0时，消费端要从0开始消费这个queue上消息，只有这种情况超出了正常的预期。* > > > 我们做个假设，假设新消费组上线时，都是从queue的maxOffset开始消费消息。又如果一个topic在一个broker上面有4个queue， > 新消费组上线后，开始从这四个queue的最后位置消费消息，这时我突然扩容这个topic到8个queue， > 那么消费端去namesrv上拿到这8个queue的信息需要一个心跳周期，按默认配置是30秒左右。这个心跳周期内， > 新扩展的queue上完全可能有新消息进来。 > > > 当消费端拿到4个新扩展queue的信息后，去broker端拉取消息时， > *broker还是把这4个扩容queue当作新queue来处理的。按照我们的假设，最终消费端会从这4个新queue的maxOffset开始消费。* > *这就有可能丢失了这4个扩容queue的前面一些消息，有可能会很多消息，而这些消息完全是在新消费组上线后发送出来的！！* > > > *有消息漏消费了！这就是为什么新消费组不能都是从maxOffset开始消费的。* > > > 这样原因就清楚了，RocketMQ的设计是合理的，导致了重复消费是不可避免的，但是风险是巨大的。*这也体现了RocketMQ的一个重大设计原则：* > *宁可重复消费无数消息，也绝不漏掉一条消息。*就跟国民党当年喊得 “宁可错杀千人，也绝不放过一个” 一样。 > > > *（七），解决方案*。我订阅的topic上面可能有几十万条消息没有删除过，难道新上线消费组时，几十万条消息要重新消费吗？ > > > 目前我想到的有两种解决方案，1.可以使用mqadmin中的resetOffsetByTime命令来跳过某一个时间点之前的消息。 > 2.新消费组的消费者启动时，自己去过滤老的消息或者根据时间忽略以前的消息。当然肯定还有其他的解决方案，诸位看官可以思考下。 > > > *（八）最终总结下。*新消费组上线时还是要处理好历史消息的，无论怎样处理，要提前做好准备。有可能消费到大量历史消息， > 这是RocketMQ的本身机制导致的，它的配置有更深层的含义。 > > > 本篇结束，请听下回分解。 >
Re: How to get message from DLQ?
Hi, The dead letter queue isn't expected to consume from it, but you could change the permission by `MQAdmin` and subscribe the DLQ topic manually if you want. As for your another question `Why the new topic name is not "%DLQ%" + oldTopicName ?`, obviously, a topic may have many subscriptions and the DLQ is related to the consumer group, not the topic. Regards, yukon On Mon, Mar 26, 2018 at 12:05 PM, 那~ <3097904...@qq.com> wrote: > > Hey guys, > > Help us with one hand please! When you have some free time if you touch > <http://python.jobbole.com/88940/>... > I'm analysising a web page: > <http://blog.mrriddler.com/2017/02/10/%E8%AE%A1%E7%AE%97%E6%9C%BA%E4%BD%93%E7%B3%BB-%E7%BC%96%E8%AF%91%E4%BD%93%E7%B3%BB%E6%BC%AB%E6%B8%B8/#menu> > try your best to open the cover version of DLQ > <http://www.cnblogs.com/yuxc/archive/2012/02/09/2344474.html> messages > <http://blog.mrriddler.com/2017/02/10/%E8%AE%A1%E7%AE%97%E6%9C%BA%E4%BD%93%E7%B3%BB-%E7%BC%96%E8%AF%91%E4%BD%93%E7%B3%BB%E6%BC%AB%E6%B8%B8/#menu> > > > <http://blog.mrriddler.com/2017/02/10/%E8%AE%A1%E7%AE%97%E6%9C%BA%E4%BD%93%E7%B3%BB-%E7%BC%96%E8%AF%91%E4%BD%93%E7%B3%BB%E6%BC%AB%E6%B8%B8/#menu> > >>> > > Smile more often, then start a fresh new mind with re-thinking，:p) > > Thanks & best regards, > Sebastien. > > -- 原始邮件 -- > *发件人:* "张桂杰"; > *发送时间:* 2018年3月24日(星期六) 晚上7:10 > *收件人:* "users"; > *主题:* How to get message from DLQ? > > Hi: > > My Question is about DLQ. > > We know that a message would be send to A DLQ if the consumer can not > handle the message for 1+ maxReConsumeTimes. So, How can I get the message > from the DLQ? > > I find that if the message is send to DLQ , A topic named > "%DLQ%+consumerGroup will be created with permission 'WRITE'(pem=2). (Why > the new topic name is not "%DLQ%" + oldTopicName ? ) > > When I try to subscribe a topic with the name above, I can not get any > message.After a debug, I found the reason is I have no read permission for > this special topic. > > > > btw: I started a namesrv and a master broker in my macbook. > >
Re: rocketmq性能提升相关问题咨询
Hi， 第一个问题，确实这里涉及到了一次拷包，这里如果不聚合的话，写真正的File的IO次数会更频繁的，对于你说的Body非常的的情况下，这种情况瓶颈一般在磁盘上，但这里有优化的余地，可以节约点CPU。 第二个问题，现在主从同步双写的话确实会因为线程阻塞导致性能受到影响，我们也正在改造成全异步的形式。 祝好 2018-03-13 11:56 GMT+08:00 李煜洲 : > hi，大家好。我是美团基础架构部的李煜洲，最近在对rocketmq做一些性能方面的测试，在阅读代码的时候发现两个问题，希望和大家讨论一下 > 1、AppendMessageResult doAppend函数，作用是把具体的消息格式化并刷到Commitlog的byteb > uffer里面，但是感觉处理逻辑有些性能损耗，我看代码是先把message的消息内容以及等等一些信息统一写 > 到名为msgStoreItemMemory的bytebuffer里面，然后再把msgStoreItemMemory刷到底层comm > itLog的bytebuffer里面，感觉如果我的单条消息的body非常大的话，反复拷贝来拷贝去带来的性能开销还是蛮大的 > 2、发送消息后处理落盘方式和主从同步的代码部分，感觉这部分完全可以异步化的，但是我看代码使用了wait和notify的方式进行了， > rocketmq也是用的reactor网络模型，在真实网络环境下，主从同步不及时是比较容易出现的问题，一旦用了wait+notify > 意味着上面的处理线程会被堵住，整体的吞吐就上不去了，周末对rocketmq做性能测试，主要耗时点有一个就是加锁解锁，感觉和这个有很大关系 >
[ANNOUNCE] Github Issue has been enabled for Apache RocketMQ
Hi all, We have enabled Github Issue for RocketMQ, and the JIRA system will be disabled soon, please use Github Issue for the new bug report and feature request. Please kindly note that the issue tracker is **ONLY** used for bug report and feature request. Any question or RocketMQ proposal please use our mailing lists[1]. More details please refer to our issue template[2]. Welcome to help us migrate existing JIRA issues to Github Issue, we really appreciate that. [1]. http://rocketmq.apache.org/about/contact/ [2]. https://github.com/apache/rocketmq/blob/master/.github/ISSUE_TEMPLATE.md Regards, yukon
Re: Active / Passive in WAN
Hi, Thanks for your interests in RocketMQ. Let's start with the ordered message of RocketMQ. Each topic has multiple message queues spread over multiple broker server nodes, messages in the same queue can be consumed in order if you use OrderConsumer. The concept message queue is similar to the known partition, and it's a FIFO queue. By default, a topic has 8 queues in each broker node, the producer client is responsible for sending a kind of messages to the fixed partition with a sharding key. For example, The chat messages of the same chat channel should be sent to the same message queue in order to consume them orderly. Obviously, if we adjust the message queue number to 1 of a topic, all the messages sent to the topic can be consumed in order. We have an example shows how to use ordered message[1]. In the example, we use `OrderId` as sharding key and modular hashing to select a message queue. Now, back to your question, we assume that the both BrokerA in site A and BrokerB in site B both have TopicX. When site A is active, all the messages are sent to BrokerA and consumed from BrokerA, TopicX in passive BrokerB has no read or write perm. In switch stage: 1. Firstly, set TopicX perm in BrokerB to write-only, set TopicX perm in BrokerA to read-only. Now, all the new messages are sent to the BrokerB but can't be consumed yet. 2. Then, set TopicX perm in BrokerB to read-write after the messages already queued in BrokerA consumed, then disable the perm in BrokerA. Now, BrokerA is passive. All the operations can be finished by the MQAdmin tool, please have a try and let me know if you have any other question. [1]. http://rocketmq.apache.org/docs/order-example/ Regards, yukon On Thu, Feb 1, 2018 at 5:56 PM, Sebastien Briquet wrote: > Hi, > > We have the need for message queuing where the message order is the most > important point in our use case. RocketMQ seems the best choices here and > we will give it a try in the coming weeks (at my daily work). > > However, I need to be able to answer a question first: How does it work in > a active/passive multi-site environment? > > For instance: I have a cluster located in a site A, which is active. The > site B is passive until a switch is decided. So between the time the site A > is still active (because of message already queued) and the site B becomes > active and start queuing new messages, how can I guarantee messages are > ordered between sites? > > I cannot find any documentation about this specifically (or I didn't > understood the existing one :p) > > Many thanks in advance! > Sebastien. > >
Re: RocketMQ offset getting difference between broker and consumer
Hi, There are two offsets for a queue/partition: * broker min offset: the minimum message offset in the current broker, which points to the earliest message. * broker max offset: the maximum message offset in the current broker, which points to the newest message. As for consumer, should concern the consumer offset(min offset<= consumer offset <= max offset). The consumer offset will be updated automatically according to the consumer consumed progress if you are using the PushConsumer, and the consumer offset can be changed easily through MQAdmin tool. If you noticed that the consumer offset always is lower than broker max offset, you should keep a watchful eye on your consume logic, check whether it's blocked by some remoting calls. It's worth noting that the consumer offset is not accurate and real-time if you are using TAG/SQL to filter the message. Let me know if you have any other questions. Regards, yukon On Mon, Jan 22, 2018 at 11:07 PM, 智齿 wrote: > > Hi Sir, > I'm a new guy who just join the RocketMQ community. > I have no idea what happened before I came. > > Then, below what I said is my personal advice: > "If a foreign guy come to my home and say"Hello，RocketMQ", I'll feel > happy:), at least no personal emotion； > But if this man come to my bed room without knock the door. I may have > emotion" :( > We're human beings, sometimes we may face someone rude/something bad: the > "broker"； we may have uncomfortable sences. > but the consumer with gentle actions will help us make our product better, > this is a CN tradition. > As the EN saying:"Customer is the god". > > We're friends in community. This is a deal so that we can communicate. > > Thanks a lot for your knidly advice. > Although a little guy in community I'll try my best to help you. > I can see the mail. Thanks for your reply. No need send a private e-mail > to me, if your error didn't happen now. > > Any questions you can feel free to seek services from me or the other > community > guys. > If there is somewhere special make you feel confused, my apology. > > Regards > JARVIS > > -- 原始邮件 -- > *发件人:* "sachin.shashi";; > *发送时间:* 2018年1月22日(星期一) 晚上10:59 > *收件人:* "users-help";"users" s...@rocketmq.apache.org>;"issues"; > *主题:* RocketMQ offset getting difference between broker and consumer > > Hi all, > Recently we observed that, RocketMQ offset getting difference between > broker offset and consumer offset,e.g. > I have created topic with 4 partitions, whenever message coming to > partition whose broker offset not matches to consumer offset messages not > getting picked up, > Not sure why getting difference between both offset, > Kindly suggest what could be cause for this difference and what needs to > be done so that it should not differ. > > Regards > SACHIN >
Re: need help on RocketMQ error
I guess it’s a bug in older client, update your SDK to resolve this issue. Sachin Singh 于2018年1月22日 周一下午1:51写道： > Hi All, > I am getting error in RocketMQ log, I need help what is the cause, checked > permission are correct for deployed wars. > > Regards > SACHIN >
[ANNOUNCE] Release Apache RocketMQ 4.2.0
Hi all, Apache RocketMQ is a distributed messaging and streaming platform with low latency, high performance and reliability, trillion-level capacity and flexible scalability. The Apache RocketMQ community would like to announce the release of Apache RocketMQ 4.2.0. This release supports some new features: 1. Support transportation layer security 2. Support log4j2 in client 3. Support more flexible flow control mechanism in client Also, fixed some known bugs and some performance issues. More details regarding Apache RocketMQ 4.2.0 can be found at: https://rocketmq.apache.org/ The release artifacts can be downloaded here: https://dist.apache.org/repos/dist/release/rocketmq/4.2.0 The release notes can be found here: https://rocketmq.apache.org/release_notes/release-notes-4.2.0/ Thanks, The Apache RocketMQ Team
Re: [STARTUP]Apache RocketMQ Shenzhen Meetup
Hi, I would like to share a topic `Operations Strategy in Apache RocketMQ` in this meetup, hope this topic could help users build a good understanding of their cluster. Let me know if you guys are interested in this topic -:) Regards, yukon On Mon, Nov 27, 2017 at 4:36 PM, Von Gosling wrote: > Hi, > > Sorry, update the meetup time, GMT+8 TimeZone[1], Dec. 16 pm., 2017:-). > > > [1] https://time.artjoey.com/ > > Von Gosling > > > 在 2017年11月27日，16:25，Xin Wang 写道： > > Hi Von, > Is the time correct? 2018?? - Xin > > 2017-11-27 16:15 GMT+08:00 Von Gosling : > >> Dear Apache RocketMQ Community, >> >> I am excited to call for the second meetup in Shenzhen, China at Dec. 16 >> pm., 2018. Please have a look at your schedule, and consider submitting a >> talk to the community. The overarching message about this meetup is >> “Apache RocketMQ: Messaging and Streaming’s Technology Today“, and we wish >> to highlight the fact that Apache RocketMQ is where tomorrow is invented - >> you are the ones inventing it. >> >> >> For further updated about this Meetup, follow us on Twitter, >> @ApacheRocketMQ, or contact me - vongosl...@apache.org - with any >> questions or concerns. >> >> >> >> Thanks! >> Von Gosling > > > > > -- > Thanks, > Xin > > >
[INFO] Announcing Apache RocketMQ as a TLP
Hi folks, Apache RocketMQ project has been graduated on 2017-09-20, all the graduation common tasks are mostly done. RocketMQ couldn't graduate smoothly without everyone's efforts. Congratulations and thanks, everybody. There are some details need our attention: ## RocketMQ Repos All the repos of RocketMQ have been moved to Gitbox, so primary is on Github now. Below are the remote URLs: * apache/rocketmq: https://github.com/apache/rocketmq.git, g...@github.com: apache/rocketmq.git * apache/rocketmq-site: https://github.com/apache/rocketmq-site.git , g...@github.com:apache/rocketmq-site.git * apache/rocketmq-externals: https://github.com/apache/ rocketmq-externals.git, g...@github.com:apache/rocketmq-externals.git The old remote URLs(git-wip-us.apache.org) are abandoned, please be sure to change your remotes on your local repos to Github since they have been removed from git-wip. For contributors, welcome star the main repo[1] of RocketMQ, the community will really appreciate your contributions. For committers, you are recommended to link ASF and Github IDs as well as setup 2factor auth on Github[2], then you could manage pull requests through GitHub dashboard. ## PMC and Committer Currently, RocketMQ has 10 PMC members and 15 committers[3], we need more people to join RocketMQ community. Everybody is cordially welcome to join the community and contribute to Apache RocketMQ, please refer to our contribution guide[4]. ## Mailing Lists RocketMQ has enabled the new group of mailing list: d...@rocketmq.apache.org users@rocketmq.apache.org comm...@rocketmq.apache.org iss...@rocketmq.apache.org Feel free to subscribe these list, just visit the contact page[5], and click `Subscribe`. Also, you are welcome to use priv...@rocketmq.apache.org, if you have any concern around RocketMQ security. ## RocketMQ Website The old domain(rocketmq.incubator.apache.org) of RocketMQ website is deprecated and will be redirected to TLP domain[6]. Welcome to star the site, and you will find the newest documents and roadmap here. ## Released Artifacts Apache RocketMQ has two formal releases during incubation, 4.0.0-incubating, and 4.1.0-incubating, the related download links have been moved, please refer to our download page[7] to fetch the newest link if you need. The first TLP version 4.2.0 will be released soon. Regards, yukon [1]. https://github.com/apache/rocketmq [2]. https://gitbox.apache.org/setup/ [3]. http://people.apache.org/phonebook.html?pmc=rocketmq [4]. http://rocketmq.apache.org/docs/how-to-contribute/ [5]. http://rocketmq.apache.org/about/contact/ [6]. http://rocketmq.apache.org/ [7]. http://rocketmq.apache.org/dowloading/releases/
Re: contribute windows cpp code to rocketmq-cpp
> I noticed that many rocketmq users requests windows .NET version, we could contirute this version to community which is also the kernel implementation of aliyun MQ windows .NET Indeed, the community really needs .NET SDK, thanks qiwei for your contribution. BTW, if any contributor has interests in CPP/.NET SDK, please work with qiwei together -:) Regards, yukon On Thu, Sep 28, 2017 at 2:46 PM, dongeforever wrote: > Nice. Thanks qiwei.wqw 's work. > > BTW, the content of the email will look better if the links could be put in > the end and numbered with [1] [2] ... > > Best Regards > dongeforever > > 2017-09-28 10:54 GMT+08:00 王启伟(孝孺) : > > > Hi dev&users, Not long ago , we contribute Linux cpp code to Apache > > RocketMQ community: https://github.com/apache/incubator-rocketmq- > > externals/tree/master/rocketmq-cpp. As I said before, we will > contribute > > windows cpp code later after contributed linux cpp code. Now, we will > > contribute windows cpp code to incubator-rocketmq-externals, which is the > > kernel implementation of aliyun MQ windows cpp(http://docs-aliyun.cn- > > hangzhou.oss.aliyun-inc.com/assets/attach/29555/cn_zh/ > > 1503837541521/aliyun-mq-windows-cpp-sdk.zip?spm=5176. > > doc29555.2.3.6GCGE2&file=aliyun-mq-windows-cpp-sdk.zip) It is stable > > with following characteristics: > > 1>. disaster recovery ability Based on nameServer snapshot and > > network disaster recovery strategy, no real-time impact on > > publish/subscribe when anyone of broker or nameSrv was broken2>. low > > latency publish latency < 2ms, subscribe latency < 10ms3>. High > > publish/subsricbe TPS For 16 message queues, publish TPS > 3W, > > subsricbe TPS > 15W4>. support all rocketmq features Such as > > broadcast/cluster model, orderly publish/subscribe, timed msg, consumer > > status query and so on. I noticed that many rocketmq users requests > > windows .NET version, we could contirute this version to community which > is > > also the kernel implementation of aliyun MQ windows .NET( > > http://docs-aliyun.cn-hangzhou.oss.aliyun-inc.com/ > > assets/attach/29561/cn_zh/1503837624451/aliyun-mq- > > windows-net-sdk.zip?spm=5176.doc29561.2.3.huJ1IQ&file= > > aliyun-mq-windows-net-sdk.zip) Most importantly, we are looking forward > > to new volunteers, let's build a more perfect CPP/.NET client for > RocketMQ! > > >
