Re: [DISCUSSION] Delayed message delivery

On Tue, Apr 16, 2019 at 8:08 PM Ezequiel Lovelle wrote: > > Hi Matteo! > > Great work! Really neat and clear, I like it! > > My 2 cents, I prefer adding deliverAt() and deliverAfter() on > ProducerBuidler rather than TypedMessageBuilder. > That would result in a more limited version because delay

Re: [DISCUSSION] Delayed message delivery

Hi Matteo! Great work! Really neat and clear, I like it! My 2 cents, I prefer adding deliverAt() and deliverAfter() on ProducerBuidler rather than TypedMessageBuilder. That would result in a more limited version because delay will be the same for all the messages, but I think it covers most of th

Re: [DISCUSSION] Delayed message delivery

Thanks everyone for the feedback. I actually went through and gave it a shot at implementing this on https://github.com/apache/pulsar/pull/4062 I think this implementation should address all the concern exposed in this thread. Please everyone involved take a deep review of the change. Thanks, M

Re: [DISCUSSION] Delayed message delivery

On Fri, Mar 8, 2019 at 11:11 PM Ezequiel Lovelle wrote: > > Seems like we are implementing per message timers. > > As per pr #3155 , nope. Each > message won't have a Timer class per se, > just a long field representing its expiration deadline and will

Re: [DISCUSSION] Delayed message delivery

> Seems like we are implementing per message timers. As per pr #3155 , nope. Each message won't have a Timer class per se, just a long field representing its expiration deadline and will be just one, and only one, scheduled task per consumer at any given

Re: [DISCUSSION] Delayed message delivery

Seems like we are implementing per message timers. Not aware of any log pub sub that does that expect rocketmq , not sure how performant that is. https://github.com/apache/rocketmq/blob/2b692c912d18c0f9889fd73358581bcccf37bbbe/store/src/main/java/org/apache/rocketmq/store/schedule/ScheduleMessageS

Re: [DISCUSSION] Delayed message delivery

I am trying to draw a conclusion on this email thread. > Maybe some way to plug to the broker some logic without interfering with its core? > In our business fixed delay at consumer level regardless of any producer > configuration is a big win due to easy implementation and usage. Based on Ezequ

Re: [DISCUSSION] Delayed message delivery

Sorry for hear that DLQ causes GC. Agree with discussed before, Dispatcher is a performance sensitive piece of code. If we make changes on the dispatcher, we must pay attention to memory overhead and blocking. I prefer fixed delayed message solution(aka delayed time level). User can define mult

Re: [DISCUSSION] Delayed message delivery

Hi, I am really not into any details of the proposed implementation, but was just wondering, has anyone had a look at how Uber implemented this in Cherami? Cherami seems very similar to Pulsar, its storage system also seems very similar to bookkeeper. They seem to implement delayed queues by storin

Re: [DISCUSSION] Delayed message delivery

Hi - Well, it does, but can this be implemented without building a delayQueue? It seems to me that a delayQueue both breaks resiliency if the broker goes down and would certainly add overhead. Perhaps my idea to discard responses that are too new and then retrieve once they are out of the delay

Re: [DISCUSSION] Delayed message delivery

Hi Dave! > I wonder if clients can add an optional argument to the broker call when pulling events. The argument would be the amount of delay. Any messages younger than the delay are not returned by the broker. This is exactly what https://github.com/apache/pulsar/pull/3155 does :). We still need

Re: [DISCUSSION] Delayed message delivery

Hi - My thoughts here may be completely useless but I wonder if clients can add an optional argument to the broker call when pulling events. The argument would be the amount of delay. Any messages younger than the delay are not returned by the broker. Regards, Dave > On Feb 19, 2019, at 11:47

Re: [DISCUSSION] Delayed message delivery

> The recent changes made to support DLQ caused major problems with garbage collection If garbage collection is a big concern maybe we could add some config parameter on the broker to disable the usage of this feature and return BrokerMetadataException in this situation, giving the power to the ad

Re: [DISCUSSION] Delayed message delivery

Agreed that dispatcher is a performance sensitive piece of code. Feel bad to hear that DLQ causes GC. Are there any issues tracking those items you guys identified with DLQ changes? > How is this different from a subscription running behind? As far as I understand form the discussion at #3155, I

Re: [DISCUSSION] Delayed message delivery

Delayed subscription is simpler, and probably worth doing in the broker IF done right. How is this different from a subscription running behind? Why does supporting that require this complex a change in the dispatcher, when we already support backlogged subscribers? I am extremely wary of change

Re: [DISCUSSION] Delayed message delivery

Hi all, I am going to wrap up the discussion regarding delayed delivery use cases. For arbitrary delayed delivery, there are a few +1s to doing PIP-26 in functions. I am assuming that we will go down this path, unless there are other proposals. However there is a use case Lovelle pointed out abo

Re: [DISCUSSION] Delayed message delivery

"I agree, but that is *not what #3155 tries to achieve." This typo made this phrase nonsense, sorry! On Mon, 28 Jan 2019, 16:44 Ezequiel Lovelle > What exactly is the delayed delivery use case? > > This is helpful on systems relaying on pulsar for persistent guarantees > and using it for synchro

Re: [DISCUSSION] Delayed message delivery

> What exactly is the delayed delivery use case? This is helpful on systems relaying on pulsar for persistent guarantees and using it for synchronization or some sort of checks, but on such systems is common to have some overhead committing data on persistent storage maybe due to buffered mechanis

Re: [DISCUSSION] Delayed message delivery

Considering the way pulsar is built +1 for doing PIP-26 in functions. I am more of thinking in a way like publish it pulsar we will make it available in a different queuing system if you need priority and delay messages support. Pulsar functions would go enough for this kind of use cases. On Fri,

Re: [DISCUSSION] Delayed message delivery

> Correct. PIP-26 can be implemented in Functions. I believe the last > discussion in PIP-26 thread kind of agree on functions approach. > If the community is okay with PIP-26 in functions, I think that is probably > a good approach to start. +1 for doing it in functions. -Ivan

Re: [DISCUSSION] Delayed message delivery

On Thu, Jan 24, 2019 at 8:13 AM Joe F wrote: > To me this discussion presupposes that a streaming system should provide > a service like a database. Before we discuss about how to implement this, > we should look at whether this is something that fits into what is the core > of Pulsar. I stil

Re: [DISCUSSION] Delayed message delivery

To me this discussion presupposes that a streaming system should provide a service like a database. Before we discuss about how to implement this, we should look at whether this is something that fits into what is the core of Pulsar. I still have the same concerns against doing this in the brok

Re: [DISCUSSION] Delayed message delivery

Hi all, Since this discussion have a lot of edges, I propose the following: Maybe can we make a vote or some sort of decision about what we are going to decide next? For example, I would left for a future discussion the topic of arbitrary delay for each message because is a more difficult to achi

Re: [DISCUSSION] Delayed message delivery

Hi All, Actually, I also prefer to simplify at broker side. If pulsar support set arbitrary timeout on each message, if not cluster failure or consumer failure, it needs to behave normally(on time). Otherwise, user need to understand how pulsar dispatching messages and how limit of unacked messag

Re: [DISCUSSION] Delayed message delivery

> If the goal is to minimize the amount of redeliveries from broker -> client, there are multiple ways to achieve that with the client based approach (eg. send message id and delay time instead of the full payload to consumers as Ivan proposed). But the main reason to put this logic on client side

Re: [DISCUSSION] Delayed message delivery

Just a quick correction: > And I don’t see anyone have a clear explanation on why a broker approach is less scalable than the client side approach. I haven't said it less or more scalable. I was meaning that it's "easier" to scale, in that we don't have to do lots of fancy stuff and add more and

Re: [DISCUSSION] Delayed message delivery

On Sat, Jan 19, 2019 at 9:45 AM Matteo Merli wrote: > Trying to group and compress responses here. > > > If consumer control the delayed message specific execution time we must > trust clock of consumer, this can cause delayed message process ahead of > time, some applications cannot tolerate thi

Re: [DISCUSSION] Delayed message delivery

Trying to group and compress responses here. > If consumer control the delayed message specific execution time we must trust clock of consumer, this can cause delayed message process ahead of time, some applications cannot tolerate this condition. This is a problem that cannot be solved. Even ass

Re: [DISCUSSION] Delayed message delivery

Hi All! and sorry for delay :) Probably I'm going to say some things already said, so sorry beforehand. The two main needed features I think are the proposed: A. Producer delay PIP-26. B. Consumers delay PR #3155 Of course PIP-26 would result in consumers receiving delayed messages but the impor

Re: [DISCUSSION] Delayed message delivery

> So rather than specifying the absolute timestamp that the message > should appear to the user, the dispatcher can specify the relative > delay after dispatch that it should appear to the user. As matteo said the worst case would be that the applied delay to be higher for some of the messages, if

Re: [DISCUSSION] Delayed message delivery

On Fri, Jan 18, 2019 at 2:51 PM Ivan Kelly wrote: > One thing missing from this discussion is details on the motivating > use-case. How many delayed messages per second are we expecting? And > what is the payload size? > > > If consumer control the delayed message specific execution time we must

Re: [DISCUSSION] Delayed message delivery

One thing missing from this discussion is details on the motivating use-case. How many delayed messages per second are we expecting? And what is the payload size? > If consumer control the delayed message specific execution time we must > trust clock of consumer, this can cause delayed message pro

Re: [DISCUSSION] Delayed message delivery

If consumer control the delayed message specific execution time we must trust clock of consumer, this can cause delayed message process ahead of time, some applications cannot tolerate this condition. > My concern of this category of approaches is "bandwidth" usage. It is > basically trading bandw

Re: [DISCUSSION] Delayed message delivery

On Thu, Jan 17, 2019 at 2:58 PM Matteo Merli wrote: > After a long delay (no pun intended), I finally got through the > previous discussions around the delayed message delivery proposals. > I'm referring to PIP-26 > https://github.com/apache/pulsar/wiki/PIP-26%3A-Delayed-Message-Delivery > and th

[DISCUSSION] Delayed message delivery

After a long delay (no pun intended), I finally got through the previous discussions around the delayed message delivery proposals. I'm referring to PIP-26 https://github.com/apache/pulsar/wiki/PIP-26%3A-Delayed-Message-Delivery and the Pull Request at #3155 https://github.com/apache/pulsar/pull/31