@Brad: OK I guess that would work. But it's quite an implementation compared to the simple AMQ scheduler ;-)
@James: Great question. I will definitely think about this. Have you got a generic approach to save/restore the message body and headers to/from the database? @Quinn: That's not a big issue in my case since the delay is 24 hours or less. The messages are currently in the queue, so the kaha-db-files cannot be deleted anyway. Once again thanks a lot for all your great answers! Stephan Quinn Stevenson <qu...@pronoia-solutions.com> schrieb am Fr., 4. Nov. 2016 um 17:48 Uhr: > James brings up a very good point. > > I know ActiveMQ isn’t designed for long-term messages, and it’s bit me a > couple of times with KahaDB. What happened was I had quite a few old > messages, but they were spaced out over time. KahaDB doesn’t compact > files, and it can’t delete files until all the messages in the files are > processed. Long story short - we ran out of disk. > > We addressed this part by using mKahaDB - it didn’t really “fix” the > problem, but the nature of our messaging is such that we could live with it. > > > On Nov 4, 2016, at 10:34 AM, James Green <james.mk.gr...@gmail.com> > wrote: > > > > Are you unable to move the messages into a database for "long-term" > > storage? That's what we do, and we simply scan for messages due to be > > "re-sent" and re-submit them back into the queue. > > > > I do not believe ActiveMQ was really intended to hold many messages for > > long term storage, hence we use something that was. > > > > > > On 3 November 2016 at 15:46, Stephan Burkard <sburk...@gmail.com> wrote: > > > >> Hi Camel users > >> > >> TLDR: > >> - how to implement wait/retry loop? > >> - messages "wait" in a queue > >> - each message should be reprocessed only once per interval (let's say > per > >> day) > >> - after reprocessing a message either returns to the waiting queue or > >> continues (goes out of loop) > >> - after [maxRetry] attempts it is moved away (goes out of loop) > >> > >> > >> Full story: > >> > >> As part of a typical processing workflow, I have a queue with "waiting" > >> messages. These messages are waiting for data of another system and I > have > >> no idea when the data arrives. Therefore I want to reprocess the > messages > >> in an interval, let's say once a day. If the data is still missing, > they go > >> again to the waiting queue. > >> To move messages away that would wait forever, I set and check a > maxRetry. > >> > >> How would I best implement this? Some thoughts I made: > >> > >> 1. Just throw an exception if the data is missing. The message is > >> redelivered. > >> Problem: The message is instantly redelivered. So it does the retries > >> within milliseconds. I cannot customize the interval. > >> > >> 2. Use the Camel delayer. > >> Works good in general (my current solution). I can even calculate the > >> delay based on the JMSTimestamp to take into account the time a message > >> waits to be consumed. > >> Problem: For an interval of 1 day, a message is permanently in > >> processing state. The consumer is blocked. When I shutdown the > container, > >> Camel waits 5 minutes and after restarting the message goes to the DLQ. > >> > >> 3. Use a CronScheduledRoutePolicy to start/stop the route in the given > >> interval. > >> Problem: When I start the route once a day for 5 minutes, the messages > >> with missing data are circulating until the route is stopped again. So > they > >> try to find the data multiple times per second and instantly reach the > >> maxRetry. They must try only once per interval. > >> > >> 4. Use a JMS selector as from-Endpoint that selects "JMSTimestamp < > >> (currentTime - intervalTime)" > >> Problem: The from-Endpoint URI seems to be a constant, I can't make > the > >> selector dynamic. > >> > >> 5. Use the Camel Throttler > >> Problem: The throttler sets a bandwith, that is not what we need. > >> > >> 6. Use a Timer from-Endpoint and hand over to a bean that consumes the > >> messages (newest idea, not yet tried). > >> I guess I could manage to consume all waiting messages just once (even > >> if they already return before I am finished). > >> Problems I can think of: It is not very nice to build another JMS > >> consumer (beside Camel), but I could use Spring JMS Template. > >> > >> > >> Numbers 1, 3, 4 and 5 simply do not work (or I don't use them the right > >> way) for this case. Number 2 is my current solution, number 6 a new > idea. > >> Any recommendations or other suggestions how to build this wait/retry > loop? > >> > >> Thanks a lot > >> Stephan > >> > >
