PS putting the timestamp in millis in place would also let you fire up the route on a regular basis, every 5 minutes, hour, 8 hours or day. It wouldn't matter. In all cases the first thing it does is add the number of millis in a day to the timestamp and ensuring that the current time in millis is greater than that. Obviously if it is and your update fails, then when you increment the max retries you'd also update the timestamp so that it will not reprocess until the next day.
On Thu, Nov 3, 2016 at 1:17 PM, Brad Johnson <brad.john...@mediadriver.com> wrote: > On number 3, you might add a timestamp in millis in addition to the max > retries. When you pull the object off the queue do a quick addition of the > timestamp on the object (or header) and the number of millis in a day. The > current time in millis should exceed that number or a day has not passed > yet. If it is greater than the number then you can do the same logic with > your max retries that you do now. But that would give you a guard on > something like what's happening in number 3 to prevent it from spinning and > maxing out your retries. > > On Thu, Nov 3, 2016 at 11:22 AM, Quinn Stevenson < > qu...@pronoia-solutions.com> wrote: > >> I’ve done something similar to option 3 before, the difference being that >> I shutdown the route using the Control Bus after I’d processed each message. >> >> The other option I can think of is use two routes and two queues. The >> first route would try and process the “waiting” messages, and if the >> message cannot be successfully processed, publish it to another queue. The >> second route would simply read the message from the second queue and put >> them back on the “waiting” queue. This second route could be controlled >> with the CronScheduledRoutePolicy so it would only run once a day. >> >> If you’re using ActiveMQ, you could try using the scheduler ( >> http://activemq.apache.org/delay-and-schedule-message-delivery.html < >> http://activemq.apache.org/delay-and-schedule-message-delivery.html>) to >> delay the delivery - so you’re route could always be running. However, >> instead of rolling-back the message, you’d publish it back to the same >> queue and let ActiveMQ schedule when it would be re-delivered to you. >> >> HTH >> >> >> > On Nov 3, 2016, at 9:46 AM, Stephan Burkard <sburk...@gmail.com> wrote: >> > >> > Hi Camel users >> > >> > TLDR: >> > - how to implement wait/retry loop? >> > - messages "wait" in a queue >> > - each message should be reprocessed only once per interval (let's say >> per >> > day) >> > - after reprocessing a message either returns to the waiting queue or >> > continues (goes out of loop) >> > - after [maxRetry] attempts it is moved away (goes out of loop) >> > >> > >> > Full story: >> > >> > As part of a typical processing workflow, I have a queue with "waiting" >> > messages. These messages are waiting for data of another system and I >> have >> > no idea when the data arrives. Therefore I want to reprocess the >> messages >> > in an interval, let's say once a day. If the data is still missing, >> they go >> > again to the waiting queue. >> > To move messages away that would wait forever, I set and check a >> maxRetry. >> > >> > How would I best implement this? Some thoughts I made: >> > >> > 1. Just throw an exception if the data is missing. The message is >> > redelivered. >> > Problem: The message is instantly redelivered. So it does the retries >> > within milliseconds. I cannot customize the interval. >> > >> > 2. Use the Camel delayer. >> > Works good in general (my current solution). I can even calculate the >> > delay based on the JMSTimestamp to take into account the time a message >> > waits to be consumed. >> > Problem: For an interval of 1 day, a message is permanently in >> > processing state. The consumer is blocked. When I shutdown the >> container, >> > Camel waits 5 minutes and after restarting the message goes to the DLQ. >> > >> > 3. Use a CronScheduledRoutePolicy to start/stop the route in the given >> > interval. >> > Problem: When I start the route once a day for 5 minutes, the messages >> > with missing data are circulating until the route is stopped again. So >> they >> > try to find the data multiple times per second and instantly reach the >> > maxRetry. They must try only once per interval. >> > >> > 4. Use a JMS selector as from-Endpoint that selects "JMSTimestamp < >> > (currentTime - intervalTime)" >> > Problem: The from-Endpoint URI seems to be a constant, I can't make >> the >> > selector dynamic. >> > >> > 5. Use the Camel Throttler >> > Problem: The throttler sets a bandwith, that is not what we need. >> > >> > 6. Use a Timer from-Endpoint and hand over to a bean that consumes the >> > messages (newest idea, not yet tried). >> > I guess I could manage to consume all waiting messages just once (even >> > if they already return before I am finished). >> > Problems I can think of: It is not very nice to build another JMS >> > consumer (beside Camel), but I could use Spring JMS Template. >> > >> > >> > Numbers 1, 3, 4 and 5 simply do not work (or I don't use them the right >> > way) for this case. Number 2 is my current solution, number 6 a new >> idea. >> > Any recommendations or other suggestions how to build this wait/retry >> loop? >> > >> > Thanks a lot >> > Stephan >> >> >
