Hello.
We’re using Artemis to distribute files between various systems. The workflow
is almost always the same:
1. Shell scripts pass files to client applications.
2. These client applications deliver the files to an Artemis broker using
the core protocol.
3. Another core protocol clients poll the broker repeatedly, receive
messages, store them on file systems, run shell script handlers (one instance
per file), and then accept or refuse each message individually based on its
handler return code.
Clients are massively parallelized in order to achieve good performance, so
each client has multiple sessions opened.
Now let’s focus on pollers:
* Each poll request asks for at most N files, and based on that, at most S
sessions are assigned jobs, where each job is meant do handle at most one file
at a time. If N > S, then some sessions are used repeatedly during that request.
* Each session uses exactly one consumer at a time.
* When a file arrives, it is passed to a handler.
* If the handler ends with RC 0, message.individualAcknowledge() and
session.commit() calls are performed.
* Otherwise, session.rollback() is called.
* Handlers can run for tens of seconds, or even minutes before it’s clear
whether to accept or refuse a file.
* When no file arrives, then that session’s work is done.
Unfortunately, handling failure followed by session.rollback() is where things
get complicated. As soon as a file is refused, the broker schedules it for
repeated delivery, and another session working on the very same request can
receive it again. So far, this hasn’t been much of an issue and I’ve been using
a workaround of closing the consumer, instead of proceeding with just another
session.rollback() which would increase the number of unsuccessful deliveries
for that message.
But the time has come, and now I have to implement message grouping support,
which rules out my workaround, because I need to keep my consumers open for as
long as possible.
How can I group my sessions together so that they can work on a single request
in parallel in a way that none of them receive files I’ve already rejected?
Thanks for your ideas.
Jan
