I have a pattern which I think I need advice on... I have three tasks... each a type of message consumer.
Let's call them A, B, an C. A runs once,, creates 15 messages, sends them to B... then B process these messages then generates 15 new messages. However, they need to be combined into a group, and sent to C all at once, in one composite message. So its similar to map/reduce in a way in that C should execute once with a block of these 15 messages. Conceptually I'm calling them (message group checkpoints).. but I'm wondering if there's already a more formal name for this concept. I'm not sure the best way to handle this with ActiveMQ. One strategy is that I could have one queue per C tasks (the final tasks) and then have C running and consuming them one at a time, and then performing the execution (and message commit) once it receives all 15 messages. I HAVE to get all the messages until I can make a decision, I can't stream process them unfortunately because the algorithm needs all data points. I could use a database.. but the problem there is that it would incur some database and Cassandra (in our case) doesn't handle this queue pattern very well. Another idea is to use a series of queues ... say aggregation_0, aggregation_1, aggregation_2,... then I receive these messages into the first queue (aggregation_0), then sort it, if any of the groups are finished I send them on to the final destination task. If any are unfinished then I overflow them on to aggregation1 (pre-sorted)... Thoughts? -- We’re hiring if you know of any awesome Java Devops or Linux Operations Engineers! Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile <https://plus.google.com/102718274791889610666/posts>
