Hi Dirk, the short answer is: no, the latest patch contained CAMEL-126 doesn't solve the problem that you might loose messages if there a system failure. The long answer is: it is only a first step into that direction:
a) The patch ensures that reading a message from an endpoint and delivery of pending/reordered messages occurs within a single thread (in Camel 1.4 there is still a separate thread doing the delivery). However, the transaction from the source endpoint currently isn't propagated yet. b) The patch changes the current implementation of the stream resequencer to work with a polling consumer in the same way as the batch resequencer does (btw, the batch resequencer currently isn't suitable for reliable resequencing as well, from what I can see from the source code). To make both resequencers reliable the following extensions are required: - Batch resequencer: read n messages within a single transaction (instead of n transactions as done currently in 1.4), reorder them, send them to a destination and commit the transaction. This can easily done within a single thread. - Stream resequencer: the situation is more tricky here because there is a continuous stream of messages without having candidates for transaction boundaries (in contrast to the batch resequencer where the batch size can be used to determine a transaction boundary). I'm still not 100% sure how to solve this problem but a possible solution might roughly look like this: Read messages from a source endpoint with n threads, suspend them until they have been re-ordered based on the message they have read and resume/execute them sequentially after their message becomes eligible for delivery. After delivery of the message each thread commits the transaction it started. Consequently, each message is delivered to a destination within a separate transaction (in contrast to the batch resequencer where several messages can be delivered within a single transaction) but the resequencing is reliable too. So far I had no time to work on these extensions. The initial patch of CAMEL-126 is planned to be applied with Camel 1.5, so I'll try to get the extensions done in the 1.6-SNAPSHOT because I also see the need for reliable stream resequencing in one of my upcoming projects. Even if it's not an immediate help for your problem, do the proposed extensions make sense to you? Can you see any alternatives how to solve the reliable (stream) resequencing problem? Cheers, Martin > -----Ursprüngliche Nachricht----- > Von: dirk.mahler [mailto:[EMAIL PROTECTED] > Gesendet: Freitag, 5. September 2008 15:16 > An: [email protected] > Betreff: Reliable resequencing > > > Hi, > > we integrated Camel 1.4 as JBI component into ServiceMix and tried to use > the stream resequencer to reconstruct the "natural" order of our messages. > > The routing on ServiceMix (SMX) level is defined as follows: > > SMX transactional JMS/JCA consumer -> Camel JBI Component -> SMX Bean SE > > The Camel JBI Component is configured to provide resequencing the > following > way (I left out some message body conversion stuff): > > from("jbi:endpoint:...") > .resequencer(sequenceExpression) > .stream(config) > .to("jbi:endpoint:.../beanEndpoint"); > > Resequencing seems to work but according to some simple tests it cannot be > guaranteed that all messages will reach the final consumer (in our case > the > bean endpoint) if there's a system failure: Camel acknowlegdes the message > exchange to the JMS Consumer but then holds the messages in a non- > persistent > queue before delivering them to the SMX bean endpoint. We found an > existing > JIRA issue (CAMEL-126, https://issues.apache.org/activemq/browse/CAMEL- > 126) > which at a first glance seems to address our concerns by thinking about > synchronous dispatching - would this solve our problem or is there already > another solution? > > > > ----- > Dirk Mahler > Senior Consultant > > ----------------------------------------------------------------- > buschmais GbR > Inhaber Torsten Busch, Frank Schwarz, Dirk Mahler, Tobias Israel > Adresse buschmais GbR, Leipziger Straße 93, 01127 Dresden > Telefon +49 (0) 351 3209 23-0 > Fax +49 (0) 351 3209 23-29 > Telefon +49 (0) 1577 198 295 0 > E-Mail [EMAIL PROTECTED] > Internet http://www.buschmais.de > ----------------------------------------------------------------- > -- > View this message in context: http://www.nabble.com/Reliable-resequencing- > tp19331003s22882p19331003.html > Sent from the Camel - Users mailing list archive at Nabble.com.
