Creating then starting a JMS consumer around another component
We are interested in Camel but have a specific startup sequence that we need to follow: 1) Open a JMS topic consumer for our updates, but, do not get any of the messages. 2) Query our database and build a state of the world 3) Start to get the update messages from the consumer we opened We cannot adopt 2, 1, 3, as during the time we are done with our query but are still constructing the state of the world, we might miss an update. 1, 3, 2, is also a problem as an update could present an object which our state of the world subsequently tries to build, resulting in a duplicate. We could endeavor to handle this duplicate, but it is powerful to load the bulk of our data in the state of the world phase and demand absolutely no exceptions. What we have currently implemented can also result in duplicates. The state of the world might contain an object which an update then subsequently provides, but in this case we merely reject just one update, as opposed to having an issue with the much more significant state of the world operation. How can we achieve 1, 2, 3 using Camel? The difficulty that I see is that our consumer must be receiving messages, however, it must not consume them until the state of the world completes. Thanks for your time. -- View this message in context: http://camel.465427.n5.nabble.com/Creating-then-starting-a-JMS-consumer-around-another-component-tp5736955.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Creating then starting a JMS consumer around another component
Hi, Why isn't 2,1,3 possible? Why not setup your consumer before you start up your camel context? On Wed, Aug 7, 2013 at 7:22 PM, markgize wrote: > We are interested in Camel but have a specific startup sequence that we need > to follow: > > 1) Open a JMS topic consumer for our updates, but, do not get any of the > messages. > 2) Query our database and build a state of the world > 3) Start to get the update messages from the consumer we opened > > We cannot adopt 2, 1, 3, as during the time we are done with our query but > are still constructing the state of the world, we might miss an update. 1, > 3, 2, is also a problem as an update could present an object which our state > of the world subsequently tries to build, resulting in a duplicate. We > could endeavor to handle this duplicate, but it is powerful to load the bulk > of our data in the state of the world phase and demand absolutely no > exceptions. > > What we have currently implemented can also result in duplicates. The state > of the world might contain an object which an update then subsequently > provides, but in this case we merely reject just one update, as opposed to > having an issue with the much more significant state of the world operation. > > How can we achieve 1, 2, 3 using Camel? The difficulty that I see is that > our consumer must be receiving messages, however, it must not consume them > until the state of the world completes. > > Thanks for your time. > > > > -- > View this message in context: > http://camel.465427.n5.nabble.com/Creating-then-starting-a-JMS-consumer-around-another-component-tp5736955.html > Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Creating then starting a JMS consumer around another component
Could a better solution be to use durable subscriptions or ActiveMQ virtual topics so you're not depending on the timing of your consumer starting up as the basis for not missing messages? On Wed, Aug 7, 2013 at 6:43 PM, John D. Ament wrote: > Hi, > > Why isn't 2,1,3 possible? > Why not setup your consumer before you start up your camel context? > > On Wed, Aug 7, 2013 at 7:22 PM, markgize > wrote: > > We are interested in Camel but have a specific startup sequence that we > need > > to follow: > > > > 1) Open a JMS topic consumer for our updates, but, do not get any of the > > messages. > > 2) Query our database and build a state of the world > > 3) Start to get the update messages from the consumer we opened > > > > We cannot adopt 2, 1, 3, as during the time we are done with our query > but > > are still constructing the state of the world, we might miss an update. > 1, > > 3, 2, is also a problem as an update could present an object which our > state > > of the world subsequently tries to build, resulting in a duplicate. We > > could endeavor to handle this duplicate, but it is powerful to load the > bulk > > of our data in the state of the world phase and demand absolutely no > > exceptions. > > > > What we have currently implemented can also result in duplicates. The > state > > of the world might contain an object which an update then subsequently > > provides, but in this case we merely reject just one update, as opposed > to > > having an issue with the much more significant state of the world > operation. > > > > How can we achieve 1, 2, 3 using Camel? The difficulty that I see is > that > > our consumer must be receiving messages, however, it must not consume > them > > until the state of the world completes. > > > > Thanks for your time. > > > > > > > > -- > > View this message in context: > http://camel.465427.n5.nabble.com/Creating-then-starting-a-JMS-consumer-around-another-component-tp5736955.html > > Sent from the Camel - Users mailing list archive at Nabble.com. > -- *Christian Posta* http://www.christianposta.com/blog twitter: @christianposta
RE: Creating then starting a JMS consumer around another component
Thanks for your responses. Q: Why not setup your consumer before you start up your camel context? At the moment we prepare our context: CamelContext context = new DefaultCamelContext(); context.addComponent(JMS_QUEUE_COMPONENT_NAME, new JmsComponent(new JmsConfiguration(JmsFactory.getQueueConnectionFactory(; context.addComponent(JMS_TOPIC_COMPONENT_NAME, new JmsComponent(new JmsConfiguration(JmsFactory.getTopicConnectionFactory(; Apply a route builder: String requestRouteUri = CamelHelper.buildQueueRouteUri(ServiceFactory.getRequestQueue()); String responseRouteUri = CamelHelper.buildTopicRouteUri(ServiceFactory.getResponseTopic()); String logRouteUri = CamelHelper.buildLogRouteUri(LOGGER, LoggingLevel.TRACE); from(requestRouteUri).to(logRouteUri).process(engineProcessor).to(logRouteUri).to(responseRouteUri); from(responseRouteUri).to(logRouteUri).process(messageTranslator).to(logRouteUri).process(engineProcessor); Call start and Camel does its magic. We could start our consumer outside of Camel and then insert into the context, but then we are doing work outside of Camel and lose some of the elegance. It would be nice if we could handle this use case entirely within Camel, I also would have thought our use case is fairly typical (state of the world from the DB with updates via messaging), might we possibly create a new EIP to handle this situation or is that ridiculous? Q: Could a better solution be to use durable subscriptions or ActiveMQ virtual topics so you're not depending on the timing of your consumer starting up as the basis for not missing messages? We have an arbitrary number of clients who come and go as they please so I don't see durable subscriptions working that well, however, I think this could work. Nevertheless, I would note that our current approach is relatively simple and works perfectly, it is purely implementing this approach in Camel with which we have had trouble. Ideally, I think we would not have to tweak our architecture in order to adopt Camel. From: ceposta [via Camel] [mailto:ml-node+s465427n5737003...@n5.nabble.com] Sent: Thursday, August 08, 2013 8:37 AM To: Gizejewski, Mark Subject: Re: Creating then starting a JMS consumer around another component Could a better solution be to use durable subscriptions or ActiveMQ virtual topics so you're not depending on the timing of your consumer starting up as the basis for not missing messages? On Wed, Aug 7, 2013 at 6:43 PM, John D. Ament <[hidden email]>wrote: > Hi, > > Why isn't 2,1,3 possible? > Why not setup your consumer before you start up your camel context? > > On Wed, Aug 7, 2013 at 7:22 PM, markgize <[hidden > email]> > wrote: > > We are interested in Camel but have a specific startup sequence that we > need > > to follow: > > > > 1) Open a JMS topic consumer for our updates, but, do not get any of the > > messages. > > 2) Query our database and build a state of the world > > 3) Start to get the update messages from the consumer we opened > > > > We cannot adopt 2, 1, 3, as during the time we are done with our query > but > > are still constructing the state of the world, we might miss an update. > 1, > > 3, 2, is also a problem as an update could present an object which our > state > > of the world subsequently tries to build, resulting in a duplicate. We > > could endeavor to handle this duplicate, but it is powerful to load the > bulk > > of our data in the state of the world phase and demand absolutely no > > exceptions. > > > > What we have currently implemented can also result in duplicates. The > state > > of the world might contain an object which an update then subsequently > > provides, but in this case we merely reject just one update, as opposed > to > > having an issue with the much more significant state of the world > operation. > > > > How can we achieve 1, 2, 3 using Camel? The difficulty that I see is > that > > our consumer must be receiving messages, however, it must not consume > them > > until the state of the world completes. > > > > Thanks for your time. > > > > > > > > -- > > View this message in context: > http://camel.465427.n5.nabble.com/Creating-then-starting-a-JMS-consumer-around-another-component-tp5736955.html > > Sent from the Camel - Users mailing list archive at Nabble.com. > -- *Christian Posta* http://www.christianposta.com/blog twitter: @christianposta http://www.christianposta.com/blog ____________ If you reply to this email, your message will be added to the discussion below: http://camel.465427.n5
Re: Creating then starting a JMS consumer around another component
This is a common pattern I've seen before. It's not practical to have persistent queues/topics with each update to the database table being stored due to the transient nature of the consumer population (number of consumers is variable and their liveliness has no pattern). As Mark points out you need to subscribe to updates, buffer them, get the current 'view of the world' from the authoritative source (ie the database) and then let the updates flow in - potentially discarding duplicates at the start which were already in the image due to the timing. So in camel how can we start a 'route' so that it registers the subscription but 'pause' it whilst we get the table? It seems like we might need three routes and some funky 'on/off' tap logic. I don't see an EIP that supports this. Nigel -- View this message in context: http://camel.465427.n5.nabble.com/Creating-then-starting-a-JMS-consumer-around-another-component-tp5736955p5737347.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Creating then starting a JMS consumer around another component
Actually, you should be able to use a RoutePolicy to accomplish this:
from("jms:queue").routePolicy(customPolicy).to(...) continue route ...
http://camel.apache.org/routepolicy.html
In the onExchangeBegin() method, do not allow the processing to continue
(block) until the "world has been constructed"
On Thu, Aug 15, 2013 at 9:30 AM, nigel wrote:
> This is a common pattern I've seen before. It's not practical to have
> persistent queues/topics with each update to the database table being
> stored
> due to the transient nature of the consumer population (number of consumers
> is variable and their liveliness has no pattern). As Mark points out you
> need to subscribe to updates, buffer them, get the current 'view of the
> world' from the authoritative source (ie the database) and then let the
> updates flow in - potentially discarding duplicates at the start which were
> already in the image due to the timing.
> So in camel how can we start a 'route' so that it registers the
> subscription
> but 'pause' it whilst we get the table? It seems like we might need three
> routes and some funky 'on/off' tap logic. I don't see an EIP that supports
> this.
> Nigel
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Creating-then-starting-a-JMS-consumer-around-another-component-tp5736955p5737347.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
