thanks Tim, that would be very helpful of you. I was about to dive into the camel source to see why the route wouldn't accept anything but if you have time to do a small test project I would be very grateful.
I'm running camel 2.8.1, activemq 5.5.0, spring 3.0.5 under tomcat. I've attached my pom.xml and camel-config.xml. mvn clean install then deploy to tomcat. many thanks, Alistair -- mov eax,1 mov ebx,0 int 80h
camel-config.xml
Description: XML document
pom.xml
Description: XML document
On 30 Sep 2011, at 13:23, Tim wrote:
> Alistair. I just tried the same with an embedded activemq instance and it
> works great.
> Maybe this has to do with broker settings? I can setup a tiny test project
> for you if you want to try that?
>
> On Fri, Sep 30, 2011 at 11:27 AM, Alistair Young
> <alistair.yo...@uhi.ac.uk>wrote:
>
>> I can now reproduce this every time. Sending 100 messages in very quick
>> succession to a camel route. JMX:
>>
>> <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
>> <from uri="activemq:topic:matrix"/>
>> <to uri="activemq:topic:edirectory"/>
>> </route>
>>
>> EnqueueCount = 100
>> DispatchCount = 100
>> InFlightCount = 100
>> DequeueCount = 1
>>
>> only 1 message ever gets through the route. That's 99 percent message loss,
>> every time. This happens on two servers with the same config.
>>
>> This happens with the producer on the same machine as the broker as well as
>> a remote broker.
>>
>> Is there anything I should be looking at to see where the other 99 messages
>> are? JMX says they're in flight but all dead letter queues are empty and
>> there are no errors anywhere.
>>
>> Alistair
>>
>> --
>> mov eax,1
>> mov ebx,0
>> int 80h
>>
>>
>>
>>
>> On 30 Sep 2011, at 09:12, Alistair Young wrote:
>>
>>> getting somewhere.
>>>
>>> <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
>>> <from uri="activemq:topic:matrix"/>
>>> <transacted />
>>> <process ref="matrixProcessor" />
>>> <to uri="activemq:topic:edirectory"/>
>>> </route>
>>>
>>> - producer clock is 2mins ahead of broker clock, timestampplugin enabled
>> on broker
>>> - no delay between messages at the producer = 62 out of 100 get through
>> the route, consistently
>>> - 3sec delay at producer, around 88 - 97 get through. That's as good as
>> it gets
>>> - 5sec delay at producer, 93 made it through the route
>>> - remove the route and send direct to topic in activemq, no delay, no
>> message loss, consistently
>>>
>>> so the messages are being lost in the route. No matter what the delay in
>> sending messages, some are always lost and vanish. There are no errors,
>> nothing in any dead letter queue. They simply vanish. They don't even make
>> it as far as the <process ref="matrixProcessor" />.
>>>
>>> sending from a ruby producer that blats them out far quicker than the
>> java producer is even worse. Only 1 - 3 ever get through the route. Removing
>> the route and sending to the activemq topic instead, all messages get
>> through no matter how fast they come.
>>>
>>> Alistair
>>>
>>> --
>>> mov eax,1
>>> mov ebx,0
>>> int 80h
>>>
>>>
>>>
>>>
>>> On 29 Sep 2011, at 18:10, Alistair Young wrote:
>>>
>>>> no connection pool. What was disturbing was the first message sent to
>> the broker after a restart and clean out of the data dir, disappeared.
>> There's a similar route on the broker that works fine. The only difference
>> is the producer for the wonky route is on windows and is up to 1min ahead of
>> the broker's clock. Would have thought the timestamplugin would take care of
>> that though. Can see it changing the timestamp in the logs.
>>>>
>>>> Alistair
>>>>
>>>> --------------
>>>> mov eax,1
>>>> mov ebx,0
>>>> int 80
>>>>
>>>> On 29 Sep 2011, at 17:31, Tim wrote:
>>>>
>>>>> Well with only 17 you definitely aren't hitting any prefetch limits or
>>>>> anything like that.
>>>>> Are you using a connection pool?
>>>>>
>>>>> On Thu, Sep 29, 2011 at 5:05 PM, Alistair Young <
>> alistair.yo...@uhi.ac.uk>wrote:
>>>>>
>>>>>> I think this way madness lies.
>>>>>>
>>>>>> 17 sent to topicA, dispatchCount = 15, dequeueCount = 12
>>>>>> topicB enqueueCount = 12
>>>>>>
>>>>>> so 17 came in, 12 made it through, of the 5 that went missing it
>> claims to
>>>>>> have sent 3 to topicB but they never arrived and the last 2 just
>> simply
>>>>>> vanished completely.
>>>>>>
>>>>>> What on earth?
>>>>>>
>>>>>> Alistair
>>>>>>
>>>>>> --------------
>>>>>> mov eax,1
>>>>>> mov ebx,0
>>>>>> int 80
>>>>>>
>>>>>> On 29 Sep 2011, at 15:41, Alistair Young wrote:
>>>>>>
>>>>>>> nup - cleaned out the data dir and restarted the broker. First
>> message in
>>>>>> vanished. Wasn't persisted. So something is fundamentally broken.
>>>>>>>
>>>>>>> topicA inflightCount = dispatchCount = enqueueCount = 1
>>>>>>> topicB is completely empty
>>>>>>>
>>>>>>> so the message wasn't persisted, wasn't processed, wasn't routed and
>> just
>>>>>> vanished from the broker.
>>>>>>>
>>>>>>> Alistair
>>>>>>>
>>>>>>> --------------
>>>>>>> mov eax,1
>>>>>>> mov ebx,0
>>>>>>> int 80
>>>>>>>
>>>>>>> On 29 Sep 2011, at 15:13, Alistair Young wrote:
>>>>>>>
>>>>>>>> route goes from topicA -> topicB, transacted.
>>>>>>>> topicA inflightCount = 96 and increases on each batch of incoming
>>>>>> messages
>>>>>>>> topicB dispatchCount = enqueueCount
>>>>>>>>
>>>>>>>> wondering if the missing messages are connected to topicA
>> inflightCount.
>>>>>> I noticed there are two consumers for topicB. The main consumer gets
>> its
>>>>>> messages fine. Wonder if the second consumer is a durable topic
>> consumer and
>>>>>> therefore activemq is persisting its messages but it hasn't connected
>> in a
>>>>>> very long time. Would that cause the topic to get too big? i.e.
>> messages go
>>>>>> into the topic until the limit is reached. Main consumer pulls
>> messages off
>>>>>> and messages are able to go onto topicB again. Before consumer pulls
>> and
>>>>>> after limit reached, messages can't get from topicA -> topicB, hence
>> the
>>>>>> topicA inflightCount not zero?
>>>>>>>>
>>>>>>>> Alistair
>>>>>>>>
>>>>>>>> --------------
>>>>>>>> mov eax,1
>>>>>>>> mov ebx,0
>>>>>>>> int 80
>>>>>>>>
>>>>>>>> On 29 Sep 2011, at 12:17, Tim wrote:
>>>>>>>>
>>>>>>>>> Sorry you might have tried this since I haven't been following this
>>>>>> thread.
>>>>>>>>> But can you check your jmx console.
>>>>>>>>> In particular check 2 things.. the route to see if the number of
>>>>>> exchanges
>>>>>>>>> match what you think and how if any exchanges failed.
>>>>>>>>> Also check the JMX console on activemq for the queue or topic in
>>>>>> question
>>>>>>>>> and see how many were enqueued vs dispatched.
>>>>>>>>> Check your deadletter queue from there too
>>>>>>>>>
>>>>>>>>> On Thu, Sep 29, 2011 at 12:52 PM, Alistair Young
>>>>>>>>> <alistair.yo...@uhi.ac.uk>wrote:
>>>>>>>>>
>>>>>>>>>> dunno - nothing works. Random messages are just vanishing once
>> they
>>>>>> reach
>>>>>>>>>> the broker. No trace, no logs, no dead letter queue. Just
>> vanishing.
>>>>>> I've
>>>>>>>>>> removed <transacted /> and <process> but it doesn't help. The
>> producer
>>>>>> is a
>>>>>>>>>> few secs behind the broker:
>>>>>>>>>>
>>>>>>>>>> sent : 11:25:26
>>>>>>>>>> arrived : 11:24:57
>>>>>>>>>> timstamp on message : 1317291897071 = 29 Sep 2011 10:24:57 GMT,
>>>>>> presumably
>>>>>>>>>> the timestampplugin doing this
>>>>>>>>>> message vanishes
>>>>>>>>>>
>>>>>>>>>> but all messages display this clock behaviour and not all vanish.
>>>>>>>>>>
>>>>>>>>>> Alistair
>>>>>>>>>>
>>>>>>>>>> --------------
>>>>>>>>>> mov eax,1
>>>>>>>>>> mov ebx,0
>>>>>>>>>> int 80
>>>>>>>>>>
>>>>>>>>>> On 29 Sep 2011, at 10:24, Alistair Young wrote:
>>>>>>>>>>
>>>>>>>>>>> just saw your info about transacted being before from - will
>> change
>>>>>> that
>>>>>>>>>> and monitor again.
>>>>>>>>>>>
>>>>>>>>>>> thanks,
>>>>>>>>>>>
>>>>>>>>>>> Alistair
>>>>>>>>>>>
>>>>>>>>>>> --------------
>>>>>>>>>>> mov eax,1
>>>>>>>>>>> mov ebx,0
>>>>>>>>>>> int 80
>>>>>>>>>>>
>>>>>>>>>>> On 29 Sep 2011, at 10:18, Alistair Young wrote:
>>>>>>>>>>>
>>>>>>>>>>>> just noticed a batch of identical 5 messages, three were missing
>> and
>>>>>>>>>> another single message vanished. tracer logged nothing. No errors,
>>>>>> dead
>>>>>>>>>> letter queue empty.
>>>>>>>>>>>>
>>>>>>>>>>>> One thing that happens is another machine polls the stats topic
>> in
>>>>>>>>>> activemq every 2mins. Would that cause a problem? It asks for
>> stats on
>>>>>> the
>>>>>>>>>> matrix topic, which is part of the transacted route.
>>>>>>>>>>>>
>>>>>>>>>>>> 29 September 2011 10:05:07 - Adding destination :
>>>>>>>>>> Topic:ActiveMQ.Advisory.Connection
>>>>>>>>>>>> 29 September 2011 10:05:07 - Creating new transaction with name
>>>>>> [null]:
>>>>>>>>>> PROPAGATION_REQUIRED,ISOLATION_DEFAULT
>>>>>>>>>>>> 29 September 2011 10:05:07 - Stopping connection:
>>>>>>>>>> vm://matrixBroker#285916
>>>>>>>>>>>> 29 September 2011 10:05:07 - Stopped transport:
>>>>>> vm://matrixBroker#285916
>>>>>>>>>>>> 29 September 2011 10:05:07 - Connection Stopped:
>>>>>>>>>> vm://matrixBroker#285916
>>>>>>>>>>>> 29 September 2011 10:05:07 - Setting up new connection id:
>>>>>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address:
>>>>>>>>>> vm://matrixBroker#285920
>>>>>>>>>>>> 29 September 2011 10:05:07 - Adding Connection : ConnectionInfo
>>>>>>>>>> {commandId = 1, responseRequired = true, connectionId =
>>>>>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId
>> =
>>>>>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName
>> =
>>>>>> null,
>>>>>>>>>> password = *****, brokerPath = null, brokerMasterConnector =
>> false,
>>>>>>>>>> manageable = true, clientMaster = true, faultTolerant = false}
>>>>>>>>>>>>
>>>>>>>>>>>> Alistair
>>>>>>>>>>>>
>>>>>>>>>>>> --------------
>>>>>>>>>>>> mov eax,1
>>>>>>>>>>>> mov ebx,0
>>>>>>>>>>>> int 80
>>>>>>>>>>>>
>>>>>>>>>>>> On 29 Sep 2011, at 09:36, Alistair Young wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>> <transacted/> Should be after <from>
>>>>>>>>>>>>> it is after from - do you mean it should be before?
>>>>>>>>>>>>> <route id="eDirSuccessBroadcast">
>>>>>>>>>>>>> <from uri="activemq:topic:edirectoryprocessed"/>
>>>>>>>>>>>>> <transacted />
>>>>>>>>>>>>> <process ref="groupwiseProcessor" />
>>>>>>>>>>>>> <to uri="activemq:topic:blackboard"/>
>>>>>>>>>>>>> </route>
>>>>>>>>>>>>>
>>>>>>>>>>>>> thanks for the dead letter tips, will apply them.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>>
>>>>>>>>>>>>> --------------
>>>>>>>>>>>>> mov eax,1
>>>>>>>>>>>>> mov ebx,0
>>>>>>>>>>>>> int 80
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 29 Sep 2011, at 09:20, Claus Ibsen wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> <transacted/> Should be after <from>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
>>>>>>>>>>>>>> <alistair.yo...@uhi.ac.uk> wrote:
>>>>>>>>>>>>>>>> Do you use message expiry?
>>>>>>>>>>>>>>> no
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> timestamp plugin
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> using that
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> activemq 5.5.0
>>>>>>>>>>>>>>> camel 2.8.0
>>>>>>>>>>>>>>> spring 3.0.5
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> noticed sl4j errors on startup, fixed that and now the tracer
>> is
>>>>>>>>>> logging so hopefully I can see any errors.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> <route id="matrix"
>>>>>> errorHandlerRef="matrixDeadLetterErrorHandler">
>>>>>>>>>>>>>>> <from uri="activemq:topic:matrix"/>
>>>>>>>>>>>>>>> <process ref="matrixProcessor" />
>>>>>>>>>>>>>>> <transacted />
>>>>>>>>>>>>>>> <to uri="activemq:topic:edirectory"/>
>>>>>>>>>>>>>>> </route>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> <bean id="jmsConnectionFactory"
>>>>>>>>>> class="org.apache.activemq.ActiveMQConnectionFactory"
>>>>>>>>>> depends-on="matrixBrokerID">
>>>>>>>>>>>>>>> <property name="brokerURL"
>>>>>>>>>> value="vm://matrixBroker?create=false"/>
>>>>>>>>>>>>>>> </bean>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> <bean id="jmsTransactionManager"
>>>>>>>>>> class="org.springframework.jms.connection.JmsTransactionManager">
>>>>>>>>>>>>>>> <property name="connectionFactory"
>>>>>>>>>> ref="jmsConnectionFactory"/>
>>>>>>>>>>>>>>> </bean>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> <bean id="activemq"
>>>>>>>>>> class="org.apache.activemq.camel.component.ActiveMQComponent">
>>>>>>>>>>>>>>> <property name="connectionFactory"
>>>>>>>>>> ref="jmsConnectionFactory"/>
>>>>>>>>>>>>>>> <property name="transacted" value="true"/>
>>>>>>>>>>>>>>> <property name="transactionManager"
>>>>>>>>>> ref="jmsTransactionManager"/>
>>>>>>>>>>>>>>> </bean>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> <bean id="matrixDeadLetterErrorHandler"
>>>>>>>>>> class="org.apache.camel.builder.DeadLetterChannelBuilder">
>>>>>>>>>>>>>>> <property name="deadLetterUri" value="jms:queue:dead"/>
>>>>>>>>>>>>>>> <property name="redeliveryPolicy"
>>>>>>>>>> ref="matrixRedeliveryPolicyConfig"/>
>>>>>>>>>>>>>>> </bean>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> <bean id="matrixRedeliveryPolicyConfig"
>>>>>>>>>> class="org.apache.camel.processor.RedeliveryPolicy">
>>>>>>>>>>>>>>> <property name="maximumRedeliveries" value="10"/>
>>>>>>>>>>>>>>> <property name="redeliveryDelay" value="250"/>
>>>>>>>>>>>>>>> </bean>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> thanks,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --------------
>>>>>>>>>>>>>>> mov eax,1
>>>>>>>>>>>>>>> mov ebx,0
>>>>>>>>>>>>>>> int 80
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 29 Sep 2011, at 08:53, Claus Ibsen wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Do you use message expiry?
>>>>>>>>>>>>>>>> Make sure clocks between server/clients is synced as much as
>>>>>>>>>> possible.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> There is a timestamp plugin
>>>>>>>>>>>>>>>> http://activemq.apache.org/timestampplugin.html
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> And do you use queue or topic.
>>>>>>>>>>>>>>>> What version of AMQ and Camel are you using?
>>>>>>>>>>>>>>>> And how have you configured the AMQ broker, and the Camel
>>>>>> context?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <
>>>>>> taar...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Where the logs go, if it's logged at all, still depends on
>> your
>>>>>>>>>> logger and
>>>>>>>>>>>>>>>>> how you configured it.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Here are links to how to enable logging[1] and camel
>> logging
>>>>>> FAQ[2]
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [1]
>> http://camel.apache.org/how-do-i-enable-debug-logging.html
>>>>>>>>>>>>>>>>> [2]http://camel.apache.org/logging-questions.html
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Taariq
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <
>>>>>>>>>> alistair.yo...@uhi.ac.uk>wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> which is the best trace method to use? trace="true", or
>>>>>>>>>> camelTracer and
>>>>>>>>>>>>>>>>>> traceFormatter beans? and where does the log end up? I've
>>>>>> tried
>>>>>>>>>> them all but
>>>>>>>>>>>>>>>>>> no log appears.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> mov eax,1
>>>>>>>>>>>>>>>>>> mov ebx,0
>>>>>>>>>>>>>>>>>> int 80h
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 28 Sep 2011, at 12:08, Marco Westermann wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I suggest enable tracing to see exactly what happens in
>> your
>>>>>>>>>> route.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> regards, Marco
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Am 28.09.2011 13:01, schrieb Alistair Young:
>>>>>>>>>>>>>>>>>>>> I now have a dead letter channel which is empty after
>> losing
>>>>>> 9
>>>>>>>>>> out of 10
>>>>>>>>>>>>>>>>>> messages. I also added a logging handler which logged
>> nothing.
>>>>>>>>>> Verified the
>>>>>>>>>>>>>>>>>> messages arrived at the broker, then they just vanished.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Claus Ibsen
>>>>>>>>>>>>>>>> -----------------
>>>>>>>>>>>>>>>> FuseSource
>>>>>>>>>>>>>>>> Email: cib...@fusesource.com
>>>>>>>>>>>>>>>> Web: http://fusesource.com
>>>>>>>>>>>>>>>> Twitter: davsclaus, fusenews
>>>>>>>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Claus Ibsen
>>>>>>>>>>>>>> -----------------
>>>>>>>>>>>>>> FuseSource
>>>>>>>>>>>>>> Email: cib...@fusesource.com
>>>>>>>>>>>>>> Web: http://fusesource.com
>>>>>>>>>>>>>> Twitter: davsclaus, fusenews
>>>>>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>
>>
>>
