getting somewhere.
<route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
<from uri="activemq:topic:matrix"/>
<transacted />
<process ref="matrixProcessor" />
<to uri="activemq:topic:edirectory"/>
</route>
- producer clock is 2mins ahead of broker clock, timestampplugin enabled on
broker
- no delay between messages at the producer = 62 out of 100 get through the
route, consistently
- 3sec delay at producer, around 88 - 97 get through. That's as good as it gets
- 5sec delay at producer, 93 made it through the route
- remove the route and send direct to topic in activemq, no delay, no message
loss, consistently
so the messages are being lost in the route. No matter what the delay in
sending messages, some are always lost and vanish. There are no errors, nothing
in any dead letter queue. They simply vanish. They don't even make it as far as
the <process ref="matrixProcessor" />.
sending from a ruby producer that blats them out far quicker than the java
producer is even worse. Only 1 - 3 ever get through the route. Removing the
route and sending to the activemq topic instead, all messages get through no
matter how fast they come.
Alistair
--
mov eax,1
mov ebx,0
int 80h
On 29 Sep 2011, at 18:10, Alistair Young wrote:
> no connection pool. What was disturbing was the first message sent to the
> broker after a restart and clean out of the data dir, disappeared. There's a
> similar route on the broker that works fine. The only difference is the
> producer for the wonky route is on windows and is up to 1min ahead of the
> broker's clock. Would have thought the timestamplugin would take care of that
> though. Can see it changing the timestamp in the logs.
>
> Alistair
>
> --------------
> mov eax,1
> mov ebx,0
> int 80
>
> On 29 Sep 2011, at 17:31, Tim wrote:
>
>> Well with only 17 you definitely aren't hitting any prefetch limits or
>> anything like that.
>> Are you using a connection pool?
>>
>> On Thu, Sep 29, 2011 at 5:05 PM, Alistair Young
>> <alistair.yo...@uhi.ac.uk>wrote:
>>
>>> I think this way madness lies.
>>>
>>> 17 sent to topicA, dispatchCount = 15, dequeueCount = 12
>>> topicB enqueueCount = 12
>>>
>>> so 17 came in, 12 made it through, of the 5 that went missing it claims to
>>> have sent 3 to topicB but they never arrived and the last 2 just simply
>>> vanished completely.
>>>
>>> What on earth?
>>>
>>> Alistair
>>>
>>> --------------
>>> mov eax,1
>>> mov ebx,0
>>> int 80
>>>
>>> On 29 Sep 2011, at 15:41, Alistair Young wrote:
>>>
>>>> nup - cleaned out the data dir and restarted the broker. First message in
>>> vanished. Wasn't persisted. So something is fundamentally broken.
>>>>
>>>> topicA inflightCount = dispatchCount = enqueueCount = 1
>>>> topicB is completely empty
>>>>
>>>> so the message wasn't persisted, wasn't processed, wasn't routed and just
>>> vanished from the broker.
>>>>
>>>> Alistair
>>>>
>>>> --------------
>>>> mov eax,1
>>>> mov ebx,0
>>>> int 80
>>>>
>>>> On 29 Sep 2011, at 15:13, Alistair Young wrote:
>>>>
>>>>> route goes from topicA -> topicB, transacted.
>>>>> topicA inflightCount = 96 and increases on each batch of incoming
>>> messages
>>>>> topicB dispatchCount = enqueueCount
>>>>>
>>>>> wondering if the missing messages are connected to topicA inflightCount.
>>> I noticed there are two consumers for topicB. The main consumer gets its
>>> messages fine. Wonder if the second consumer is a durable topic consumer and
>>> therefore activemq is persisting its messages but it hasn't connected in a
>>> very long time. Would that cause the topic to get too big? i.e. messages go
>>> into the topic until the limit is reached. Main consumer pulls messages off
>>> and messages are able to go onto topicB again. Before consumer pulls and
>>> after limit reached, messages can't get from topicA -> topicB, hence the
>>> topicA inflightCount not zero?
>>>>>
>>>>> Alistair
>>>>>
>>>>> --------------
>>>>> mov eax,1
>>>>> mov ebx,0
>>>>> int 80
>>>>>
>>>>> On 29 Sep 2011, at 12:17, Tim wrote:
>>>>>
>>>>>> Sorry you might have tried this since I haven't been following this
>>> thread.
>>>>>> But can you check your jmx console.
>>>>>> In particular check 2 things.. the route to see if the number of
>>> exchanges
>>>>>> match what you think and how if any exchanges failed.
>>>>>> Also check the JMX console on activemq for the queue or topic in
>>> question
>>>>>> and see how many were enqueued vs dispatched.
>>>>>> Check your deadletter queue from there too
>>>>>>
>>>>>> On Thu, Sep 29, 2011 at 12:52 PM, Alistair Young
>>>>>> <alistair.yo...@uhi.ac.uk>wrote:
>>>>>>
>>>>>>> dunno - nothing works. Random messages are just vanishing once they
>>> reach
>>>>>>> the broker. No trace, no logs, no dead letter queue. Just vanishing.
>>> I've
>>>>>>> removed <transacted /> and <process> but it doesn't help. The producer
>>> is a
>>>>>>> few secs behind the broker:
>>>>>>>
>>>>>>> sent : 11:25:26
>>>>>>> arrived : 11:24:57
>>>>>>> timstamp on message : 1317291897071 = 29 Sep 2011 10:24:57 GMT,
>>> presumably
>>>>>>> the timestampplugin doing this
>>>>>>> message vanishes
>>>>>>>
>>>>>>> but all messages display this clock behaviour and not all vanish.
>>>>>>>
>>>>>>> Alistair
>>>>>>>
>>>>>>> --------------
>>>>>>> mov eax,1
>>>>>>> mov ebx,0
>>>>>>> int 80
>>>>>>>
>>>>>>> On 29 Sep 2011, at 10:24, Alistair Young wrote:
>>>>>>>
>>>>>>>> just saw your info about transacted being before from - will change
>>> that
>>>>>>> and monitor again.
>>>>>>>>
>>>>>>>> thanks,
>>>>>>>>
>>>>>>>> Alistair
>>>>>>>>
>>>>>>>> --------------
>>>>>>>> mov eax,1
>>>>>>>> mov ebx,0
>>>>>>>> int 80
>>>>>>>>
>>>>>>>> On 29 Sep 2011, at 10:18, Alistair Young wrote:
>>>>>>>>
>>>>>>>>> just noticed a batch of identical 5 messages, three were missing and
>>>>>>> another single message vanished. tracer logged nothing. No errors,
>>> dead
>>>>>>> letter queue empty.
>>>>>>>>>
>>>>>>>>> One thing that happens is another machine polls the stats topic in
>>>>>>> activemq every 2mins. Would that cause a problem? It asks for stats on
>>> the
>>>>>>> matrix topic, which is part of the transacted route.
>>>>>>>>>
>>>>>>>>> 29 September 2011 10:05:07 - Adding destination :
>>>>>>> Topic:ActiveMQ.Advisory.Connection
>>>>>>>>> 29 September 2011 10:05:07 - Creating new transaction with name
>>> [null]:
>>>>>>> PROPAGATION_REQUIRED,ISOLATION_DEFAULT
>>>>>>>>> 29 September 2011 10:05:07 - Stopping connection:
>>>>>>> vm://matrixBroker#285916
>>>>>>>>> 29 September 2011 10:05:07 - Stopped transport:
>>> vm://matrixBroker#285916
>>>>>>>>> 29 September 2011 10:05:07 - Connection Stopped:
>>>>>>> vm://matrixBroker#285916
>>>>>>>>> 29 September 2011 10:05:07 - Setting up new connection id:
>>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address:
>>>>>>> vm://matrixBroker#285920
>>>>>>>>> 29 September 2011 10:05:07 - Adding Connection : ConnectionInfo
>>>>>>> {commandId = 1, responseRequired = true, connectionId =
>>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId =
>>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName =
>>> null,
>>>>>>> password = *****, brokerPath = null, brokerMasterConnector = false,
>>>>>>> manageable = true, clientMaster = true, faultTolerant = false}
>>>>>>>>>
>>>>>>>>> Alistair
>>>>>>>>>
>>>>>>>>> --------------
>>>>>>>>> mov eax,1
>>>>>>>>> mov ebx,0
>>>>>>>>> int 80
>>>>>>>>>
>>>>>>>>> On 29 Sep 2011, at 09:36, Alistair Young wrote:
>>>>>>>>>
>>>>>>>>>>> <transacted/> Should be after <from>
>>>>>>>>>> it is after from - do you mean it should be before?
>>>>>>>>>> <route id="eDirSuccessBroadcast">
>>>>>>>>>> <from uri="activemq:topic:edirectoryprocessed"/>
>>>>>>>>>> <transacted />
>>>>>>>>>> <process ref="groupwiseProcessor" />
>>>>>>>>>> <to uri="activemq:topic:blackboard"/>
>>>>>>>>>> </route>
>>>>>>>>>>
>>>>>>>>>> thanks for the dead letter tips, will apply them.
>>>>>>>>>>
>>>>>>>>>> Alistair
>>>>>>>>>>
>>>>>>>>>> --------------
>>>>>>>>>> mov eax,1
>>>>>>>>>> mov ebx,0
>>>>>>>>>> int 80
>>>>>>>>>>
>>>>>>>>>> On 29 Sep 2011, at 09:20, Claus Ibsen wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi
>>>>>>>>>>>
>>>>>>>>>>> <transacted/> Should be after <from>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
>>>>>>>>>>> <alistair.yo...@uhi.ac.uk> wrote:
>>>>>>>>>>>>> Do you use message expiry?
>>>>>>>>>>>> no
>>>>>>>>>>>>
>>>>>>>>>>>>> timestamp plugin
>>>>>>>>>>>>
>>>>>>>>>>>> using that
>>>>>>>>>>>>
>>>>>>>>>>>> activemq 5.5.0
>>>>>>>>>>>> camel 2.8.0
>>>>>>>>>>>> spring 3.0.5
>>>>>>>>>>>>
>>>>>>>>>>>> noticed sl4j errors on startup, fixed that and now the tracer is
>>>>>>> logging so hopefully I can see any errors.
>>>>>>>>>>>>
>>>>>>>>>>>> <route id="matrix"
>>> errorHandlerRef="matrixDeadLetterErrorHandler">
>>>>>>>>>>>> <from uri="activemq:topic:matrix"/>
>>>>>>>>>>>> <process ref="matrixProcessor" />
>>>>>>>>>>>> <transacted />
>>>>>>>>>>>> <to uri="activemq:topic:edirectory"/>
>>>>>>>>>>>> </route>
>>>>>>>>>>>>
>>>>>>>>>>>> <bean id="jmsConnectionFactory"
>>>>>>> class="org.apache.activemq.ActiveMQConnectionFactory"
>>>>>>> depends-on="matrixBrokerID">
>>>>>>>>>>>> <property name="brokerURL"
>>>>>>> value="vm://matrixBroker?create=false"/>
>>>>>>>>>>>> </bean>
>>>>>>>>>>>>
>>>>>>>>>>>> <bean id="jmsTransactionManager"
>>>>>>> class="org.springframework.jms.connection.JmsTransactionManager">
>>>>>>>>>>>> <property name="connectionFactory"
>>>>>>> ref="jmsConnectionFactory"/>
>>>>>>>>>>>> </bean>
>>>>>>>>>>>>
>>>>>>>>>>>> <bean id="activemq"
>>>>>>> class="org.apache.activemq.camel.component.ActiveMQComponent">
>>>>>>>>>>>> <property name="connectionFactory"
>>>>>>> ref="jmsConnectionFactory"/>
>>>>>>>>>>>> <property name="transacted" value="true"/>
>>>>>>>>>>>> <property name="transactionManager"
>>>>>>> ref="jmsTransactionManager"/>
>>>>>>>>>>>> </bean>
>>>>>>>>>>>>
>>>>>>>>>>>> <bean id="matrixDeadLetterErrorHandler"
>>>>>>> class="org.apache.camel.builder.DeadLetterChannelBuilder">
>>>>>>>>>>>> <property name="deadLetterUri" value="jms:queue:dead"/>
>>>>>>>>>>>> <property name="redeliveryPolicy"
>>>>>>> ref="matrixRedeliveryPolicyConfig"/>
>>>>>>>>>>>> </bean>
>>>>>>>>>>>>
>>>>>>>>>>>> <bean id="matrixRedeliveryPolicyConfig"
>>>>>>> class="org.apache.camel.processor.RedeliveryPolicy">
>>>>>>>>>>>> <property name="maximumRedeliveries" value="10"/>
>>>>>>>>>>>> <property name="redeliveryDelay" value="250"/>
>>>>>>>>>>>> </bean>
>>>>>>>>>>>>
>>>>>>>>>>>> thanks,
>>>>>>>>>>>>
>>>>>>>>>>>> Alistair
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --------------
>>>>>>>>>>>> mov eax,1
>>>>>>>>>>>> mov ebx,0
>>>>>>>>>>>> int 80
>>>>>>>>>>>>
>>>>>>>>>>>> On 29 Sep 2011, at 08:53, Claus Ibsen wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>
>>>>>>>>>>>>> Do you use message expiry?
>>>>>>>>>>>>> Make sure clocks between server/clients is synced as much as
>>>>>>> possible.
>>>>>>>>>>>>>
>>>>>>>>>>>>> There is a timestamp plugin
>>>>>>>>>>>>> http://activemq.apache.org/timestampplugin.html
>>>>>>>>>>>>>
>>>>>>>>>>>>> And do you use queue or topic.
>>>>>>>>>>>>> What version of AMQ and Camel are you using?
>>>>>>>>>>>>> And how have you configured the AMQ broker, and the Camel
>>> context?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <
>>> taar...@gmail.com>
>>>>>>> wrote:
>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Where the logs go, if it's logged at all, still depends on your
>>>>>>> logger and
>>>>>>>>>>>>>> how you configured it.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Here are links to how to enable logging[1] and camel logging
>>> FAQ[2]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
>>>>>>>>>>>>>> [2]http://camel.apache.org/logging-questions.html
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Taariq
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <
>>>>>>> alistair.yo...@uhi.ac.uk>wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> which is the best trace method to use? trace="true", or
>>>>>>> camelTracer and
>>>>>>>>>>>>>>> traceFormatter beans? and where does the log end up? I've
>>> tried
>>>>>>> them all but
>>>>>>>>>>>>>>> no log appears.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> mov eax,1
>>>>>>>>>>>>>>> mov ebx,0
>>>>>>>>>>>>>>> int 80h
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 28 Sep 2011, at 12:08, Marco Westermann wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I suggest enable tracing to see exactly what happens in your
>>>>>>> route.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> regards, Marco
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Am 28.09.2011 13:01, schrieb Alistair Young:
>>>>>>>>>>>>>>>>> I now have a dead letter channel which is empty after losing
>>> 9
>>>>>>> out of 10
>>>>>>>>>>>>>>> messages. I also added a logging handler which logged nothing.
>>>>>>> Verified the
>>>>>>>>>>>>>>> messages arrived at the broker, then they just vanished.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Claus Ibsen
>>>>>>>>>>>>> -----------------
>>>>>>>>>>>>> FuseSource
>>>>>>>>>>>>> Email: cib...@fusesource.com
>>>>>>>>>>>>> Web: http://fusesource.com
>>>>>>>>>>>>> Twitter: davsclaus, fusenews
>>>>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Claus Ibsen
>>>>>>>>>>> -----------------
>>>>>>>>>>> FuseSource
>>>>>>>>>>> Email: cib...@fusesource.com
>>>>>>>>>>> Web: http://fusesource.com
>>>>>>>>>>> Twitter: davsclaus, fusenews
>>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>>>
>
