I think this way madness lies.
17 sent to topicA, dispatchCount = 15, dequeueCount = 12
topicB enqueueCount = 12
so 17 came in, 12 made it through, of the 5 that went missing it claims to have
sent 3 to topicB but they never arrived and the last 2 just simply vanished
completely.
What on earth?
Alistair
--------------
mov eax,1
mov ebx,0
int 80
On 29 Sep 2011, at 15:41, Alistair Young wrote:
> nup - cleaned out the data dir and restarted the broker. First message in
> vanished. Wasn't persisted. So something is fundamentally broken.
>
> topicA inflightCount = dispatchCount = enqueueCount = 1
> topicB is completely empty
>
> so the message wasn't persisted, wasn't processed, wasn't routed and just
> vanished from the broker.
>
> Alistair
>
> --------------
> mov eax,1
> mov ebx,0
> int 80
>
> On 29 Sep 2011, at 15:13, Alistair Young wrote:
>
>> route goes from topicA -> topicB, transacted.
>> topicA inflightCount = 96 and increases on each batch of incoming messages
>> topicB dispatchCount = enqueueCount
>>
>> wondering if the missing messages are connected to topicA inflightCount. I
>> noticed there are two consumers for topicB. The main consumer gets its
>> messages fine. Wonder if the second consumer is a durable topic consumer and
>> therefore activemq is persisting its messages but it hasn't connected in a
>> very long time. Would that cause the topic to get too big? i.e. messages go
>> into the topic until the limit is reached. Main consumer pulls messages off
>> and messages are able to go onto topicB again. Before consumer pulls and
>> after limit reached, messages can't get from topicA -> topicB, hence the
>> topicA inflightCount not zero?
>>
>> Alistair
>>
>> --------------
>> mov eax,1
>> mov ebx,0
>> int 80
>>
>> On 29 Sep 2011, at 12:17, Tim wrote:
>>
>>> Sorry you might have tried this since I haven't been following this thread.
>>> But can you check your jmx console.
>>> In particular check 2 things.. the route to see if the number of exchanges
>>> match what you think and how if any exchanges failed.
>>> Also check the JMX console on activemq for the queue or topic in question
>>> and see how many were enqueued vs dispatched.
>>> Check your deadletter queue from there too
>>>
>>> On Thu, Sep 29, 2011 at 12:52 PM, Alistair Young
>>> <alistair.yo...@uhi.ac.uk>wrote:
>>>
>>>> dunno - nothing works. Random messages are just vanishing once they reach
>>>> the broker. No trace, no logs, no dead letter queue. Just vanishing. I've
>>>> removed <transacted /> and <process> but it doesn't help. The producer is a
>>>> few secs behind the broker:
>>>>
>>>> sent : 11:25:26
>>>> arrived : 11:24:57
>>>> timstamp on message : 1317291897071 = 29 Sep 2011 10:24:57 GMT, presumably
>>>> the timestampplugin doing this
>>>> message vanishes
>>>>
>>>> but all messages display this clock behaviour and not all vanish.
>>>>
>>>> Alistair
>>>>
>>>> --------------
>>>> mov eax,1
>>>> mov ebx,0
>>>> int 80
>>>>
>>>> On 29 Sep 2011, at 10:24, Alistair Young wrote:
>>>>
>>>>> just saw your info about transacted being before from - will change that
>>>> and monitor again.
>>>>>
>>>>> thanks,
>>>>>
>>>>> Alistair
>>>>>
>>>>> --------------
>>>>> mov eax,1
>>>>> mov ebx,0
>>>>> int 80
>>>>>
>>>>> On 29 Sep 2011, at 10:18, Alistair Young wrote:
>>>>>
>>>>>> just noticed a batch of identical 5 messages, three were missing and
>>>> another single message vanished. tracer logged nothing. No errors, dead
>>>> letter queue empty.
>>>>>>
>>>>>> One thing that happens is another machine polls the stats topic in
>>>> activemq every 2mins. Would that cause a problem? It asks for stats on the
>>>> matrix topic, which is part of the transacted route.
>>>>>>
>>>>>> 29 September 2011 10:05:07 - Adding destination :
>>>> Topic:ActiveMQ.Advisory.Connection
>>>>>> 29 September 2011 10:05:07 - Creating new transaction with name [null]:
>>>> PROPAGATION_REQUIRED,ISOLATION_DEFAULT
>>>>>> 29 September 2011 10:05:07 - Stopping connection:
>>>> vm://matrixBroker#285916
>>>>>> 29 September 2011 10:05:07 - Stopped transport: vm://matrixBroker#285916
>>>>>> 29 September 2011 10:05:07 - Connection Stopped:
>>>> vm://matrixBroker#285916
>>>>>> 29 September 2011 10:05:07 - Setting up new connection id:
>>>> ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address:
>>>> vm://matrixBroker#285920
>>>>>> 29 September 2011 10:05:07 - Adding Connection : ConnectionInfo
>>>> {commandId = 1, responseRequired = true, connectionId =
>>>> ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId =
>>>> ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName = null,
>>>> password = *****, brokerPath = null, brokerMasterConnector = false,
>>>> manageable = true, clientMaster = true, faultTolerant = false}
>>>>>>
>>>>>> Alistair
>>>>>>
>>>>>> --------------
>>>>>> mov eax,1
>>>>>> mov ebx,0
>>>>>> int 80
>>>>>>
>>>>>> On 29 Sep 2011, at 09:36, Alistair Young wrote:
>>>>>>
>>>>>>>> <transacted/> Should be after <from>
>>>>>>> it is after from - do you mean it should be before?
>>>>>>> <route id="eDirSuccessBroadcast">
>>>>>>> <from uri="activemq:topic:edirectoryprocessed"/>
>>>>>>> <transacted />
>>>>>>> <process ref="groupwiseProcessor" />
>>>>>>> <to uri="activemq:topic:blackboard"/>
>>>>>>> </route>
>>>>>>>
>>>>>>> thanks for the dead letter tips, will apply them.
>>>>>>>
>>>>>>> Alistair
>>>>>>>
>>>>>>> --------------
>>>>>>> mov eax,1
>>>>>>> mov ebx,0
>>>>>>> int 80
>>>>>>>
>>>>>>> On 29 Sep 2011, at 09:20, Claus Ibsen wrote:
>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> <transacted/> Should be after <from>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
>>>>>>>> <alistair.yo...@uhi.ac.uk> wrote:
>>>>>>>>>> Do you use message expiry?
>>>>>>>>> no
>>>>>>>>>
>>>>>>>>>> timestamp plugin
>>>>>>>>>
>>>>>>>>> using that
>>>>>>>>>
>>>>>>>>> activemq 5.5.0
>>>>>>>>> camel 2.8.0
>>>>>>>>> spring 3.0.5
>>>>>>>>>
>>>>>>>>> noticed sl4j errors on startup, fixed that and now the tracer is
>>>> logging so hopefully I can see any errors.
>>>>>>>>>
>>>>>>>>> <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
>>>>>>>>> <from uri="activemq:topic:matrix"/>
>>>>>>>>> <process ref="matrixProcessor" />
>>>>>>>>> <transacted />
>>>>>>>>> <to uri="activemq:topic:edirectory"/>
>>>>>>>>> </route>
>>>>>>>>>
>>>>>>>>> <bean id="jmsConnectionFactory"
>>>> class="org.apache.activemq.ActiveMQConnectionFactory"
>>>> depends-on="matrixBrokerID">
>>>>>>>>> <property name="brokerURL"
>>>> value="vm://matrixBroker?create=false"/>
>>>>>>>>> </bean>
>>>>>>>>>
>>>>>>>>> <bean id="jmsTransactionManager"
>>>> class="org.springframework.jms.connection.JmsTransactionManager">
>>>>>>>>> <property name="connectionFactory"
>>>> ref="jmsConnectionFactory"/>
>>>>>>>>> </bean>
>>>>>>>>>
>>>>>>>>> <bean id="activemq"
>>>> class="org.apache.activemq.camel.component.ActiveMQComponent">
>>>>>>>>> <property name="connectionFactory"
>>>> ref="jmsConnectionFactory"/>
>>>>>>>>> <property name="transacted" value="true"/>
>>>>>>>>> <property name="transactionManager"
>>>> ref="jmsTransactionManager"/>
>>>>>>>>> </bean>
>>>>>>>>>
>>>>>>>>> <bean id="matrixDeadLetterErrorHandler"
>>>> class="org.apache.camel.builder.DeadLetterChannelBuilder">
>>>>>>>>> <property name="deadLetterUri" value="jms:queue:dead"/>
>>>>>>>>> <property name="redeliveryPolicy"
>>>> ref="matrixRedeliveryPolicyConfig"/>
>>>>>>>>> </bean>
>>>>>>>>>
>>>>>>>>> <bean id="matrixRedeliveryPolicyConfig"
>>>> class="org.apache.camel.processor.RedeliveryPolicy">
>>>>>>>>> <property name="maximumRedeliveries" value="10"/>
>>>>>>>>> <property name="redeliveryDelay" value="250"/>
>>>>>>>>> </bean>
>>>>>>>>>
>>>>>>>>> thanks,
>>>>>>>>>
>>>>>>>>> Alistair
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --------------
>>>>>>>>> mov eax,1
>>>>>>>>> mov ebx,0
>>>>>>>>> int 80
>>>>>>>>>
>>>>>>>>> On 29 Sep 2011, at 08:53, Claus Ibsen wrote:
>>>>>>>>>
>>>>>>>>>> Hi
>>>>>>>>>>
>>>>>>>>>> Do you use message expiry?
>>>>>>>>>> Make sure clocks between server/clients is synced as much as
>>>> possible.
>>>>>>>>>>
>>>>>>>>>> There is a timestamp plugin
>>>>>>>>>> http://activemq.apache.org/timestampplugin.html
>>>>>>>>>>
>>>>>>>>>> And do you use queue or topic.
>>>>>>>>>> What version of AMQ and Camel are you using?
>>>>>>>>>> And how have you configured the AMQ broker, and the Camel context?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <taar...@gmail.com>
>>>> wrote:
>>>>>>>>>>> Hi
>>>>>>>>>>>
>>>>>>>>>>> Where the logs go, if it's logged at all, still depends on your
>>>> logger and
>>>>>>>>>>> how you configured it.
>>>>>>>>>>>
>>>>>>>>>>> Here are links to how to enable logging[1] and camel logging FAQ[2]
>>>>>>>>>>>
>>>>>>>>>>> [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
>>>>>>>>>>> [2]http://camel.apache.org/logging-questions.html
>>>>>>>>>>>
>>>>>>>>>>> Taariq
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <
>>>> alistair.yo...@uhi.ac.uk>wrote:
>>>>>>>>>>>
>>>>>>>>>>>> which is the best trace method to use? trace="true", or
>>>> camelTracer and
>>>>>>>>>>>> traceFormatter beans? and where does the log end up? I've tried
>>>> them all but
>>>>>>>>>>>> no log appears.
>>>>>>>>>>>>
>>>>>>>>>>>> Alistair
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> mov eax,1
>>>>>>>>>>>> mov ebx,0
>>>>>>>>>>>> int 80h
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 28 Sep 2011, at 12:08, Marco Westermann wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I suggest enable tracing to see exactly what happens in your
>>>> route.
>>>>>>>>>>>>>
>>>>>>>>>>>>> regards, Marco
>>>>>>>>>>>>>
>>>>>>>>>>>>> Am 28.09.2011 13:01, schrieb Alistair Young:
>>>>>>>>>>>>>> I now have a dead letter channel which is empty after losing 9
>>>> out of 10
>>>>>>>>>>>> messages. I also added a logging handler which logged nothing.
>>>> Verified the
>>>>>>>>>>>> messages arrived at the broker, then they just vanished.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Claus Ibsen
>>>>>>>>>> -----------------
>>>>>>>>>> FuseSource
>>>>>>>>>> Email: cib...@fusesource.com
>>>>>>>>>> Web: http://fusesource.com
>>>>>>>>>> Twitter: davsclaus, fusenews
>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Claus Ibsen
>>>>>>>> -----------------
>>>>>>>> FuseSource
>>>>>>>> Email: cib...@fusesource.com
>>>>>>>> Web: http://fusesource.com
>>>>>>>> Twitter: davsclaus, fusenews
>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>
>
