Do you have a good reason to have networkTTL="5", if there is only one
broker, the default value of 1 will be fine.
On 11 May 2012 16:32, Chris Robison <chrisdrobi...@gmail.com> wrote:
> If it helps, I've attached the configuration files for both machines.
>
>
> On Fri, May 11, 2012 at 9:07 AM, Chris Robison <chrisdrobi...@gmail.com>
> wrote:
>>
>> One thing I have noticed though is that when I restart the broker on the
>> other end, everything starts to work again until it floods again.
>>
>>
>> On Fri, May 11, 2012 at 9:05 AM, Chris Robison <chrisdrobi...@gmail.com>
>> wrote:
>>>
>>> I'm on 5.6. And all brokers remain up and running.
>>>
>>>
>>> On Fri, May 11, 2012 at 8:48 AM, Gary Tully <gary.tu...@gmail.com> wrote:
>>>>
>>>> what version are you on?
>>>> Is there any chance that the broker at the other end of the network
>>>> bridge is shutting down?
>>>>
>>>> On 11 May 2012 14:34, Chris Robison <chrisdrobi...@gmail.com> wrote:
>>>> > I have a network of brokers and I am using a request-reply system
>>>> > across
>>>> > that network, but I've noticed that after a few request-replys, the
>>>> > broker
>>>> > network will get flooded with TempQueue advisory messages, then some
>>>> > of the
>>>> > of the brokers will be reported as slow consumers and the consumers
>>>> > won't
>>>> > receive messages after that. The advisory messages seem to be repeated
>>>> > (e.g. add temp destination, remove temp destination) for the same temp
>>>> > destinations. Not quite sure how to handle this. Any suggestions?
>>>> >
>>>> > Here are some of the errors are other things I'm seeing repeatedly in
>>>> > logs
>>>> > when I enable debug. In fact, when this flood occurs, tons of log gets
>>>> > generated all at once.
>>>> >
>>>> > 2012-05-11 05:24:37,248 | DEBUG | Error occured while processing async
>>>> > command: DestinationInfo {commandId = 40578, responseRequired = false,
>>>> > connectionId = ID:MSTMIP102-59588-1336713700699-3:2, destination =
>>>> > temp-queue://ID:POTMIP102-54337-634722930065323786-1:18:1,
>>>> > operationType =
>>>> > 0, timeout = 0, brokerPath = [ID:MSSD5D101-38580-1336708634042-0:1]},
>>>> > exception: java.lang.IllegalStateException: Timer already cancelled. |
>>>> > org.apache.activemq.broker.TransportConnection.Service | ActiveMQ
>>>> > Transport: ssl:///10.60.252.28:54493
>>>> > java.lang.IllegalStateException: Timer already cancelled.
>>>> > at java.util.Timer.sched(Timer.java:397)
>>>> > at java.util.Timer.schedule(Timer.java:248)
>>>> > at
>>>> >
>>>> > org.apache.activemq.thread.Scheduler.schedualPeriodically(Scheduler.java:49)
>>>> > at org.apache.activemq.broker.region.Queue.start(Queue.java:889)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:143)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:289)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>>>> > at
>>>> >
>>>> > org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:161)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>>>> > at
>>>> >
>>>> > org.apache.activemq.security.AuthorizationBroker.addDestination(AuthorizationBroker.java:81)
>>>> > at
>>>> >
>>>> > org.apache.activemq.security.AuthorizationBroker.addDestinationInfo(AuthorizationBroker.java:51)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.MutableBrokerFilter.addDestinationInfo(MutableBrokerFilter.java:223)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.TransportConnection.processAddDestination(TransportConnection.java:477)
>>>> > at
>>>> >
>>>> > org.apache.activemq.command.DestinationInfo.visit(DestinationInfo.java:122)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:150)
>>>> > at
>>>> >
>>>> > org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
>>>> > at
>>>> >
>>>> > org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
>>>> > at
>>>> >
>>>> > org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:135)
>>>> > at
>>>> >
>>>> > org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:124)
>>>> > at
>>>> > org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:103)
>>>> > at
>>>> >
>>>> > org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
>>>> > at
>>>> >
>>>> > org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
>>>> > at
>>>> >
>>>> > org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteConsumerAdvisory(DemandForwardingBridgeSupport.java:588)
>>>> > at
>>>> >
>>>> > org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:420)
>>>> > at
>>>> >
>>>> > org.apache.activemq.network.DemandForwardingBridgeSupport$2.onCommand(DemandForwardingBridgeSupport.java:151)
>>>> > at
>>>> >
>>>> > org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
>>>> > at
>>>> >
>>>> > org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
>>>> > at
>>>> >
>>>> > org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
>>>> > at
>>>> >
>>>> > org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:229)
>>>> > at
>>>> >
>>>> > org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
>>>> > at
>>>> >
>>>> > org.apache.activemq.transport.tcp.SslTransport.doConsume(SslTransport.java:91)
>>>> > at
>>>> >
>>>> > org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:222)
>>>> > at
>>>> > org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:204)
>>>> >
>>>> >
>>>> >
>>>> > 2012-05-11 05:24:37,248 | WARN | Async error occurred:
>>>> > java.lang.IllegalStateException: Timer already cancelled. |
>>>> > org.apache.activemq.broker.TransportConnection.Service | ActiveMQ
>>>> > Transport: ssl:///10.60.252.28:54493
>>>> > java.lang.IllegalStateException: Timer already cancelled.
>>>> > at java.util.Timer.sched(Timer.java:397)
>>>> > at java.util.Timer.schedule(Timer.java:248)
>>>> > at
>>>> >
>>>> > org.apache.activemq.thread.Scheduler.schedualPeriodically(Scheduler.java:49)
>>>> > at org.apache.activemq.broker.region.Queue.start(Queue.java:889)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:143)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:289)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>>>> > at
>>>> >
>>>> > org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:161)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>>>> > at
>>>> >
>>>> > org.apache.activemq.security.AuthorizationBroker.addDestination(AuthorizationBroker.java:81)
>>>> > at
>>>> >
>>>> > org.apache.activemq.security.AuthorizationBroker.addDestinationInfo(AuthorizationBroker.java:51)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.MutableBrokerFilter.addDestinationInfo(MutableBrokerFilter.java:223)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.TransportConnection.processAddDestination(TransportConnection.java:477)
>>>> > at
>>>> >
>>>> > org.apache.activemq.command.DestinationInfo.visit(DestinationInfo.java:122)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:150)
>>>> > at
>>>> >
>>>> > org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
>>>> > at
>>>> >
>>>> > org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
>>>> > at
>>>> >
>>>> > org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:135)
>>>> > at
>>>> >
>>>> > org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:124)
>>>> > at
>>>> > org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:103)
>>>> > at
>>>> >
>>>> > org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
>>>> > at
>>>> >
>>>> > org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
>>>> > at
>>>> >
>>>> > org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteConsumerAdvisory(DemandForwardingBridgeSupport.java:588)
>>>> > at
>>>> >
>>>> > org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:420)
>>>> > at
>>>> >
>>>> > org.apache.activemq.network.DemandForwardingBridgeSupport$2.onCommand(DemandForwardingBridgeSupport.java:151)
>>>> > at
>>>> >
>>>> > org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
>>>> > at
>>>> >
>>>> > org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
>>>> > at
>>>> >
>>>> > org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
>>>> > at
>>>> >
>>>> > org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:229)
>>>> > at
>>>> >
>>>> > org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
>>>> > at
>>>> >
>>>> > org.apache.activemq.transport.tcp.SslTransport.doConsume(SslTransport.java:91)
>>>> > at
>>>> >
>>>> > org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:222)
>>>> > at
>>>> > org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:204)
>>>> >
>>>> > 2012-05-11 05:24:37,249 | DEBUG | mstmip102.vosh.net adding
>>>> > destination:
>>>> > topic://ActiveMQ.Advisory.Connection |
>>>> > org.apache.activemq.broker.region.AbstractRegion | StopAsync:vm://
>>>> > mstmip102.vosh.net#0-23
>>>> > 2012-05-11 05:24:37,250 | WARN | Failed to remove connection
>>>> > ConnectionInfo {commandId = 3, responseRequired = false, connectionId
>>>> > =
>>>> > ID:msipap102-50908-1336690011745-7:1, clientId =
>>>> > mstmip102_msipap102.vosh.net_outbound, clientIp =
>>>> > vm://mstmip102.vosh.net#0,
>>>> > userName = mqipauser, password = *****, brokerPath = null,
>>>> > brokerMasterConnector = false, manageable = false, clientMaster =
>>>> > true,
>>>> > faultTolerant = false, failoverReconnect = false}, reason:
>>>> > java.lang.IllegalStateException: Timer already cancelled. |
>>>> > org.apache.activemq.broker.TransportConnection.Service |
>>>> > StopAsync:vm://
>>>> > mstmip102.vosh.net#0-23
>>>> > 2012-05-11 05:24:37,250 | DEBUG | Exception detail: |
>>>> > org.apache.activemq.broker.TransportConnection.Service |
>>>> > StopAsync:vm://
>>>> > mstmip102.vosh.net#0-23
>>>> > java.lang.IllegalStateException: Timer already cancelled.
>>>> > at java.util.Timer.sched(Timer.java:397)
>>>> > at java.util.Timer.schedule(Timer.java:248)
>>>> > at
>>>> >
>>>> > org.apache.activemq.thread.Scheduler.schedualPeriodically(Scheduler.java:49)
>>>> > at org.apache.activemq.broker.region.Topic.start(Topic.java:530)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:143)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:286)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>>>> > at
>>>> >
>>>> > org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:161)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>>>> > at
>>>> >
>>>> > org.apache.activemq.security.AuthorizationBroker.addDestination(AuthorizationBroker.java:81)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:481)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:305)
>>>> > at
>>>> >
>>>> > org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:533)
>>>> > at
>>>> >
>>>> > org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:464)
>>>> > at
>>>> >
>>>> > org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:459)
>>>> > at
>>>> >
>>>> > org.apache.activemq.advisory.AdvisoryBroker.removeConnection(AdvisoryBroker.java:234)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.BrokerFilter.removeConnection(BrokerFilter.java:113)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.TransactionBroker.removeConnection(TransactionBroker.java:351)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.BrokerFilter.removeConnection(BrokerFilter.java:113)
>>>> > at
>>>> >
>>>> > org.apache.activemq.security.JaasAuthenticationBroker.removeConnection(JaasAuthenticationBroker.java:95)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.BrokerFilter.removeConnection(BrokerFilter.java:113)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.MutableBrokerFilter.removeConnection(MutableBrokerFilter.java:119)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:766)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1068)
>>>> > at
>>>> >
>>>> > org.apache.activemq.broker.TransportConnection$4.run(TransportConnection.java:996)
>>>> > at java.lang.Thread.run(Thread.java:722)
>>>> >
>>>> > 2012-05-11 05:24:37,252 | DEBUG | mstmip102.vosh.net adding
>>>> > destination:
>>>> > temp-queue://ID:POTMIP102-54337-634722930065323786-1:18:1 |
>>>> > org.apache.activemq.broker.region.AbstractRegion | ActiveMQ Transport:
>>>> > ssl:///172.17.5.78:59658
>>>> >
>>>> > 2012-05-11 05:24:37,341 | DEBUG | bridging (mstmip102.vosh.net ->
>>>> > msipap101.vosh.net) ID:MSTMIP102-59588-1336713700699-1:1:0:0:25184,
>>>> > consumer: ID:MSTMIP102-59588-1336713700699-4:2:1:1, destination
>>>> > topic://ActiveMQ.Advisory.TempQueue, brokerPath:
>>>> > [ID:MSTMIP102-59588-1336713700699-0:1], message: ActiveMQMessage
>>>> > {commandId
>>>> > = 0, responseRequired = false, messageId =
>>>> > ID:MSTMIP102-59588-1336713700699-1:1:0:0:25184, originalDestination =
>>>> > null,
>>>> > originalTransactionId = null, producerId =
>>>> > ID:MSTMIP102-59588-1336713700699-4:1:1:1, destination =
>>>> > topic://ActiveMQ.Advisory.TempQueue, transactionId = null, expiration
>>>> > = 0,
>>>> > timestamp = 0, arrival = 0, brokerInTime = 1336713871588,
>>>> > brokerOutTime =
>>>> > 1336713877341, correlationId = null, replyTo = null, persistent =
>>>> > false,
>>>> > type = Advisory, priority = 0, groupID = null, groupSequence = 0,
>>>> > targetConsumerId = null, compressed = false, userID = null, content =
>>>> > null,
>>>> > marshalledProperties = org.apache.activemq.util.ByteSequence@567edec8,
>>>> > dataStructure = DestinationInfo {commandId = 0, responseRequired =
>>>> > false,
>>>> > connectionId = ID:MSTMIP102-59588-1336713700699-3:2, destination =
>>>> > temp-queue://ID:POTMIP102-54337-634722930065323786-1:18:1,
>>>> > operationType =
>>>> > 0, timeout = 0, brokerPath = null}, redeliveryCounter = 0, size =
>>>> > 1024,
>>>> > properties = {originBrokerURL=ssl://MSTMIP102:61616,
>>>> > originBrokerId=ID:MSTMIP102-59588-1336713700699-0:1, originBrokerName=
>>>> > mstmip102.vosh.net}, readOnlyProperties = false, readOnlyBody = false,
>>>> > droppable = false} |
>>>> > org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ
>>>> > Connection Dispatcher: vm://mstmip102.vosh.net#4
>>>> >
>>>> > 2012-05-11 05:24:37,341 | DEBUG | bridging (mstmip102.vosh.net ->
>>>> > msipap101.vosh.net) ID:MSTMIP102-59588-1336713700699-1:1:0:0:25185,
>>>> > consumer: ID:MSTMIP102-59588-1336713700699-4:2:1:1, destination
>>>> > topic://ActiveMQ.Advisory.TempQueue, brokerPath:
>>>> > [ID:MSTMIP102-59588-1336713700699-0:1], message: ActiveMQMessage
>>>> > {commandId
>>>> > = 0, responseRequired = false, messageId =
>>>> > ID:MSTMIP102-59588-1336713700699-1:1:0:0:25185, originalDestination =
>>>> > null,
>>>> > originalTransactionId = null, producerId =
>>>> > ID:MSTMIP102-59588-1336713700699-4:1:1:1, destination =
>>>> > topic://ActiveMQ.Advisory.TempQueue, transactionId = null, expiration
>>>> > = 0,
>>>> > timestamp = 0, arrival = 0, brokerInTime = 1336713871588,
>>>> > brokerOutTime =
>>>> > 1336713877341, correlationId = null, replyTo = null, persistent =
>>>> > false,
>>>> > type = Advisory, priority = 0, groupID = null, groupSequence = 0,
>>>> > targetConsumerId = null, compressed = false, userID = null, content =
>>>> > null,
>>>> > marshalledProperties = org.apache.activemq.util.ByteSequence@235448d5,
>>>> > dataStructure = DestinationInfo {commandId = 0, responseRequired =
>>>> > false,
>>>> > connectionId = ID:MSTMIP102-59588-1336713700699-3:2, destination =
>>>> > temp-queue://ID:POTMIP102-54337-634722930065323786-1:18:1,
>>>> > operationType =
>>>> > 1, timeout = 0, brokerPath = null}, redeliveryCounter = 0, size =
>>>> > 1024,
>>>> > properties = {originBrokerURL=ssl://MSTMIP102:61616,
>>>> > originBrokerId=ID:MSTMIP102-59588-1336713700699-0:1, originBrokerName=
>>>> > mstmip102.vosh.net}, readOnlyProperties = false, readOnlyBody = false,
>>>> > droppable = false} |
>>>> > org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ
>>>> > Connection Dispatcher: vm://mstmip102.vosh.net#4
>>>> >
>>>> > 2012-05-11 13:29:40,053 | WARN | TopicSubscription:
>>>> > consumer=ID:MSTMIP102-32947-1336741838889-4:2:1:1, destinations=7,
>>>> > dispatched=1000, delivered=7723, matched=1001, discarded=0: has twice
>>>> > its
>>>> > prefetch limit pending, without an ack; it appears to be slow |
>>>> > org.apache.activemq.broker.region.TopicSubscription | ActiveMQ
>>>> > Transport:
>>>> > ssl:///10.60.252.28:54614
>>>> >
>>>> > 2012-05-11 13:29:42,540 | WARN | TopicSubscription:
>>>> > consumer=ID:MSTMIP102-32947-1336741838889-3:2:1:1, destinations=6,
>>>> > dispatched=1000, delivered=9922, matched=1001, discarded=0: has twice
>>>> > its
>>>> > prefetch limit pending, without an ack; it appears to be slow |
>>>> > org.apache.activemq.broker.region.TopicSubscription | ActiveMQ
>>>> > Transport:
>>>> > ssl:///10.60.252.28:54614
>>>>
>>>>
>>>>
>>>> --
>>>> http://fusesource.com
>>>> http://blog.garytully.com
>>>
>>>
>>
>
--
http://fusesource.com
http://blog.garytully.com
