-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
I think that eliminates my hypothesis.
Are you running with an absolutely bog-standard Messaging module config? Is the
logging you've shown direct from the log or have you "grep"ed out the ActiveMQ
related parts?
The delay appears to occur between AMQ starting up and adding destinations
("localhost adding destination: topic://fedora.apim.update") which makes me
wonder if something other than ActiveMQ startup code is causing a delay (e.g.
if there is some kind of obscure condition between GSearch and Fedora).
- ---
A. Soroka
Software & Systems Engineering :: Online Library Environment
the University of Virginia Library
On Aug 13, 2012, at 12:38 PM, André Lapointe wrote:
> Here is a ActiveMQ log at startup (almost 5 minutes of hang this morning) :
>
> INFO 2012-08-13 11:23:50.545 [main] (org.fcrepo.server.Server) Loading bean
> definitions for org.fcrepo.server.storage.DefaultExternalContentManager
> DEBUG 2012-08-13 11:23:51.033 [main]
> (org.apache.activemq.broker.jmx.ManagementContext) Probably not using JRE
> 1.4: mx4j.tools.naming.NamingService
> DEBUG 2012-08-13 11:23:51.037 [JMX connector]
> (org.apache.activemq.broker.jmx.ManagementContext) Starting
> JMXConnectorServer...
> INFO 2012-08-13 11:23:51.088 [JMX connector]
> (org.apache.activemq.broker.jmx.ManagementContext) JMX consoles can connect
> to service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
> INFO 2012-08-13 11:23:51.125 [main]
> (org.apache.activemq.store.kahadb.plist.PListStore)
> PListStore:/usr/local/fedora/data/activemq-data/localhost/tmp_storage started
> INFO 2012-08-13 11:23:51.127 [main]
> (org.apache.activemq.broker.BrokerService) Using Persistence Adapter:
> KahaDBPersistenceAdapter[/usr/local/fedora/data/activemq-data/localhost/KahaDB]
> INFO 2012-08-13 11:23:51.689 [main]
> (org.apache.activemq.store.kahadb.MessageDatabase) KahaDB is version 3
> INFO 2012-08-13 11:23:51.709 [main]
> (org.apache.activemq.store.kahadb.MessageDatabase) Recovering from the
> journal ...
> INFO 2012-08-13 11:23:51.709 [main]
> (org.apache.activemq.store.kahadb.MessageDatabase) Recovery replayed 2
> operations from the journal in 0.011 seconds.
> INFO 2012-08-13 11:23:51.717 [main]
> (org.apache.activemq.broker.BrokerService) ActiveMQ 5.5.0 JMS Message Broker
> (localhost) is starting
> INFO 2012-08-13 11:23:51.717 [main]
> (org.apache.activemq.broker.BrokerService) For help or more information
> please see: http://activemq.apache.org/
> DEBUG 2012-08-13 11:28:39.938 [main]
> (org.apache.activemq.broker.region.AbstractRegion) localhost adding
> destination: topic://fedora.apim.update
> DEBUG 2012-08-13 11:28:39.976 [main]
> (org.apache.activemq.broker.region.TopicRegion) Restoring durable
> subscription: SubscriptionInfo {subscribedDestination =
> topic://fedora.apim.update, destination = topic://fedora.apim.update,
> clientId = fedoragsearch0, subscriptionName = fedora.apim.update, selector =
> null}
> DEBUG 2012-08-13 11:28:40.017 [main]
> (org.apache.activemq.broker.TransportConnector) Publishing:
> tcp://localhost:61616 for broker transport URI: tcp://localhost:61616
> DEBUG 2012-08-13 11:28:40.017 [main]
> (org.apache.activemq.broker.region.AbstractRegion) localhost adding
> destination: topic://ActiveMQ.Advisory.Topic
> TRACE 2012-08-13 11:28:40.025 [main]
> (org.apache.activemq.broker.jmx.ManagedRegionBroker) Sub with key:
> fedoragsearch0:fedora.apim.update, already registered
> INFO 2012-08-13 11:28:40.030 [main]
> (org.apache.activemq.transport.TransportServerThreadSupport) Listening for
> connections at: tcp://localhost:61616
> INFO 2012-08-13 11:28:40.030 [main]
> (org.apache.activemq.broker.TransportConnector) Connector
> tcp://localhost:61616 Started
> INFO 2012-08-13 11:28:40.032 [main]
> (org.apache.activemq.broker.BrokerService) ActiveMQ JMS Message Broker
> (localhost, ID:corpus.ato.uqam.ca-42176-1344871431734-0:1) started
> DEBUG 2012-08-13 11:28:40.033 [main]
> (org.apache.activemq.transport.vm.VMTransportFactory) binding to broker:
> localhost
> INFO 2012-08-13 11:28:40.035 [main]
> (org.apache.activemq.broker.TransportConnector) Connector vm://localhost
> Started
> DEBUG 2012-08-13 11:28:40.075 [ActiveMQ Task-2]
> (org.apache.activemq.broker.TransportConnection) Setting up new connection
> id: ID:corpus.ato.uqam.ca-42176-1344871431734-2:1, address: vm://localhost#0
> DEBUG 2012-08-13 11:28:40.075 [ActiveMQ Task-2]
> (org.apache.activemq.broker.region.AbstractRegion) localhost adding
> destination: topic://ActiveMQ.Advisory.Connection
> DEBUG 2012-08-13 11:28:40.089 [ActiveMQ Task-3]
> (org.apache.activemq.broker.region.AbstractRegion) localhost adding consumer:
> ID:corpus.ato.uqam.ca-42176-1344871431734-2:1:-1:1 for destination:
> ActiveMQ.Advisory.TempQueue,ActiveMQ.Advisory.TempTopic
>
> André
>
> On Mon, Aug 13, 2012 at 12:27 PM, [email protected] <[email protected]>
> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Is there any logging coming out of ActiveMQ right before the section you've
> shown us? I'm wondering if it is having any difficulty rebuilding the KahaDB
> store on which it relies...
>
> - ---
> A. Soroka
> Software & Systems Engineering :: Online Library Environment
> the University of Virginia Library
>
> On Aug 13, 2012, at 12:18 PM, André Lapointe wrote:
>
> > Hi,
> >
> > Thanks Scott.
> >
> > My Fedora installation does not use queues it uses topics for messaging
> > (default installation). I don't have anything stuck anywhere, I have check
> > with a JMX console.
> >
> > GSearch is subscribed to these topics and it receives messages correctly.
> > The topics are empty afterwards.
> >
> > Any other idea ?
> >
> > André
> >
> > On Fri, Aug 10, 2012 at 4:06 PM, Scott Prater <[email protected]> wrote:
> > Hello, André,
> >
> > The fact that the amount of time that Fedora takes to start up is
> > variable suggests to me that ActiveMQ is trying to resend messages it
> > has stuck in its queue. Few messages, quick startup, lots of messages,
> > slow startup. You could verify that by deleting the messages in the
> > queue, shutting down Fedora, starting it back up again, and seeing if it
> > starts immediately; then perform a number of API-M actions (ingests,
> > updates, etc.), examining the state of the queue, shutting down Fedora,
> > starting it up again, and see if it takes a while.
> >
> > See http://activemq.apache.org/how-do-i-purge-a-queue.html for
> > information on purging a queue.
> >
> > Once you've verified that is, in fact, the problem, then you'll need to
> > see why messages are getting stuck in the queue: perhaps no configured
> > message consumer?
> >
> > -- Scott
> >
> > On 08/10/2012 02:15 PM, André Lapointe wrote:
> > > Hi,
> > >
> > > On Tomcat startup, our fedora installation hangs for a few minutes,
> > > sometimes for as long as an hour. It seems that ActiveMQ is the problem
> > > because when its turned off, the startup executes in a few second.
> > >
> > > I have turn on TRACE logging for ActiveMQ and here is the log traces
> > > before and after the hangup :
> > >
> > > INFO 2012-08-10 10:34:44.059 [main]
> > > (org.apache.activemq.store.kahadb.MessageDatabase) Recovery replayed 2
> > > operations from the journal in 0.017 seconds.
> > > INFO 2012-08-10 10:34:44.065 [main]
> > > (org.apache.activemq.broker.BrokerService) ActiveMQ 5.5.0 JMS Message
> > > Broker (localhost) is starting
> > > INFO 2012-08-10 10:34:44.065 [main]
> > > (org.apache.activemq.broker.BrokerService) For help or more information
> > > please see: http://activemq.apache.org/
> > > DEBUG 2012-08-10 10:45:28.699 [main]
> > > (org.apache.activemq.broker.region.AbstractRegion) localhost adding
> > > destination: topic://fedora.apim.update
> > > DEBUG 2012-08-10 10:45:28.736 [main]
> > > (org.apache.activemq.broker.region.TopicRegion) Restoring durable
> > > subscription: SubscriptionInfo {subscribedDestination =
> > > topic://fedora.apim.update, destination = topic://fedora.apim.update,
> > > clientId = fedoragsearch0, subscriptionName = fedora.apim.update,
> > > selector = null}
> > > DEBUG 2012-08-10 10:45:28.776 [main]
> > > (org.apache.activemq.broker.TransportConnector) Publishing:
> > > tcp://localhost:61616 for broker transport URI: tcp://localhost:61616
> > > DEBUG 2012-08-10 10:45:28.777 [main]
> > > (org.apache.activemq.broker.region.AbstractRegion) localhost adding
> > > destination: topic://ActiveMQ.Advisory.Topic
> > > TRACE 2012-08-10 10:45:28.785 [main]
> > > (org.apache.activemq.broker.jmx.ManagedRegionBroker) Sub with key:
> > > fedoragsearch0:fedora.apim.update, already registered
> > > INFO 2012-08-10 10:45:28.790 [main]
> > > (org.apache.activemq.transport.TransportServerThreadSupport) Listening
> > > for connections at: tcp://localhost:61616
> > >
> > > In this case it hung for a little bit more than 10 minutes. I've been
> > > looking for a solution all day long and I can't find anything. Any idea ?
> > >
> > > Also, I have tried to remove everything in
> > > $FEDORA_HOME/data/activemq-data/localhost/KahaDB/ and it rebuilds itself
> > > at startup fast. Where does the info is taken to rebuild the activemq
> > > data ? In the FOXML ?
> > >
> > > Thanks,
> > >
> > > André
> > >
> > >
> > >
> > >
> > >
> > > ------------------------------------------------------------------------------
> > > Live Security Virtual Conference
> > > Exclusive live event will cover all the ways today's security and
> > > threat landscape has changed and how IT managers can respond. Discussions
> > > will include endpoint security, mobile security and the latest in malware
> > > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> > >
> > >
> > >
> > > _______________________________________________
> > > Fedora-commons-users mailing list
> > > [email protected]
> > > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
> >
> >
> > --
> > Scott Prater
> > Shared Development Group
> > General Library System
> > University of Wisconsin - Madison
> > [email protected]
> > 5-5415
> >
> >
> > ------------------------------------------------------------------------------
> > Live Security Virtual Conference
> > Exclusive live event will cover all the ways today's security and
> > threat landscape has changed and how IT managers can respond. Discussions
> > will include endpoint security, mobile security and the latest in malware
> > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> > _______________________________________________
> > Fedora-commons-users mailing list
> > [email protected]
> > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
> >
> > ------------------------------------------------------------------------------
> > Live Security Virtual Conference
> > Exclusive live event will cover all the ways today's security and
> > threat landscape has changed and how IT managers can respond. Discussions
> > will include endpoint security, mobile security and the latest in malware
> > threats.
> > http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/_______________________________________________
> > Fedora-commons-users mailing list
> > [email protected]
> > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
> Comment: GPGTools - http://gpgtools.org
>
> iQEcBAEBAgAGBQJQKSrzAAoJEATpPYSyaoIks1YH+wWyUWTcBiYYA3Hita4s1PV9
> 4mZatEonfa4jfZp0WtNVZwsDRmiAcgTbenLJwTkx6BbLHt2C+ufM5o2IgHygkhKZ
> Wl8ZB75f5AMI7idngD4YXA3k3J95NUmwvtrRxMmcaivV3j5i8fYHkbol3xA4QAM+
> UQywoKeDP4wZolpGZrIRWUu39UiaEvQvbBqaHCLk9koiB2dpRwTLvYq5NAYW3s00
> uenSKJW+znL7aOI4Tb251zmFmTCk1TkTQI6oeypmlTNm4vvr5tsS8B9MRT+HxMTB
> EtM2FT3PGuLBwFqzmos4UCNoMYcTPv/atvjOt09vcjeIM4qezbz76yiaYAWhy3M=
> =flZW
> -----END PGP SIGNATURE-----
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Fedora-commons-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats.
> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/_______________________________________________
> Fedora-commons-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
iQEcBAEBAgAGBQJQKTn1AAoJEATpPYSyaoIk0moIAIVczAvOaqDFihrciTfhavHk
p5X2zdOD50EpUnEqMl4MGt+Dl3kJuRj3I2kMYH7fmMaiK1KZME2NiEVCSiM9MuQm
paB5OlBUyPSxggcqrTbucVJKlPK9iQbbJHpUt7Df7Rj+rnqGn9hm0ji79WXa80tu
Tj19BCaWjl8MfGkT/IglRbe/Qn5aE2T5A1UGIVoMP0jxs2J1Jy9rDtf0nK7k1aPz
LFvBooAGG2eNCZjH/4gGzrfwDS91FdHaLuF4d4ozOaGwY98ycE1vA+QnbT/oD/Ig
2815iLBoMRmyZ7s1VW+ToTi62qcYaJfvZjx8OhRrWBzaU4MAFde2CRoUfTGr/80=
=ZTxi
-----END PGP SIGNATURE-----
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Fedora-commons-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
