Re: Queue Recovery loads the entire data?
Jason, you should try removing the changes you made to your configuration and verify if the correct driver is auto-detected. You should see one of the following log lines at startup: try { adapter = finder.newInstance(dirverName); LOG.info("*Database "** + kind + **" driver override recognized for *: [" + dirverName + "] - adapter: " + adapter.getClass()); } catch (Throwable e) { LOG.info("*Database "** + kind + **" driver override not found for* : [" + dirverName + "]. Will use default implementation."); } On Wed, Feb 7, 2018 at 8:31 AM, jasons wrote: > I couldn't see the limitQuery() support in the DefaultJDBCAdapter of > AMQ5.10, > but it did arrive in AMQ5.13. Our system is currently on 5.10 so we > upgraded a test system to 5.13.4. I updated the jdbcPersistenceAdapter > element to look like this... > > > adapter="oracleJDBCAdapter" dataSource="#dataSource" > transactionIsolation="2" useDatabaseLock="false" /> > > > ..but encountered this exception on startup: > > Initialization of bean failed; nested exception is > org.springframework.beans.ConversionNotSupportedException: Failed to > convert > property value of type 'java.lang.String' to required type > 'org.apache.activemq.store.jdbc.JDBCAdapter' for property 'adapter'; > nested > exception is java.lang.IllegalStateException: Cannot convert value of type > [java.lang.String] to required type > [org.apache.activemq.store.jdbc.JDBCAdapter] for property 'adapter': no > matching editors or conversion strategy found > at > org.springframework.beans.factory.support.BeanDefinitionValueResolver. > resolveInnerBean(BeanDefinitionValueResolver.java:290) > [spring-beans.jar:4.0.6.RELEASE] > at > org.springframework.beans.factory.support.BeanDefinitionValueResolver. > resolveValueIfNecessary(BeanDefinitionValueResolver.java:129) > [spring-beans.jar:4.0.6.RELEASE] > at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFac > tory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1456) > [spring-beans.jar:4.0.6.RELEASE] > at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFac > tory.populateBean(AbstractAutowireCapableBeanFactory.java:1197) > [spring-beans.jar:4.0.6.RELEASE] > at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFac > tory.doCreateBean(AbstractAutowireCapableBeanFactory.java:537) > [spring-beans.jar:4.0.6.RELEASE] > at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFac > tory.createBean(AbstractAutowireCapableBeanFactory.java:475) > [spring-beans.jar:4.0.6.RELEASE] > > Can anyone give some pointers on what I'm doing wrong? Or did I > misunderstand is the "adapter" attribute not yet implemented? > > > > -- > Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User- > f2341805.html >
Re: Queue Recovery loads the entire data?
I couldn't see the limitQuery() support in the DefaultJDBCAdapter of AMQ5.10, but it did arrive in AMQ5.13. Our system is currently on 5.10 so we upgraded a test system to 5.13.4. I updated the jdbcPersistenceAdapter element to look like this... ..but encountered this exception on startup: Initialization of bean failed; nested exception is org.springframework.beans.ConversionNotSupportedException: Failed to convert property value of type 'java.lang.String' to required type 'org.apache.activemq.store.jdbc.JDBCAdapter' for property 'adapter'; nested exception is java.lang.IllegalStateException: Cannot convert value of type [java.lang.String] to required type [org.apache.activemq.store.jdbc.JDBCAdapter] for property 'adapter': no matching editors or conversion strategy found at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:290) [spring-beans.jar:4.0.6.RELEASE] at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:129) [spring-beans.jar:4.0.6.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1456) [spring-beans.jar:4.0.6.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1197) [spring-beans.jar:4.0.6.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:537) [spring-beans.jar:4.0.6.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475) [spring-beans.jar:4.0.6.RELEASE] Can anyone give some pointers on what I'm doing wrong? Or did I misunderstand is the "adapter" attribute not yet implemented? -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Queue Recovery loads the entire data?
Nathan, I do have access to Oracle in dev/test so might be able to help with your line of enquiry if you want to share some further thoughts and/or you think it might expedite. Cheers all and thanks for sharing. We really appreciate the work of all the good people in this community. -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Queue Recovery loads the entire data?
Got it. I'll try it today. -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Queue Recovery loads the entire data?
We're working on what sounds like the same issue Jason. We have not confirmed that we can force AMQ to use the OracleJDBCAdapter but that is our current line. I'm also investigating why detection is not working. The driver name hasn't changed so it's not obvious. I have a lead I'm running out right now. Unfortunately we don't have Oracle in Dev or Test so I'm limited in what I can do. On Mon, Feb 5, 2018 at 8:51 AM, Tim Bain wrote: > Jason, > > No, that's not relevant. The question is not whether the Oracle driver is > used, but whether ActiveMQ recognizes that fact and responds by using its > own Oracle-specific subclass of DefaultJDBCAdapter ( > https://github.com/apache/activemq/blob/master/activemq- > jdbc-store/src/main/java/org/apache/activemq/store/jdbc/ > adapter/OracleJDBCAdapter.java). > You'll see that there's not much to that class, but that it has an > Oracle-specific way to limit the number of results in a response. So if > ActiveMQ didn't realize that it should be using that class, then you might > see the same behavior as nathanwray, and telling ActiveMQ via configuration > that you're using Oracle might result in correct behavior. > > Tim > > On Feb 5, 2018 4:54 AM, "jasons" wrote: > > > Hi Tim, in case you missed it, or in case I'm missing something..but > > doesn't my stack trace confirm Oracle driver selection by the > > oracle.jdbc.driver packages? > > > > Cheers > > > > Jason. > > > > > > > > -- > > Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User- > > f2341805.html > > >
Re: Queue Recovery loads the entire data?
Jason, No, that's not relevant. The question is not whether the Oracle driver is used, but whether ActiveMQ recognizes that fact and responds by using its own Oracle-specific subclass of DefaultJDBCAdapter ( https://github.com/apache/activemq/blob/master/activemq-jdbc-store/src/main/java/org/apache/activemq/store/jdbc/adapter/OracleJDBCAdapter.java). You'll see that there's not much to that class, but that it has an Oracle-specific way to limit the number of results in a response. So if ActiveMQ didn't realize that it should be using that class, then you might see the same behavior as nathanwray, and telling ActiveMQ via configuration that you're using Oracle might result in correct behavior. Tim On Feb 5, 2018 4:54 AM, "jasons" wrote: > Hi Tim, in case you missed it, or in case I'm missing something..but > doesn't my stack trace confirm Oracle driver selection by the > oracle.jdbc.driver packages? > > Cheers > > Jason. > > > > -- > Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User- > f2341805.html >
Re: Queue Recovery loads the entire data?
Hi Tim, in case you missed it, or in case I'm missing something..but doesn't my stack trace confirm Oracle driver selection by the oracle.jdbc.driver packages? Cheers Jason. -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Queue Recovery loads the entire data?
Jason, Look at the current thread with user nathanwray. He saw behavior that sounded the same as what you're seeing, which he found was caused by the ActiveMQ code failing to detect that his JDBC driver was an Oracle driver. He was able to work around the problem by explicitly setting the database type in his activemq.xml file. Can you test the same workaround and let us know if it works for you? Thanks, Tim On Jan 31, 2018 12:15 AM, "jasons" wrote: > Hi, I have a customer running AMQ 5.10.0, JDBC persistence and no > journalling > like the OP. We saw the same query as the OP last week..also during a > restart of the JBoss app server running AMQ embedded. Oracle AWR reports > showed this SQL was executed 14,000 times in 1 hour, effectively grinding > the DB to a stallunfortunately also impacting other apps on the same > instance. The broker ultimately then failed to the first instance owing > to > the time to start exceeding the JBoss deployment timeout limit (which we > had > to subsequently increase). > > Has anyone offered any good suggestions to your dilema? Our customer's > ACTIVEMQ_MSGS table cardinality matching the "WHERE CONTAINER =" predicate > was at 300,000 records so its little wonder the query performs like a dog > on > each execution. jstacks captured during one of the lengthy startup > attempts > looked like this: > "Broker Starting Thread" prio=10 tid=0x7fdd38a1b000 nid=0x3bd6 runnable > [0x7fdd70e65000] >java.lang.Thread.State: RUNNABLE > at java.net.SocketInputStream.socketRead0(Native Method) > at java.net.SocketInputStream.read(SocketInputStream.java:152) > at java.net.SocketInputStream.read(SocketInputStream.java:122) > at oracle.net.ns.Packet.receive(Packet.java:300) > at oracle.net.ns.DataPacket.receive(DataPacket.java:106) > at > oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315) > at oracle.net.ns.NetInputStream.read(NetInputStream.java:260) > at oracle.net.ns.NetInputStream.read(NetInputStream.java:185) > at oracle.net.ns.NetInputStream.read(NetInputStream.java:102) > at > oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket( > T4CSocketInputStreamWrapper.java:124) > at > oracle.jdbc.driver.T4CSocketInputStreamWrapper.read( > T4CSocketInputStreamWrapper.java:80) > at > oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137) > at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290) > at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192) > at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531) > at > oracle.jdbc.driver.T4CPreparedStatement.doOall8( > T4CPreparedStatement.java:207) > at > oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe( > T4CPreparedStatement.java:884) > at > oracle.jdbc.driver.OracleStatement.executeMaybeDescribe( > OracleStatement.java:1167) > at > oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout( > OracleStatement.java:1289) > at > oracle.jdbc.driver.OraclePreparedStatement.executeInternal( > OraclePreparedStatement.java:3584) > at > oracle.jdbc.driver.OraclePreparedStatement.executeQuery( > OraclePreparedStatement.java:3628) > - locked <0x000666a73b80> (a oracle.jdbc.driver.T4CConnection) > at > oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery( > OraclePreparedStatementWrapper.java:1493) > at > org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeQuery( > WrappedPreparedStatement.java:462) > at > org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter. > doGetDurableSubscriberMessageCount(DefaultJDBCAdapter.java:681) > at > org.apache.activemq.store.jdbc.JDBCTopicMessageStore.getMessageCount( > JDBCTopicMessageStore.java:392) > at > org.apache.activemq.store.ProxyTopicMessageStore.getMessageCount( > ProxyTopicMessageStore.java:140) > at > org.apache.activemq.broker.region.cursors.TopicStorePrefetch.getStoreSize( > TopicStorePrefetch.java:88) > - locked <0x000652a9bc60> (a > org.apache.activemq.broker.region.cursors.TopicStorePrefetch) > at > org.apache.activemq.broker.region.cursors.AbstractStoreCursor.resetSize( > AbstractStoreCursor.java:68) > at > org.apache.activemq.broker.region.cursors.TopicStorePrefetch.( > TopicStorePrefetch.java:56) > at > org.apache.activemq.broker.region.cursors.StoreDurableSubscriberCursor. > add(StoreDurableSubscriberCursor.java:119) > - locked <0x000652a66f38> (a > org.apache.activemq.broker.region.cursors.StoreDurableSubscriberCursor) > at > org.apache.activemq.broker.region.PrefetchSubscription. > add(PrefetchSubscription.java:589) > - locked <0x000652a5ef98> (a java.lang.Object) > at > org.apache.activemq.broker.region.DurableTopicSubscription.add( > DurableTopicSubscription.java:111) > at > org.apache.
Re: Queue Recovery loads the entire data?
Hi, I have a customer running AMQ 5.10.0, JDBC persistence and no journalling like the OP. We saw the same query as the OP last week..also during a restart of the JBoss app server running AMQ embedded. Oracle AWR reports showed this SQL was executed 14,000 times in 1 hour, effectively grinding the DB to a stallunfortunately also impacting other apps on the same instance. The broker ultimately then failed to the first instance owing to the time to start exceeding the JBoss deployment timeout limit (which we had to subsequently increase). Has anyone offered any good suggestions to your dilema? Our customer's ACTIVEMQ_MSGS table cardinality matching the "WHERE CONTAINER =" predicate was at 300,000 records so its little wonder the query performs like a dog on each execution. jstacks captured during one of the lengthy startup attempts looked like this: "Broker Starting Thread" prio=10 tid=0x7fdd38a1b000 nid=0x3bd6 runnable [0x7fdd70e65000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at oracle.net.ns.Packet.receive(Packet.java:300) at oracle.net.ns.DataPacket.receive(DataPacket.java:106) at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315) at oracle.net.ns.NetInputStream.read(NetInputStream.java:260) at oracle.net.ns.NetInputStream.read(NetInputStream.java:185) at oracle.net.ns.NetInputStream.read(NetInputStream.java:102) at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124) at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80) at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137) at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290) at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192) at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531) at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207) at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:884) at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1167) at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1289) at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3584) at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3628) - locked <0x000666a73b80> (a oracle.jdbc.driver.T4CConnection) at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1493) at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:462) at org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doGetDurableSubscriberMessageCount(DefaultJDBCAdapter.java:681) at org.apache.activemq.store.jdbc.JDBCTopicMessageStore.getMessageCount(JDBCTopicMessageStore.java:392) at org.apache.activemq.store.ProxyTopicMessageStore.getMessageCount(ProxyTopicMessageStore.java:140) at org.apache.activemq.broker.region.cursors.TopicStorePrefetch.getStoreSize(TopicStorePrefetch.java:88) - locked <0x000652a9bc60> (a org.apache.activemq.broker.region.cursors.TopicStorePrefetch) at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.resetSize(AbstractStoreCursor.java:68) at org.apache.activemq.broker.region.cursors.TopicStorePrefetch.(TopicStorePrefetch.java:56) at org.apache.activemq.broker.region.cursors.StoreDurableSubscriberCursor.add(StoreDurableSubscriberCursor.java:119) - locked <0x000652a66f38> (a org.apache.activemq.broker.region.cursors.StoreDurableSubscriberCursor) at org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:589) - locked <0x000652a5ef98> (a java.lang.Object) at org.apache.activemq.broker.region.DurableTopicSubscription.add(DurableTopicSubscription.java:111) at org.apache.activemq.broker.region.Topic.addSubscription(Topic.java:160) at org.apache.activemq.broker.region.TopicRegion.addSubscriptionsForDestination(TopicRegion.java:256) at org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:144) at org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:325) at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:167) at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:167) at org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:184) at org.apache.activemq.broker.Br
Re: Queue Recovery loads the entire data?
FWIW 4.2 does this paging to/from disk. On 5/22/07, Pandey <[EMAIL PROTECTED]> wrote: Hi, I am using JDBCPersistence (without journaling) in my product. There are times when : - my broker goes down - there are already some messages in the queue as they were not processed yet. - my broker comes up again and restores the queue(s) On the third step, it seems to be executing a simple 'select id,msg from activemq_msgs where container = ? orderby id' query to get ALL the enqueued data. This results in the loading of ENTIRE Data on my VM. IS there a way to let that NOT happen? Is there a way to make the adapter read queue data in chunks? (maybe ... 100 messages at a time .. read the next 100 when 90 of the first 100 have been processed) ? Any pointers would certainly help! Thanks! Vipul -- View this message in context: http://www.nabble.com/Queue-Recovery-loads-the-entire-data--tf3793599s2354.html#a10729587 Sent from the ActiveMQ - User mailing list archive at Nabble.com. -- James --- http://macstrac.blogspot.com/