What would the critical analyzer do if it couldn’t write to the large-message or page store now? I didn’t think the restart was that out of line, if a little hopeless in this case.
From: Clebert Suconic <[email protected]> Date: Friday, September 16, 2022 at 2:38 PM To: [email protected] <[email protected]> Subject: Re: Critical error sending large messages to mysql I fixed some critical analyzer issues. You should upgrade. Or just turn off the critical analyzer. On Thu, Sep 15, 2022 at 5:01 PM Stephen Baker < [email protected]> wrote: > The message size actually doesn’t matter at all. The problem is simply > calling updateBlob when the connection string contains replication. It > appears that they have fixed very similar looking issues recently so I’m > hopeful it won’t take long on their side. > > From: Vilius Šumskas <[email protected]> > Date: Thursday, September 15, 2022 at 3:14 PM > To: [email protected] <[email protected]> > Subject: RE: Critical error sending large messages to mysql > Hi, > > how large is the message? Have you checked max_allowed_packet > configuration parameter, specifically > https://dev.mysql.com/doc/mysql-replication-excerpt/8.0/en/replication-features-max-allowed-packet.html > ? > > -- > Vilius > > -----Original Message----- > From: Stephen Baker <[email protected]> > Sent: Thursday, September 15, 2022 8:31 PM > To: [email protected] > Subject: Re: Critical error sending large messages to mysql > > To follow up, I have tracked this down to a bug in connector/j. I am > working on a simple proof of concept and a ticket for them now. > > For any artemis mysql users that are curious, the problem only happens > when using a jdbc:mysql:replication:// connection string. It seems that > updateBlob is broken in that case. > > From: Stephen Baker <[email protected]> > Date: Thursday, September 15, 2022 at 10:57 AM > To: [email protected] <[email protected]> > Subject: Critical error sending large messages to mysql I don’t have exact > reproduction steps yet (otherwise I would have filed an issue), but on > Artemis 2.22 using a mysql backed journal our QA can reliably send messages > that crash the server: > > ``` > 2022-09-15 10:42:24,843 WARN [org.apache.activemq.artemis.journal] > AMQ142021: Error on IO callback, class com.sun.proxy.$Proxy21 cannot be > cast to class com.mysql.cj.jdbc.ClientPreparedStatement > (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement are > in unnamed module of loader java.net.URLClassLoader @3a5ed7a6) > 2022-09-15 10:42:24,845 ERROR [org.apache.activemq.artemis.core.server] > AMQ222010: Critical IO Error, shutting down the server. > file=org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile@3452c05f, > message=Error writing to JDBC file.: java.lang.ClassCastException: class > com.sun.proxy.$Proxy21 cannot be cast to class > com.mysql.cj.jdbc.ClientPreparedStatement (com.sun.proxy.$Proxy21 and > com.mysql.cj.jdbc.ClientPreparedStatement are in unnamed module of loader > java.net.URLClassLoader @3a5ed7a6) > at > com.mysql.cj.jdbc.result.UpdatableResultSet.syncUpdate(UpdatableResultSet.java:1143) > [mysql-connector-java-8.0.28.jar:8.0.28] > at > com.mysql.cj.jdbc.result.UpdatableResultSet.updateBlob(UpdatableResultSet.java:1283) > [mysql-connector-java-8.0.28.jar:8.0.28] > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) [java.base:] > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > [java.base:] > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > [java.base:] > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > [java.base:] > at > com.mysql.cj.jdbc.ha.MultiHostConnectionProxy$JdbcInterfaceProxy.invoke(MultiHostConnectionProxy.java:107) > [mysql-connector-java-8.0.28.jar:8.0.28] > at com.sun.proxy.$Proxy22.updateBlob(Unknown Source) > at > org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411) > [commons-dbcp2-2.7.0.jar:2.7.0] > at > org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411) > [commons-dbcp2-2.7.0.jar:2.7.0] > at > org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411) > [commons-dbcp2-2.7.0.jar:2.7.0] > at > org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFileFactoryDriver.writeToFile(JDBCSequentialFileFactoryDriver.java:284) > [artemis-jdbc-store-2.22.0.jar:2.22.0] > at > org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.internalWrite(JDBCSequentialFile.java:167) > [artemis-jdbc-store-2.22.0.jar:2.22.0] > at > org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.internalWrite(JDBCSequentialFile.java:203) > [artemis-jdbc-store-2.22.0.jar:2.22.0] > at > org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.lambda$scheduleWrite$1(JDBCSequentialFile.java:214) > [artemis-jdbc-store-2.22.0.jar:2.22.0] > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) > [artemis-commons-2.22.0.jar:] > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) > [artemis-commons-2.22.0.jar:] > at > org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65) > [artemis-commons-2.22.0.jar:] > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > [java.base:] > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > [java.base:] > at > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > [artemis-commons-2.22.0.jar:] > > 2022-09-15 10:42:24,848 WARN [org.apache.activemq.artemis.journal] > AMQ142021: Error on IO callback, Error writing to JDBC file. > 2022-09-15 10:42:24,854 WARN [org.apache.activemq.artemis.core.client] > AMQ212004: Failed to connect to server. > 2022-09-15 10:42:24,909 WARN [org.apache.activemq.artemis.journal] > AMQ142021: Error on IO callback, class com.sun.proxy.$Proxy21 cannot be > cast to class com.mysql.cj.jdbc.ClientPreparedStatement > (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement are > in unnamed module of loader java.net.URLClassLoader @3a5ed7a6) > 2022-09-15 10:42:24,926 WARN [org.apache.activemq.artemis.journal] > AMQ142021: Error on IO callback, Error writing to JDBC file. > ``` > As you can see, whatever the original problem was it is hidden by the > ClassCastException in Artemis. > > When I learn more I will file an issue (unless the problem has been > resolved in a later version) and hopefully a fix. > > Stephen E. Baker > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do > not click links or open attachments unless you recognize the sender and > know the content is safe. > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do > not click links or open attachments unless you recognize the sender and > know the content is safe. > -- Clebert Suconic [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
