To be clear, the issue in this case has nothing to do with the critical analyzer.
The problem is with a bug in the MySQL JDBC driver which is generating an error interpreted by the broker as a "critical IO error" (as expected) which causes the broker to shut-down (again, as expected). To resolve the error the MySQL bug needs to be fixed. Justin On Sat, Sep 17, 2022 at 8:39 PM Clebert Suconic <[email protected]> wrote: > The critical analyzer was meant to prevent dead locks in the broker. Say > the broker failed it would kill itself. > > We had a few false positives a few versions ago. > > > Either disable it (which is fine, by setting it to log) > > > The critical analyzer will have no effect out of IO issues. Critical IO > issues will still stop the broker if they happen. The best action is to > disable it. > > > > On Fri, Sep 16, 2022 at 5:24 PM Stephen Baker < > [email protected]> wrote: > > > What would the critical analyzer do if it couldn’t write to the > > large-message or page store now? I didn’t think the restart was that out > of > > line, if a little hopeless in this case. > > > > From: Clebert Suconic <[email protected]> > > Date: Friday, September 16, 2022 at 2:38 PM > > To: [email protected] <[email protected]> > > Subject: Re: Critical error sending large messages to mysql > > I fixed some critical analyzer issues. You should upgrade. > > > > Or just turn off the critical analyzer. > > On Thu, Sep 15, 2022 at 5:01 PM Stephen Baker < > > [email protected]> wrote: > > > > > The message size actually doesn’t matter at all. The problem is simply > > > calling updateBlob when the connection string contains replication. It > > > appears that they have fixed very similar looking issues recently so > I’m > > > hopeful it won’t take long on their side. > > > > > > From: Vilius Šumskas <[email protected]> > > > Date: Thursday, September 15, 2022 at 3:14 PM > > > To: [email protected] <[email protected]> > > > Subject: RE: Critical error sending large messages to mysql > > > Hi, > > > > > > how large is the message? Have you checked max_allowed_packet > > > configuration parameter, specifically > > > > > > https://dev.mysql.com/doc/mysql-replication-excerpt/8.0/en/replication-features-max-allowed-packet.html > > > ? > > > > > > -- > > > Vilius > > > > > > -----Original Message----- > > > From: Stephen Baker <[email protected]> > > > Sent: Thursday, September 15, 2022 8:31 PM > > > To: [email protected] > > > Subject: Re: Critical error sending large messages to mysql > > > > > > To follow up, I have tracked this down to a bug in connector/j. I am > > > working on a simple proof of concept and a ticket for them now. > > > > > > For any artemis mysql users that are curious, the problem only happens > > > when using a jdbc:mysql:replication:// connection string. It seems that > > > updateBlob is broken in that case. > > > > > > From: Stephen Baker <[email protected]> > > > Date: Thursday, September 15, 2022 at 10:57 AM > > > To: [email protected] <[email protected]> > > > Subject: Critical error sending large messages to mysql I don’t have > > exact > > > reproduction steps yet (otherwise I would have filed an issue), but on > > > Artemis 2.22 using a mysql backed journal our QA can reliably send > > messages > > > that crash the server: > > > > > > ``` > > > 2022-09-15 10:42:24,843 WARN [org.apache.activemq.artemis.journal] > > > AMQ142021: Error on IO callback, class com.sun.proxy.$Proxy21 cannot be > > > cast to class com.mysql.cj.jdbc.ClientPreparedStatement > > > (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement > are > > > in unnamed module of loader java.net.URLClassLoader @3a5ed7a6) > > > 2022-09-15 10:42:24,845 ERROR [org.apache.activemq.artemis.core.server] > > > AMQ222010: Critical IO Error, shutting down the server. > > > > > > file=org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile@3452c05f > > , > > > message=Error writing to JDBC file.: java.lang.ClassCastException: > class > > > com.sun.proxy.$Proxy21 cannot be cast to class > > > com.mysql.cj.jdbc.ClientPreparedStatement (com.sun.proxy.$Proxy21 and > > > com.mysql.cj.jdbc.ClientPreparedStatement are in unnamed module of > loader > > > java.net.URLClassLoader @3a5ed7a6) > > > at > > > > > > com.mysql.cj.jdbc.result.UpdatableResultSet.syncUpdate(UpdatableResultSet.java:1143) > > > [mysql-connector-java-8.0.28.jar:8.0.28] > > > at > > > > > > com.mysql.cj.jdbc.result.UpdatableResultSet.updateBlob(UpdatableResultSet.java:1283) > > > [mysql-connector-java-8.0.28.jar:8.0.28] > > > at > > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > > > Method) [java.base:] > > > at > > > > > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > > [java.base:] > > > at > > > > > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > [java.base:] > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > > > [java.base:] > > > at > > > > > > com.mysql.cj.jdbc.ha.MultiHostConnectionProxy$JdbcInterfaceProxy.invoke(MultiHostConnectionProxy.java:107) > > > [mysql-connector-java-8.0.28.jar:8.0.28] > > > at com.sun.proxy.$Proxy22.updateBlob(Unknown Source) > > > at > > > > > > org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411) > > > [commons-dbcp2-2.7.0.jar:2.7.0] > > > at > > > > > > org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411) > > > [commons-dbcp2-2.7.0.jar:2.7.0] > > > at > > > > > > org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411) > > > [commons-dbcp2-2.7.0.jar:2.7.0] > > > at > > > > > > org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFileFactoryDriver.writeToFile(JDBCSequentialFileFactoryDriver.java:284) > > > [artemis-jdbc-store-2.22.0.jar:2.22.0] > > > at > > > > > > org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.internalWrite(JDBCSequentialFile.java:167) > > > [artemis-jdbc-store-2.22.0.jar:2.22.0] > > > at > > > > > > org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.internalWrite(JDBCSequentialFile.java:203) > > > [artemis-jdbc-store-2.22.0.jar:2.22.0] > > > at > > > > > > org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.lambda$scheduleWrite$1(JDBCSequentialFile.java:214) > > > [artemis-jdbc-store-2.22.0.jar:2.22.0] > > > at > > > > > > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) > > > [artemis-commons-2.22.0.jar:] > > > at > > > > > > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) > > > [artemis-commons-2.22.0.jar:] > > > at > > > > > > org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65) > > > [artemis-commons-2.22.0.jar:] > > > at > > > > > > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > > > [java.base:] > > > at > > > > > > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > > [java.base:] > > > at > > > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > [artemis-commons-2.22.0.jar:] > > > > > > 2022-09-15 10:42:24,848 WARN [org.apache.activemq.artemis.journal] > > > AMQ142021: Error on IO callback, Error writing to JDBC file. > > > 2022-09-15 10:42:24,854 WARN [org.apache.activemq.artemis.core.client] > > > AMQ212004: Failed to connect to server. > > > 2022-09-15 10:42:24,909 WARN [org.apache.activemq.artemis.journal] > > > AMQ142021: Error on IO callback, class com.sun.proxy.$Proxy21 cannot be > > > cast to class com.mysql.cj.jdbc.ClientPreparedStatement > > > (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement > are > > > in unnamed module of loader java.net.URLClassLoader @3a5ed7a6) > > > 2022-09-15 10:42:24,926 WARN [org.apache.activemq.artemis.journal] > > > AMQ142021: Error on IO callback, Error writing to JDBC file. > > > ``` > > > As you can see, whatever the original problem was it is hidden by the > > > ClassCastException in Artemis. > > > > > > When I learn more I will file an issue (unless the problem has been > > > resolved in a later version) and hopefully a fix. > > > > > > Stephen E. Baker > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. > Do > > > not click links or open attachments unless you recognize the sender and > > > know the content is safe. > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. > Do > > > not click links or open attachments unless you recognize the sender and > > > know the content is safe. > > > > > -- > > Clebert Suconic > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do > > not click links or open attachments unless you recognize the sender and > > know the content is safe. > > > -- > Clebert Suconic >
