The message size actually doesn’t matter at all. The problem is simply calling updateBlob when the connection string contains replication. It appears that they have fixed very similar looking issues recently so I’m hopeful it won’t take long on their side.
From: Vilius Šumskas <[email protected]> Date: Thursday, September 15, 2022 at 3:14 PM To: [email protected] <[email protected]> Subject: RE: Critical error sending large messages to mysql Hi, how large is the message? Have you checked max_allowed_packet configuration parameter, specifically https://dev.mysql.com/doc/mysql-replication-excerpt/8.0/en/replication-features-max-allowed-packet.html ? -- Vilius -----Original Message----- From: Stephen Baker <[email protected]> Sent: Thursday, September 15, 2022 8:31 PM To: [email protected] Subject: Re: Critical error sending large messages to mysql To follow up, I have tracked this down to a bug in connector/j. I am working on a simple proof of concept and a ticket for them now. For any artemis mysql users that are curious, the problem only happens when using a jdbc:mysql:replication:// connection string. It seems that updateBlob is broken in that case. From: Stephen Baker <[email protected]> Date: Thursday, September 15, 2022 at 10:57 AM To: [email protected] <[email protected]> Subject: Critical error sending large messages to mysql I don’t have exact reproduction steps yet (otherwise I would have filed an issue), but on Artemis 2.22 using a mysql backed journal our QA can reliably send messages that crash the server: ``` 2022-09-15 10:42:24,843 WARN [org.apache.activemq.artemis.journal] AMQ142021: Error on IO callback, class com.sun.proxy.$Proxy21 cannot be cast to class com.mysql.cj.jdbc.ClientPreparedStatement (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement are in unnamed module of loader java.net.URLClassLoader @3a5ed7a6) 2022-09-15 10:42:24,845 ERROR [org.apache.activemq.artemis.core.server] AMQ222010: Critical IO Error, shutting down the server. file=org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile@3452c05f, message=Error writing to JDBC file.: java.lang.ClassCastException: class com.sun.proxy.$Proxy21 cannot be cast to class com.mysql.cj.jdbc.ClientPreparedStatement (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement are in unnamed module of loader java.net.URLClassLoader @3a5ed7a6) at com.mysql.cj.jdbc.result.UpdatableResultSet.syncUpdate(UpdatableResultSet.java:1143) [mysql-connector-java-8.0.28.jar:8.0.28] at com.mysql.cj.jdbc.result.UpdatableResultSet.updateBlob(UpdatableResultSet.java:1283) [mysql-connector-java-8.0.28.jar:8.0.28] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [java.base:] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [java.base:] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [java.base:] at java.base/java.lang.reflect.Method.invoke(Method.java:566) [java.base:] at com.mysql.cj.jdbc.ha.MultiHostConnectionProxy$JdbcInterfaceProxy.invoke(MultiHostConnectionProxy.java:107) [mysql-connector-java-8.0.28.jar:8.0.28] at com.sun.proxy.$Proxy22.updateBlob(Unknown Source) at org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411) [commons-dbcp2-2.7.0.jar:2.7.0] at org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411) [commons-dbcp2-2.7.0.jar:2.7.0] at org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411) [commons-dbcp2-2.7.0.jar:2.7.0] at org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFileFactoryDriver.writeToFile(JDBCSequentialFileFactoryDriver.java:284) [artemis-jdbc-store-2.22.0.jar:2.22.0] at org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.internalWrite(JDBCSequentialFile.java:167) [artemis-jdbc-store-2.22.0.jar:2.22.0] at org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.internalWrite(JDBCSequentialFile.java:203) [artemis-jdbc-store-2.22.0.jar:2.22.0] at org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.lambda$scheduleWrite$1(JDBCSequentialFile.java:214) [artemis-jdbc-store-2.22.0.jar:2.22.0] at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.22.0.jar:] at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.22.0.jar:] at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65) [artemis-commons-2.22.0.jar:] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [java.base:] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [java.base:] at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.22.0.jar:] 2022-09-15 10:42:24,848 WARN [org.apache.activemq.artemis.journal] AMQ142021: Error on IO callback, Error writing to JDBC file. 2022-09-15 10:42:24,854 WARN [org.apache.activemq.artemis.core.client] AMQ212004: Failed to connect to server. 2022-09-15 10:42:24,909 WARN [org.apache.activemq.artemis.journal] AMQ142021: Error on IO callback, class com.sun.proxy.$Proxy21 cannot be cast to class com.mysql.cj.jdbc.ClientPreparedStatement (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement are in unnamed module of loader java.net.URLClassLoader @3a5ed7a6) 2022-09-15 10:42:24,926 WARN [org.apache.activemq.artemis.journal] AMQ142021: Error on IO callback, Error writing to JDBC file. ``` As you can see, whatever the original problem was it is hidden by the ClassCastException in Artemis. When I learn more I will file an issue (unless the problem has been resolved in a later version) and hopefully a fix. Stephen E. Baker [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe. [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
