Hi, Every once in a while, we get the following errors from the supervisor log. When it happens, they appear in bunches, consistently.
* Read a frame size of xxx, which is bigger than the maximum allowable buffer size for ALL connections. * Read an invalid frame size of -xxx. Are you using TFramedTransport on the client side? * Read an invalid frame size of 0. Are you using TFramedTransport on the client side? Following these errors, we see the following types of stack traces appearing. We have upgraded from Storm 1.2.1 and are currently running Storm 2.4.0. As per our understanding, this error is related to the supervisor.thrift.max_buffer_size config. From our investigation, in Storm 1, this config was set to use Long.MAX_VALUE, when storm 1 was using thrift 0.9.3. See thrift/AbstractNonblockingServer.java at 0.9.3 * apache/thrift (github.com)<https://github.com/apache/thrift/blob/0.9.3/lib/java/src/org/apache/thrift/server/AbstractNonblockingServer.java#L52-L76>. But this was changed to 256 MB when Storm 2 was upgraded to use thrift 0.13.0. See thrift/AbstractNonblockingServer.java at v0.13.0 * apache/thrift (github.com)<https://github.com/apache/thrift/blob/v0.13.0/lib/java/src/org/apache/thrift/server/AbstractNonblockingServer.java#L52-L76>. Storm 2 defaults this to 1MB. See https://github.com/apache/storm/blob/a432e99bca526886655cc1d5b2453a09b302b5ca/conf/defaults.yaml#L190. We have tried to use the default of 1MB as well as 256MB. However, this leads to this error occurring very frequently. So we increased it to 1GB and it doesn't occur as often, although it still does. We have tried to increase this further by setting it to 2GB, but that seems to have caused the supervisor service to be constantly restarted, causing workers to not even spin up. So we have settled on 1GB. Can someone please help us better understand what is going on here and how we can address this issue? o.a.s.t.s.AbstractNonblockingServer$FrameBuffer pool-11-thread-6 [WARN] Exception while invoking! org.apache.storm.thrift.transport.TTransportException: Cannot read. Remote side has closed. Tried to read 4 bytes, but only got 0 bytes. (This is often indicative of an internal error on the server side. Please check your server logs.) at org.apache.storm.thrift.transport.TTransport.readAll(TTransport.java:88) ~[storm-shaded-deps-2.4.0.jar:2.4.0] at org.apache.storm.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:141) ~[storm-shaded-deps-2.4.0.jar:2.4.0] at org.apache.storm.thrift.transport.TFramedTransport.read(TFramedTransport.java:109) ~[storm-shaded-deps-2.4.0.jar:2.4.0] at org.apache.storm.thrift.transport.TTransport.readAll(TTransport.java:86) ~[storm-shaded-deps-2.4.0.jar:2.4.0] at org.apache.storm.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:455) ~[storm-shaded-deps-2.4.0.jar:2.4.0] at org.apache.storm.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:354) ~[storm-shaded-deps-2.4.0.jar:2.4.0] at org.apache.storm.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:243) ~[storm-shaded-deps-2.4.0.jar:2.4.0] at org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:27) ~[storm-shaded-deps-2.4.0.jar:2.4.0] at org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:172) ~[storm-client-2.4.0.jar:2.4.0] at org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:524) [storm-shaded-deps-2.4.0.jar:2.4.0] at org.apache.storm.thrift.server.Invocation.run(Invocation.java:18) [storm-shaded-deps-2.4.0.jar:2.4.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_332] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_332] at java.lang.Thread.run(Thread.java:750) [?:1.8.0_332] o.a.s.t.s.AbstractNonblockingServer$FrameBuffer pool-11-thread-12 [WARN] Exception while invoking! org.apache.storm.thrift.protocol.TProtocolException: Bad version in readMessageBegin at org.apache.storm.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:247) ~[storm-shaded-deps-2.4.0.jar:2.4.0] at org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:27) ~[storm-shaded-deps-2.4.0.jar:2.4.0] at org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:172) ~[storm-client-2.4.0.jar:2.4.0] at org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:524) [storm-shaded-deps-2.4.0.jar:2.4.0] at org.apache.storm.thrift.server.Invocation.run(Invocation.java:18) [storm-shaded-deps-2.4.0.jar:2.4.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_332] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_332] at java.lang.Thread.run(Thread.java:750) [?:1.8.0_332] o.a.s.t.s.AbstractNonblockingServer$FrameBuffer pool-11-thread-15 [WARN] Exception while invoking! org.apache.storm.thrift.protocol.TProtocolException: Length exceeded max allowed: 364060257 at org.apache.storm.thrift.protocol.TBinaryProtocol.checkStringReadLength(TBinaryProtocol.java:438) ~[storm-shaded-deps-2.4.0.jar:2.4.0] at org.apache.storm.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:409) ~[storm-shaded-deps-2.4.0.jar:2.4.0] at org.apache.storm.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:254) ~[storm-shaded-deps-2.4.0.jar:2.4.0] at org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:27) ~[storm-shaded-deps-2.4.0.jar:2.4.0] at org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:172) ~[storm-client-2.4.0.jar:2.4.0] at org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:524) [storm-shaded-deps-2.4.0.jar:2.4.0] at org.apache.storm.thrift.server.Invocation.run(Invocation.java:18) [storm-shaded-deps-2.4.0.jar:2.4.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_332] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_332] at java.lang.Thread.run(Thread.java:750) [?:1.8.0_332] o.a.s.t.s.THsHaServer Thread-5 [ERROR] run() exiting due to uncaught error java.lang.OutOfMemoryError: Java heap space at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) ~[?:1.8.0_332] at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[?:1.8.0_332] at org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.read(AbstractNonblockingServer.java:371) ~[storm-shaded-deps-2.4.0.jar:2.4.0] at org.apache.storm.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:203) ~[storm-shaded-deps-2.4.0.jar:2.4.0] at org.apache.storm.thrift.server.TNonblockingServer$SelectAcceptThread.select(TNonblockingServer.java:203) ~[storm-shaded-deps-2.4.0.jar:2.4.0] at org.apache.storm.thrift.server.TNonblockingServer$SelectAcceptThread.run(TNonblockingServer.java:154) [storm-shaded-deps-2.4.0.jar:2.4.0] Thanks, Binh ________________________________ This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. Your privacy is important to us. Accenture uses your personal data only in compliance with data protection laws. For further information on how Accenture processes your personal data, please see our privacy statement at https://www.accenture.com/us-en/privacy-policy. ______________________________________________________________________________________ www.accenture.com
