MG>confirm for andor below (with instructions for JMX) ________________________________ From: Shawn Heisey <[email protected]> Sent: Thursday, April 5, 2018 11:58 AM To: [email protected] Subject: Re: Is the current max packet length available via the API?
On 4/5/2018 3:44 AM, Andor Molnar wrote: > You can get the current jute.maxbuffer setting from a running > ZooKeeper instance by querying ZooKeeperServerBean via JMX. I'm not sure how I would do that in a client program. It might be trivial, but it's not something I've ever done. MG>explained here in zkserver.sh # See the following page for extensive details on setting # up the JVM to accept JMX remote management: # http://java.sun.com/javase/6/docs/technotes/guides/management/agent.html # by default we allow local JMX connections if [ "x$JMXLOCALONLY" = "x" ] then JMXLOCALONLY=false fi #JMXDISABLE must not be null if [ "x$JMXDISABLE" = "x" ] then echo "ZooKeeper JMX enabled by default" >&2 if [ "x$JMXPORT" = "x" ] then # for some reason these two options are necessary on jdk6 on Ubuntu # accord to the docs they are not necessary, but otw jconsole cannot # do a local attach ZOOMAIN="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=$JMXLOCALONLY org.apache.zookeeper.server.quorum.QuorumPeerMain" else #AUTH credentials go here if [ "x$JMXAUTH" = "x" ] then JMXAUTH=false fi #FOR SSL/TLS if [ "x$JMXSSL" = "x" ] then JMXSSL=false fi #suggest use default of JMXLOG4J=true if [ "x$JMXLOG4J" = "x" ] then JMXLOG4J=true fi echo "ZooKeeper remote JMX Port set to $JMXPORT" >&2 echo "ZooKeeper remote JMX authenticate set to $JMXAUTH" >&2 echo "ZooKeeper remote JMX ssl set to $JMXSSL" >&2 echo "ZooKeeper remote JMX log4j set to $JMXLOG4J" >&2 ZOOMAIN="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=$JMXPORT -Dcom.sun.management.jmxremote.authenticate=$JMXAUTH -Dcom.sun.management.jmxremote.ssl=$JMXSSL -Dzookeeper.jmx.log4j.disable=$JMXLOG4J org.apache.zookeeper.server.quorum.QuorumPeerMain" fi else echo "JMX disabled by user request" >&2 ZOOMAIN="org.apache.zookeeper.server.quorum.QuorumPeerMain" fi MG>everything you need to setup JMX is located MG>at http://java.sun.com/javase/6/docs/technotes/guides/management/agent.html > Currently there're 2 usage of the setting in ZK: 1) server-client > communication which is by default 4MB, 2) server-server communication > which is by default 1MB. They can't be set individually, but can be > overriden with the jute.maxbuffer system property. I'm looking for a way to ask the ZK client to give me the value it is currently using as its max packet length. I'm only going to be logging a warning to inform the user about which file may have caused a problem due to size, not preventing the attempt at uploading the file, so I'm not opposed to falling back to a hard-coded value if I can't figure it out. I can look for the jute.maxbuffer sysprop, but if ZK will tell me what it's actually using, I'd prefer that. MG>andor is correct packetLen is set in org.apache.zookeeper.ClientCnxn MG> public static final int packetLen = Integer.getInteger("jute.maxbuffer", 0496 * 1024); MG>as packetLen is final... it cannot be overidden to any other value Does the max packet length cover ONLY the size of the znode data, or does the znode name get included in that? Asked another way: Should I subtract a little bit from the max packet length (maybe 128 or 256) before I compare the file size, or just compare the unchanged value? I did discover that the ZkClientConfig.CLIENT_MAX_PACKET_LENGTH_DEFAULT field I mentioned before is not available in 3.4.x, it seems to have been added to a 3.5 version. Since Solr uses 3.4.x and won't upgrade until there is a stable 3.5 release, I can't use that. MG>confirmed CLIENT_MAX_PACKET_LENGTH_DEFAULT is not avail in 3.4.x I do think that the ZK client should log something useful when the max packet length is exceeded -- if that's even possible. MG>happens here in org.apache.zookeeper.ClientCnxnSocket protected void readLength() throws IOException { int len = incomingBuffer.getInt(); if (len < 0 || len >= ClientCnxn.packetLen) { throw new IOException("Packet len" + len + " is out of range!"); MG> The user in this scenario is running the latest version of Solr that was available at the time, which includes ZK 3.4.10 for its client. The error message indicated socket problems, but didn't have any information about the cause. When running under java 9, they got this as the error: WARN - 2018-04-04 09:05:28.194; org.apache.zookeeper.ClientCnxn$SendThread; Session 0x100244e8ffb0004 for server localhost/127.0.0.1:2181, unexpected error, closing socket connection and attempting reconnect java.io.IOException: Connection reset by peer With Java 8, they got this: WARN - 2018-04-04 09:10:11.879; org.apache.zookeeper.ClientCnxn$SendThread; Session 0x10024db7e280002 for server localhost/0:0:0:0:0:0:0:1:2181, unexpected error, closing socket connection and attempting reconnect java.io.IOException: Protocol wrong type for socket In both cases, the stacktrace listed a bunch of sun classes and then a couple of methods in zookeeper's ClientCnxnSocketNIO class. When I asked them what their ZK server log said, that's when I figured out the problem: 2018-04-04 14:06:01,361 [myid:] - WARN [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@383] - Exception causing close of session 0x10024db7e280006: Len error 5327937 > Do I understand correctly that Solr uploads file to ZooKeeper? Solr *itself* won't typically be uploading data to ZK that can exceed the max packet size. It is typically done either with a separate commandline program (the ZkCLI class the commandline program uses is included in Solr), or by a client program using the SolrJ library (which is part of Solr like ZkCLI, but usable by itself). The action being performed is an upload of a configuration for a Solr index. Solr does sometimes run into the problem described in ZOOKEEPER-1162, but this is due to the number of children in a znode, where each one has minimal data. Thanks, Shawn
