Hi, Follow the process, and more data about my SOLR + ZOOKEEPER.
root 48425 1 26 Jul29 ? 03:00:39 java -server -Xms28g -Xmx32g -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -XX:-OmitStackTraceInFastThrow -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/solr/server/logs/solr_gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=9 -XX:GCLogFileSize=20M -DzkClientTimeout=15000 -DzkHost=177.55.55.152:2181,177.55.55.153:2181,177.55.55.154:2181, 177.55.55.155:2181,177.55.55.156:2181 -Dsolr.log.dir=/solr/server/logs -Djetty.port=8983 -DSTOP.PORT=7983 -DSTOP.KEY=solrrocks -Duser.timezone=UTC -Djetty.home=/solr/server -Dsolr.solr.home=/solr/server/solr -Dsolr.data.home= -Dsolr.install.dir=/solr -Dsolr.default.confdir=/solr/server/solr/configsets/_default/conf -Xss256k -Dsolr.jetty.https.port=8983 -Dsolr.log.muteconsole -XX:OnOutOfMemoryError=/solr/bin/oom_solr.sh 8983 /solr/server/logs -jar start.jar --module=http root 48163 1 0 Jul29 ? 00:01:33 java -Dzookeeper.log.dir=/zoop/bin/../logs -Dzookeeper.log.file=zookeeper-root-server-eddison0001.log -Dzookeeper.root.logger=INFO,CONSOLE -XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemoryError=kill -9 %p -cp /zoop/bin/../zookeeper-server/target/classes:/zoop/bin/../build/classes:/zoop/bin/../zookeeper-server/target/lib/*.jar:/zoop/bin/../build/lib/*.jar:/zoop/bin/../lib/zookeeper-jute-3.5.5.jar:/zoop/bin/../lib/zookeeper-3.5.5.jar:/zoop/bin/../lib/slf4j-log4j12-1.7.25.jar:/zoop/bin/../lib/slf4j-api-1.7.25.jar:/zoop/bin/../lib/netty-all-4.1.29.Final.jar:/zoop/bin/../lib/log4j-1.2.17.jar:/zoop/bin/../lib/json-simple-1.1.1.jar:/zoop/bin/../lib/jline-2.11.jar:/zoop/bin/../lib/jetty-util-9.4.17.v20190418.jar:/zoop/bin/../lib/jetty-servlet-9.4.17.v20190418.jar:/zoop/bin/../lib/jetty-server-9.4.17.v20190418.jar:/zoop/bin/../lib/jetty-security-9.4.17.v20190418.jar:/zoop/bin/../lib/jetty-io-9.4.17.v20190418.jar:/zoop/bin/../lib/jetty-http-9.4.17.v20190418.jar:/zoop/bin/../lib/javax.servlet-api-3.1.0.jar:/zoop/bin/../lib/jackson-databind-2.9.8.jar:/zoop/bin/../lib/jackson-core-2.9.8.jar:/zoop/bin/../lib/jackson-annotations-2.9.0.jar:/zoop/bin/../lib/commons-cli-1.2.jar:/zoop/bin/../lib/audience-annotations-0.5.0.jar:/zoop/bin/../zookeeper-*.jar:/zoop/bin/../zookeeper-server/src/main/resources/lib/*.jar:/zoop/bin/../conf: -*Xmx1000m -Xmx4096m* -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /zoop/conf/zoo.cfg root 91749 91497 0 09:43 pts/0 00:00:00 grep --color=auto -i zook [09:43:55] root@eddison0001:~$ Answer the question, look the ZOOKEEPER process, 2 xmx. By the way, I changed the SOLR (28xms - 32 xmx) because after 5 days ago using the SOLR, I received the message about Heap Memory in SOLR. Nowadays I don't have more message about Heap Memory in Solr. Regards, Em ter, 30 de jul de 2019 às 08:41, Jörn Franke <jornfra...@gmail.com> escreveu: > 2 xmx does not make sense, > > Your heap seems unusual large usually your heap should be much smaller > than available memory so solr can use it for index caching which is off-heap > > > Am 30.07.2019 um 13:25 schrieb Rodrigo Oliveira < > adamantina.rodr...@gmail.com>: > > > > Hi, > > > > My environment have 5 servers with solr + zookeeper in the same hosts. > > > > However, I've had 48 RAM each servers (solr - xms 28 gb and xmx - 32 gb). > > > > Looking for my servers and process, in zookeepers has xmx 1000 mb and xmx > > 4096 mb (last item, was change for me). > > > > Why 2 values for xmx? > > > > Regards, > > > > Em ter, 30 de jul de 2019 04:44, Dominique Bejean < > dominique.bej...@eolya.fr> > > escreveu: > > > >> Hi, > >> > >> I don’t find any documentation about the parameter > >> zookeeper_server_java_heaps > >> in zoo.cfg. > >> The way to control java heap size is either the java.env file of the > >> zookeeper-env.sh file. In zookeeper-env.sh > >> SERVER_JVMFLAGS="-Xmx=512m" > >> > >> How many RAM on your server ? > >> > >> Regards > >> > >> Dominique > >> > >> > >> > >> > >> Le lun. 29 juil. 2019 à 20:35, Rodrigo Oliveira < > >> adamantina.rodr...@gmail.com> a écrit : > >> > >>> Hi, > >>> > >>> After 3 days running, my zookeeper showing this error. > >>> > >>> 2019-07-29 15:10:41,906 [myid:1] - WARN > >>> [RecvWorker:4332550065071534382:QuorumCnxManager$RecvWorker@1176] - > >>> Connection broken for id 4332550065071534382, my id = 1, error = > >>> java.io.IOException: Received packet with invalid packet: 824196618 > >>> at > >>> > >>> > >> > org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1163) > >>> 2019-07-29 15:10:41,906 [myid:1] - WARN > >>> [RecvWorker:4332550065071534382:QuorumCnxManager$RecvWorker@1179] - > >>> Interrupting SendWorker > >>> 2019-07-29 15:10:41,907 [myid:1] - WARN > >>> [SendWorker:4332550065071534382:QuorumCnxManager$SendWorker@1092] - > >>> Interrupted while waiting for message on queue > >>> java.lang.InterruptedException > >>> at > >>> > >>> > >> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) > >>> at > >>> > >>> > >> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) > >>> at > >>> > java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) > >>> at > >>> > >>> > >> > org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1243) > >>> at > >>> > >>> > >> > org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:78) > >>> at > >>> > >>> > >> > org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1080) > >>> 2019-07-29 15:10:41,907 [myid:1] - WARN > >>> [SendWorker:4332550065071534382:QuorumCnxManager$SendWorker@1102] - > >> Send > >>> worker leaving thread id 4332550065071534382 my id = 1 > >>> 2019-07-29 15:10:41,917 [myid:1] - INFO [/177.55.55.152:3888 > >>> :QuorumCnxManager$Listener@888] - Received connection request / > >>> 177.55.55.63:53972 > >>> 2019-07-29 15:10:41,920 [myid:1] - WARN > >>> [RecvWorker:4332550065071534382:QuorumCnxManager$RecvWorker@1176] - > >>> Connection broken for id 4332550065071534382, my id = 1, error = > >>> java.io.IOException: Received packet with invalid packet: 840973834 > >>> at > >>> > >>> > >> > org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1163) > >>> 2019-07-29 15:10:41,921 [myid:1] - WARN > >>> [RecvWorker:4332550065071534382:QuorumCnxManager$RecvWorker@1179] - > >>> Interrupting SendWorker > >>> 2019-07-29 15:10:41,922 [myid:1] - WARN > >>> [SendWorker:4332550065071534382:QuorumCnxManager$SendWorker@1092] - > >>> Interrupted while waiting for message on queue > >>> java.lang.InterruptedException > >>> at > >>> > >>> > >> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) > >>> at > >>> > >>> > >> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) > >>> at > >>> > java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) > >>> at > >>> > >>> > >> > org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1243) > >>> at > >>> > >>> > >> > org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:78) > >>> at > >>> > >>> > >> > org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1080) > >>> 2019-07-29 15:10:41,922 [myid:1] - WARN > >>> [SendWorker:4332550065071534382:QuorumCnxManager$SendWorker@1102] - > >> Send > >>> worker leaving thread id 4332550065071534382 my id = 1 > >>> 2019-07-29 15:10:41,932 [myid:1] - INFO [/177.55.55.152:3888 > >>> :QuorumCnxManager$Listener@888] - Received connection request / > >>> 177.55.55.63:38633 > >>> 2019-07-29 15:10:41,933 [myid:1] - WARN > >>> [RecvWorker:4332550065071534638:QuorumCnxManager$RecvWorker@1176] - > >>> Connection broken for id 4332550065071534638, my id = 1, error = > >>> java.io.IOException: Received packet with invalid packet: 807419402 > >>> at > >>> > >>> > >> > org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1163) > >>> 2019-07-29 15:10:41,933 [myid:1] - WARN > >>> [RecvWorker:4332550065071534638:QuorumCnxManager$RecvWorker@1179] - > >>> Interrupting SendWorker > >>> 2019-07-29 15:10:41,934 [myid:1] - WARN > >>> [SendWorker:4332550065071534638:QuorumCnxManager$SendWorker@1092] - > >>> Interrupted while waiting for message on queue > >>> java.lang.InterruptedException > >>> at > >>> > >>> > >> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) > >>> at > >>> > >>> > >> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) > >>> at > >>> > java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) > >>> at > >>> > >>> > >> > org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1243) > >>> at > >>> > >>> > >> > org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:78) > >>> at > >>> > >>> > >> > org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1080) > >>> 2019-07-29 15:10:41,935 [myid:1] - WARN > >>> [SendWorker:4332550065071534638:QuorumCnxManager$SendWorker@1102] - > >> Send > >>> worker leaving thread id 4332550065071534638 my id = 1 > >>> 2019-07-29 15:10:41,981 [myid:1] - WARN > >>> [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of > >>> session > >>> 0x0: Len error 1008750420 > >>> 2019-07-29 15:10:41,987 [myid:1] - WARN > >>> [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of > >>> session > >>> 0x0: Len error 1008750420 > >>> 2019-07-29 15:10:41,992 [myid:1] - WARN > >>> [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of > >>> session > >>> 0x0: Len error 1008750420 > >>> 2019-07-29 15:10:42,008 [myid:1] - WARN > >>> [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of > >>> session > >>> 0x0: Len error 1008750420 > >>> 2019-07-29 15:10:45,375 [myid:1] - WARN > >>> [NIOWorkerThread-11:NIOServerCnxn@366] - Unable to read additional > data > >>> from client sessionid 0x400044d531804b6, likely client has closed > socket > >>> 2019-07-29 15:10:46,062 [myid:1] - WARN > >>> [NIOWorkerThread-4:NIOServerCnxn@366] - Unable to read additional data > >>> from client sessionid 0x300044d504c0471, likely client has closed > socket > >>> 2019-07-29 15:11:12,965 [myid:1] - WARN [qtp1789550256-23:URIUtil@396 > ] > >> - > >>> > >>> > >> > /%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/etc/passwd > >>> org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8! > >>> byte C0 in state 0 > >>> 2019-07-29 15:11:17,588 [myid:1] - INFO [/177.153.44.152:3888 > >>> :QuorumCnxManager$Listener@888] - Received connection request / > >>> 177.55.55.63:33721 > >>> java.lang.OutOfMemoryError: Java heap space > >>> Dumping heap to java_pid97598.hprof ... > >>> 2019-07-29 15:11:17,590 [myid:1] - WARN > >>> [RecvWorker:1:QuorumCnxManager$RecvWorker@1176] - Connection broken > for > >>> id > >>> 1, my id = 1, error = > >>> java.io.IOException: Received packet with invalid packet: 1966420818 > >>> at > >>> > >>> > >> > org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1163) > >>> 2019-07-29 15:11:19,452 [myid:1] - WARN > >>> [RecvWorker:1:QuorumCnxManager$RecvWorker@1179] - Interrupting > >> SendWorker > >>> Heap dump file created [638922790 bytes in 3,132 secs] > >>> # > >>> # java.lang.OutOfMemoryError: Java heap space > >>> # -XX:OnOutOfMemoryError="kill -9 %p" > >>> # Executing /bin/sh -c "kill -9 97598"... > >>> > >>> That's right. Any tips ou tricks to workaround this error? How to > >> increase > >>> the memory in Zookeeper? > >>> > >>> Look my zoo.conf. > >>> > >>> [15:30:09] root@eddison0001:/zoop/conf$ cat zoo.cfg > >>> # The number of milliseconds of each tick > >>> tickTime=2000 > >>> # The number of ticks that the initial > >>> # synchronization phase can take > >>> initLimit=10 > >>> # The number of ticks that can pass between > >>> # sending a request and getting an acknowledgement > >>> syncLimit=5 > >>> # the directory where the snapshot is stored. > >>> # do not use /tmp for storage, /tmp here is just > >>> # example sakes. > >>> dataDir=/zoop/zookeeperdata > >>> # the port at which the clients will connect > >>> clientPort=2181 > >>> # the maximum number of client connections. > >>> # increase this if you need to handle more clients > >>> #maxClientCnxns=60 > >>> # > >>> # Be sure to read the maintenance section of the > >>> # administrator guide before turning on autopurge. > >>> # > >>> # > >>> > >> > http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance > >>> # > >>> # The number of snapshots to retain in dataDir > >>> #autopurge.snapRetainCount=3 > >>> # Purge task interval in hours > >>> # Set to "0" to disable auto purge feature > >>> #autopurge.purgeInterval=1 > >>> oom_heap_dump_dir=/tmp > >>> zookeeper_server_java_heaps=4G > >>> > >>> server.1=177.55.55.152:2888:3888 > >>> server.2=177.55.55.153:2888:3888 > >>> server.3=177.55.55.154:2888:3888 > >>> server.4=177.55.55.155:2888:3888 > >>> server.5=177.55.55.156:2888:3888 > >>> > >>> Any workaround? > >>> > >>> Regards, > >>> > >> >
