Re: SessionExpiredException
> For the sessionexpiredexception, the solr is throwing this exception and > then the shard goes down. > > From the following discussion, it seems to be that the solr is loosing > connection to zookeeper and throws the exception. In the zoo keeper > configuration file, zoo.cfg, is it safe to increase the synclimit shown in > below snippet. > > > # The number of milliseconds of each tick > tickTime=2000 > # The number of ticks that the initial > # synchronization phase can take > initLimit=10 > # The number of ticks that can pass between > # sending a request and getting an acknowledgement > syncLimit=5 > # the directory where the snapshot is stored. > # do not use /tmp for storage, /tmp here is just > # example sakes. > dataDir=/sanfs/mnt/vol01/solr/zookeeperdata/2 > # the port at which the clients will connect > clientPort=2181 > # the maximum number of client connections. > # increase this if you need to handle more clients > #maxClientCnxns=60 > > Thanks, > Satya > > On Mon, May 8, 2017 at 12:04 PM Satya Marivada > wrote: > >> The 3g memory is doing well, performing a gc at 600-700 MB. >> >> -XX:+UseConcMarkSweepGC -XX:+UseParNewGC >> >> Here are my jvm start up >> >> The start up parameters are: >> >> java -server -Xms3g -Xmx3g -XX:NewRatio=3 -XX:SurvivorRatio=4 >> -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 >> -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 >> -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark >> -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly >> -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 >> -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled >> -XX:-OmitStackTraceInFastThrow -verbose:gc -XX:+PrintHeapAtGC >> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps >> -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime >> -Xloggc:/sanfs/mnt/vol01/solr/solr-6.3.0/server/logs/solr_gc.log >> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=9 -XX:GCLogFileSize=20M >> -DzkClientTimeout=15000 ... >> >> On Mon, May 8, 2017 at 11:50 AM Walter Underwood >> wrote: >> >>> Which garbage collector are you using? The default GC will probably give >>> long pauses. >>> >>> You need to use CMS or G1. >>> >>> wunder >>> Walter Underwood >>> wun...@wunderwood.org >>> http://observer.wunderwood.org/ (my blog) >>> >>> >>> > On May 8, 2017, at 8:48 AM, Erick Erickson >>> wrote: >>> > >>> > 3G of memory should not lead to long GC pauses unless you're running >>> > very close to the edge of available memory. Paradoxically, running >>> > with 6G of memory may lead to _fewer_ noticeable pauses since the >>> > background threads can do the work, well, in the background. >>> > >>> > Best, >>> > Erick >>> > >>> > On Mon, May 8, 2017 at 7:29 AM, Satya Marivada >>> > wrote: >>> >> Hi Piyush and Shawn, >>> >> >>> >> May I ask what is the solution for it, if it is the long gc pauses? I >>> am >>> >> skeptical about the same problem in our case too. We have started >>> with 3G >>> >> of memory for the heap. >>> >> Did you have to adjust some of the memory allotted? Very much >>> appreciated. >>> >> >>> >> Thanks, >>> >> Satya >>> >> >>> >> On Sat, May 6, 2017 at 12:36 PM Piyush Kunal >> > >>> >> wrote: >>> >> >>> >>> We already faced this issue and found out the issue to be long GC >>> pauses >>> >>> itself on either client side or server side. >>> >>> Regards, >>> >>> Piyush >>> >>> >>> >>> On Sat, May 6, 2017 at 6:10 PM, Shawn Heisey >>> wrote: >>> >>> >>> >>>> On 5/3/2017 7:32 AM, Satya Marivada wrote: >>> >>>>> I see below exceptions in my logs sometimes. What could be causing >>> it? >>> >>>>> >>> >>>>> org.apache.zookeeper.KeeperException$SessionExpiredException: >>> >>>> >>> >>>> Based on my limited research, this would tend to indicate that the >>> >>>> heartbeats ZK uses to detect when sessions have gone inactive are >>> not >>> >>>> occurring in a timely fashion. >>> >>>> >>> >>>> Common causes seem to be: >>> >>>> >>> >>>> JVM Garbage collections. These can cause the entire JVM to pause >>> for an >>> >>>> extended period of time, and this time may exceed the configured >>> >>> timeouts. >>> >>>> >>> >>>> Excess client connections to ZK. ZK limits the number of >>> connections >>> >>>> from each client address, with the idea of preventing denial of >>> service >>> >>>> attacks. If a client is misbehaving, it may make more connections >>> than >>> >>>> it should. You can try increasing the limit in the ZK config, but >>> if >>> >>>> this is the reason for the exception, then something's probably >>> wrong, >>> >>>> and you may be just hiding the real problem. >>> >>>> >>> >>>> Although we might have bugs causing the second situation, the first >>> >>>> situation seems more likely. >>> >>>> >>> >>>> Thanks, >>> >>>> Shawn >>> >>>> >>> >>>> >>> >>> >>> >>>
Re: SessionExpiredException
For the sessionexpiredexception, the solr is throwing this exception and then the shard goes down. >From the following discussion, it seems to be that the solr is loosing connection to zookeeper and throws the exception. In the zoo keeper configuration file, zoo.cfg, is it safe to increase the synclimit shown in below snippet. # The number of milliseconds of each tick tickTime=2000 # The number of ticks that the initial # synchronization phase can take initLimit=10 # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit=5 # the directory where the snapshot is stored. # do not use /tmp for storage, /tmp here is just # example sakes. dataDir=/sanfs/mnt/vol01/solr/zookeeperdata/2 # the port at which the clients will connect clientPort=2181 # the maximum number of client connections. # increase this if you need to handle more clients #maxClientCnxns=60 Thanks, Satya On Mon, May 8, 2017 at 12:04 PM Satya Marivada wrote: > The 3g memory is doing well, performing a gc at 600-700 MB. > > -XX:+UseConcMarkSweepGC -XX:+UseParNewGC > > Here are my jvm start up > > The start up parameters are: > > java -server -Xms3g -Xmx3g -XX:NewRatio=3 -XX:SurvivorRatio=4 > -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 > -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 > -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark > -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly > -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 > -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled > -XX:-OmitStackTraceInFastThrow -verbose:gc -XX:+PrintHeapAtGC > -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps > -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime > -Xloggc:/sanfs/mnt/vol01/solr/solr-6.3.0/server/logs/solr_gc.log > -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=9 -XX:GCLogFileSize=20M > -DzkClientTimeout=15000 ... > > On Mon, May 8, 2017 at 11:50 AM Walter Underwood > wrote: > >> Which garbage collector are you using? The default GC will probably give >> long pauses. >> >> You need to use CMS or G1. >> >> wunder >> Walter Underwood >> wun...@wunderwood.org >> http://observer.wunderwood.org/ (my blog) >> >> >> > On May 8, 2017, at 8:48 AM, Erick Erickson >> wrote: >> > >> > 3G of memory should not lead to long GC pauses unless you're running >> > very close to the edge of available memory. Paradoxically, running >> > with 6G of memory may lead to _fewer_ noticeable pauses since the >> > background threads can do the work, well, in the background. >> > >> > Best, >> > Erick >> > >> > On Mon, May 8, 2017 at 7:29 AM, Satya Marivada >> > wrote: >> >> Hi Piyush and Shawn, >> >> >> >> May I ask what is the solution for it, if it is the long gc pauses? I >> am >> >> skeptical about the same problem in our case too. We have started with >> 3G >> >> of memory for the heap. >> >> Did you have to adjust some of the memory allotted? Very much >> appreciated. >> >> >> >> Thanks, >> >> Satya >> >> >> >> On Sat, May 6, 2017 at 12:36 PM Piyush Kunal >> >> wrote: >> >> >> >>> We already faced this issue and found out the issue to be long GC >> pauses >> >>> itself on either client side or server side. >> >>> Regards, >> >>> Piyush >> >>> >> >>> On Sat, May 6, 2017 at 6:10 PM, Shawn Heisey >> wrote: >> >>> >> >>>> On 5/3/2017 7:32 AM, Satya Marivada wrote: >> >>>>> I see below exceptions in my logs sometimes. What could be causing >> it? >> >>>>> >> >>>>> org.apache.zookeeper.KeeperException$SessionExpiredException: >> >>>> >> >>>> Based on my limited research, this would tend to indicate that the >> >>>> heartbeats ZK uses to detect when sessions have gone inactive are not >> >>>> occurring in a timely fashion. >> >>>> >> >>>> Common causes seem to be: >> >>>> >> >>>> JVM Garbage collections. These can cause the entire JVM to pause >> for an >> >>>> extended period of time, and this time may exceed the configured >> >>> timeouts. >> >>>> >> >>>> Excess client connections to ZK. ZK limits the number of connections >> >>>> from each client address, with the idea of preventing denial of >> service >> >>>> attacks. If a client is misbehaving, it may make more connections >> than >> >>>> it should. You can try increasing the limit in the ZK config, but if >> >>>> this is the reason for the exception, then something's probably >> wrong, >> >>>> and you may be just hiding the real problem. >> >>>> >> >>>> Although we might have bugs causing the second situation, the first >> >>>> situation seems more likely. >> >>>> >> >>>> Thanks, >> >>>> Shawn >> >>>> >> >>>> >> >>> >> >>
Re: SessionExpiredException
This is on solr-6.3.0 and external zookeeper 3.4.9 On Wed, May 3, 2017 at 11:39 PM Zheng Lin Edwin Yeo wrote: > Are you using SolrCloud with external ZooKeeper, or Solr's internal > ZooKeeper? > > Also, which version of Solr are you using? > > Regards, > Edwin > > On 3 May 2017 at 21:32, Satya Marivada wrote: > > > Hi, > > > > I see below exceptions in my logs sometimes. What could be causing it? > > > > org.apache.zookeeper.KeeperException$SessionExpiredException: > > KeeperErrorCode = Session expired for /overseer > > at > > org.apache.zookeeper.KeeperException.create(KeeperException.java:127) > > at > > org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) > > > > Thanks, > > Satya > > >
Re: SessionExpiredException
The 3g memory is doing well, performing a gc at 600-700 MB. -XX:+UseConcMarkSweepGC -XX:+UseParNewGC Here are my jvm start up The start up parameters are: java -server -Xms3g -Xmx3g -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -XX:-OmitStackTraceInFastThrow -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/sanfs/mnt/vol01/solr/solr-6.3.0/server/logs/solr_gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=9 -XX:GCLogFileSize=20M -DzkClientTimeout=15000 ... On Mon, May 8, 2017 at 11:50 AM Walter Underwood wrote: > Which garbage collector are you using? The default GC will probably give > long pauses. > > You need to use CMS or G1. > > wunder > Walter Underwood > wun...@wunderwood.org > http://observer.wunderwood.org/ (my blog) > > > > On May 8, 2017, at 8:48 AM, Erick Erickson > wrote: > > > > 3G of memory should not lead to long GC pauses unless you're running > > very close to the edge of available memory. Paradoxically, running > > with 6G of memory may lead to _fewer_ noticeable pauses since the > > background threads can do the work, well, in the background. > > > > Best, > > Erick > > > > On Mon, May 8, 2017 at 7:29 AM, Satya Marivada > > wrote: > >> Hi Piyush and Shawn, > >> > >> May I ask what is the solution for it, if it is the long gc pauses? I am > >> skeptical about the same problem in our case too. We have started with > 3G > >> of memory for the heap. > >> Did you have to adjust some of the memory allotted? Very much > appreciated. > >> > >> Thanks, > >> Satya > >> > >> On Sat, May 6, 2017 at 12:36 PM Piyush Kunal > >> wrote: > >> > >>> We already faced this issue and found out the issue to be long GC > pauses > >>> itself on either client side or server side. > >>> Regards, > >>> Piyush > >>> > >>> On Sat, May 6, 2017 at 6:10 PM, Shawn Heisey > wrote: > >>> > >>>> On 5/3/2017 7:32 AM, Satya Marivada wrote: > >>>>> I see below exceptions in my logs sometimes. What could be causing > it? > >>>>> > >>>>> org.apache.zookeeper.KeeperException$SessionExpiredException: > >>>> > >>>> Based on my limited research, this would tend to indicate that the > >>>> heartbeats ZK uses to detect when sessions have gone inactive are not > >>>> occurring in a timely fashion. > >>>> > >>>> Common causes seem to be: > >>>> > >>>> JVM Garbage collections. These can cause the entire JVM to pause for > an > >>>> extended period of time, and this time may exceed the configured > >>> timeouts. > >>>> > >>>> Excess client connections to ZK. ZK limits the number of connections > >>>> from each client address, with the idea of preventing denial of > service > >>>> attacks. If a client is misbehaving, it may make more connections > than > >>>> it should. You can try increasing the limit in the ZK config, but if > >>>> this is the reason for the exception, then something's probably wrong, > >>>> and you may be just hiding the real problem. > >>>> > >>>> Although we might have bugs causing the second situation, the first > >>>> situation seems more likely. > >>>> > >>>> Thanks, > >>>> Shawn > >>>> > >>>> > >>> > >
Re: SessionExpiredException
Which garbage collector are you using? The default GC will probably give long pauses. You need to use CMS or G1. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On May 8, 2017, at 8:48 AM, Erick Erickson wrote: > > 3G of memory should not lead to long GC pauses unless you're running > very close to the edge of available memory. Paradoxically, running > with 6G of memory may lead to _fewer_ noticeable pauses since the > background threads can do the work, well, in the background. > > Best, > Erick > > On Mon, May 8, 2017 at 7:29 AM, Satya Marivada > wrote: >> Hi Piyush and Shawn, >> >> May I ask what is the solution for it, if it is the long gc pauses? I am >> skeptical about the same problem in our case too. We have started with 3G >> of memory for the heap. >> Did you have to adjust some of the memory allotted? Very much appreciated. >> >> Thanks, >> Satya >> >> On Sat, May 6, 2017 at 12:36 PM Piyush Kunal >> wrote: >> >>> We already faced this issue and found out the issue to be long GC pauses >>> itself on either client side or server side. >>> Regards, >>> Piyush >>> >>> On Sat, May 6, 2017 at 6:10 PM, Shawn Heisey wrote: >>> >>>> On 5/3/2017 7:32 AM, Satya Marivada wrote: >>>>> I see below exceptions in my logs sometimes. What could be causing it? >>>>> >>>>> org.apache.zookeeper.KeeperException$SessionExpiredException: >>>> >>>> Based on my limited research, this would tend to indicate that the >>>> heartbeats ZK uses to detect when sessions have gone inactive are not >>>> occurring in a timely fashion. >>>> >>>> Common causes seem to be: >>>> >>>> JVM Garbage collections. These can cause the entire JVM to pause for an >>>> extended period of time, and this time may exceed the configured >>> timeouts. >>>> >>>> Excess client connections to ZK. ZK limits the number of connections >>>> from each client address, with the idea of preventing denial of service >>>> attacks. If a client is misbehaving, it may make more connections than >>>> it should. You can try increasing the limit in the ZK config, but if >>>> this is the reason for the exception, then something's probably wrong, >>>> and you may be just hiding the real problem. >>>> >>>> Although we might have bugs causing the second situation, the first >>>> situation seems more likely. >>>> >>>> Thanks, >>>> Shawn >>>> >>>> >>>
Re: SessionExpiredException
3G of memory should not lead to long GC pauses unless you're running very close to the edge of available memory. Paradoxically, running with 6G of memory may lead to _fewer_ noticeable pauses since the background threads can do the work, well, in the background. Best, Erick On Mon, May 8, 2017 at 7:29 AM, Satya Marivada wrote: > Hi Piyush and Shawn, > > May I ask what is the solution for it, if it is the long gc pauses? I am > skeptical about the same problem in our case too. We have started with 3G > of memory for the heap. > Did you have to adjust some of the memory allotted? Very much appreciated. > > Thanks, > Satya > > On Sat, May 6, 2017 at 12:36 PM Piyush Kunal > wrote: > >> We already faced this issue and found out the issue to be long GC pauses >> itself on either client side or server side. >> Regards, >> Piyush >> >> On Sat, May 6, 2017 at 6:10 PM, Shawn Heisey wrote: >> >> > On 5/3/2017 7:32 AM, Satya Marivada wrote: >> > > I see below exceptions in my logs sometimes. What could be causing it? >> > > >> > > org.apache.zookeeper.KeeperException$SessionExpiredException: >> > >> > Based on my limited research, this would tend to indicate that the >> > heartbeats ZK uses to detect when sessions have gone inactive are not >> > occurring in a timely fashion. >> > >> > Common causes seem to be: >> > >> > JVM Garbage collections. These can cause the entire JVM to pause for an >> > extended period of time, and this time may exceed the configured >> timeouts. >> > >> > Excess client connections to ZK. ZK limits the number of connections >> > from each client address, with the idea of preventing denial of service >> > attacks. If a client is misbehaving, it may make more connections than >> > it should. You can try increasing the limit in the ZK config, but if >> > this is the reason for the exception, then something's probably wrong, >> > and you may be just hiding the real problem. >> > >> > Although we might have bugs causing the second situation, the first >> > situation seems more likely. >> > >> > Thanks, >> > Shawn >> > >> > >>
Re: SessionExpiredException
Hi Piyush and Shawn, May I ask what is the solution for it, if it is the long gc pauses? I am skeptical about the same problem in our case too. We have started with 3G of memory for the heap. Did you have to adjust some of the memory allotted? Very much appreciated. Thanks, Satya On Sat, May 6, 2017 at 12:36 PM Piyush Kunal wrote: > We already faced this issue and found out the issue to be long GC pauses > itself on either client side or server side. > Regards, > Piyush > > On Sat, May 6, 2017 at 6:10 PM, Shawn Heisey wrote: > > > On 5/3/2017 7:32 AM, Satya Marivada wrote: > > > I see below exceptions in my logs sometimes. What could be causing it? > > > > > > org.apache.zookeeper.KeeperException$SessionExpiredException: > > > > Based on my limited research, this would tend to indicate that the > > heartbeats ZK uses to detect when sessions have gone inactive are not > > occurring in a timely fashion. > > > > Common causes seem to be: > > > > JVM Garbage collections. These can cause the entire JVM to pause for an > > extended period of time, and this time may exceed the configured > timeouts. > > > > Excess client connections to ZK. ZK limits the number of connections > > from each client address, with the idea of preventing denial of service > > attacks. If a client is misbehaving, it may make more connections than > > it should. You can try increasing the limit in the ZK config, but if > > this is the reason for the exception, then something's probably wrong, > > and you may be just hiding the real problem. > > > > Although we might have bugs causing the second situation, the first > > situation seems more likely. > > > > Thanks, > > Shawn > > > > >
Re: SessionExpiredException
We already faced this issue and found out the issue to be long GC pauses itself on either client side or server side. Regards, Piyush On Sat, May 6, 2017 at 6:10 PM, Shawn Heisey wrote: > On 5/3/2017 7:32 AM, Satya Marivada wrote: > > I see below exceptions in my logs sometimes. What could be causing it? > > > > org.apache.zookeeper.KeeperException$SessionExpiredException: > > Based on my limited research, this would tend to indicate that the > heartbeats ZK uses to detect when sessions have gone inactive are not > occurring in a timely fashion. > > Common causes seem to be: > > JVM Garbage collections. These can cause the entire JVM to pause for an > extended period of time, and this time may exceed the configured timeouts. > > Excess client connections to ZK. ZK limits the number of connections > from each client address, with the idea of preventing denial of service > attacks. If a client is misbehaving, it may make more connections than > it should. You can try increasing the limit in the ZK config, but if > this is the reason for the exception, then something's probably wrong, > and you may be just hiding the real problem. > > Although we might have bugs causing the second situation, the first > situation seems more likely. > > Thanks, > Shawn > >
Re: SessionExpiredException
On 5/3/2017 7:32 AM, Satya Marivada wrote: > I see below exceptions in my logs sometimes. What could be causing it? > > org.apache.zookeeper.KeeperException$SessionExpiredException: Based on my limited research, this would tend to indicate that the heartbeats ZK uses to detect when sessions have gone inactive are not occurring in a timely fashion. Common causes seem to be: JVM Garbage collections. These can cause the entire JVM to pause for an extended period of time, and this time may exceed the configured timeouts. Excess client connections to ZK. ZK limits the number of connections from each client address, with the idea of preventing denial of service attacks. If a client is misbehaving, it may make more connections than it should. You can try increasing the limit in the ZK config, but if this is the reason for the exception, then something's probably wrong, and you may be just hiding the real problem. Although we might have bugs causing the second situation, the first situation seems more likely. Thanks, Shawn
Re: SessionExpiredException
Are you using SolrCloud with external ZooKeeper, or Solr's internal ZooKeeper? Also, which version of Solr are you using? Regards, Edwin On 3 May 2017 at 21:32, Satya Marivada wrote: > Hi, > > I see below exceptions in my logs sometimes. What could be causing it? > > org.apache.zookeeper.KeeperException$SessionExpiredException: > KeeperErrorCode = Session expired for /overseer > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:127) > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) > > Thanks, > Satya >
SessionExpiredException
Hi, I see below exceptions in my logs sometimes. What could be causing it? org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) Thanks, Satya