Re: Solr Permgen Exceptions when creating/removing cores

[Tri Cao]

Hey Josh,

I am not an expert in Java performance, but I would start with  dumping a the heap

and investigate with visualvm (the free tool that comes with JDK).

In my experience, the most common cause for PermGen exception is the app creates

too many interned strings. Solr (actually Lucene) interns the field names so if you have

too many fields, it might be the cause. How many fields in total across cores did you

create before the exception?

Can you reproduce the problem with the standard Solr? Is the bitnami distribution just

Solr or do they have some other libraries?

Hope this helps,

Tri

On Mar 03, 2014, at 07:28 AM, Josh <jwda...@gmail.com> wrote:

It's a windows installation using a bitnami solr installer. I incorrectly
put 64M into the configuration for this, as I had copied the test
configuration I was using to recreate the permgen issue we were seeing on
our production system (that is configured to 512M) as it takes awhile with
to recreate the issue with larger permgen values. In the test scenario
there was a small 180 document data core that's static with 8 dynamic user
cores that are used to index the unique document ids in the users view,
which is then merged into a single user core. The final user core contains
the same number of document ids as the data core and the data core is
queried against with the ids in the final merged user core as the limiter.
The user cores are then unloaded, and deleted from the drive and then the
test is reran again with the user cores re-created

We are also using the core discovery mode to store/find our cores and the
database data core is using dynamic fields with a mix of single value and
multi value fields. The user cores use a static configuration. The data is
indexed from SQL Server using jtDS for both the user and data cores. As a
note we also reversed the test case I mention above where we keep the user
cores static and dynamically create the database core and this created the
same issue only it leaked faster. We assumed this because the configuration
was larger/loaded more classes then the simpler user core.

When I get the time I'm going to put together a SolrJ test app to recreate
the issue outside of our environment to see if others see the same issue
we're seeing to rule out any kind of configuration problem. Right now we're
interacting with solr with POCO via the restful interface and it's not very
easy for us to spin this off into something someone else could use. In the
mean time we've made changes to make the user cores more static, this has
slowed down the build up of permgen to something that can be managed by a
weekly reset.

Sorry about the confusion in my initial email and I appreciate the
response. Anything about my configuration that you can think might be
useful just let me know and I can provide it. We have a work around, but it
really hampers what our long term goals were for our Solr implementation.

Thanks
Josh


On Mon, Mar 3, 2014 at 9:57 AM, Greg Walters <greg.walt...@answers.com>wrote:

Josh,

You've mentioned a couple of times that you've got PermGen set to 512M but

then you say you're running with -XX:MaxPermSize=64M. These two statements

are contradictory so are you *sure* that you're running with 512M of

PermGen? Assuming your on a *nix box can you provide `ps` output proving

this?

Thanks,

Greg

On Feb 28, 2014, at 5:22 PM, Furkan KAMACI <furkankam...@gmail.com> wrote:

> Hi;

>

> You can also check here:

>

http://stackoverflow.com/questions/3717937/cmspermgensweepingenabled-vs-cmsclassunloadingenabled

>

> Thanks;

> Furkan KAMACI

>

>

> 2014-02-26 22:35 GMT+02:00 Josh <jwda...@gmail.com>:

>

>> Thanks Timothy,

>>

>> I gave these a try and -XX:+CMSPermGenSweepingEnabled seemed to cause

the

>> error to happen more quickly. With this option on it didn't seemed to do

>> any intermittent garbage collecting that delayed the issue in with it

off.

>> I was already using a max of 512MB, and I can reproduce it with it set

this

>> high or even higher. Right now because of how we have this implemented

just

>> increasing it to something high just delays the problem :/

>>

>> Anything else you could suggest I would really appreciate.

>>

>>

>> On Wed, Feb 26, 2014 at 3:19 PM, Tim Potter <tim.pot...@lucidworks.com

>>> wrote:

>>

>>> Hi Josh,

>>>

>>> Try adding: -XX:+CMSPermGenSweepingEnabled as I think for some VM

>>> versions, permgen collection was disabled by default.

>>>

>>> Also, I use: -XX:MaxPermSize=512m -XX:PermSize=256m with Solr, so 64M

may

>>> be too small.

>>>

>>>

>>> Timothy Potter

>>> Sr. Software Engineer, LucidWorks

>>> www.lucidworks.com

>>>

>>> ________________________________________

>>> From: Josh <jwda...@gmail.com>

>>> Sent: Wednesday, February 26, 2014 12:27 PM

>>> To: solr-user@lucene.apache.org

>>> Subject: Solr Permgen Exceptions when creating/removing cores

>>>

>>> We are using the Bitnami version of Solr 4.6.0-1 on a 64bit windows

>>> installation with 64bit Java 1.7U51 and we are seeing consistent issues

>>> with PermGen exceptions. We have the permgen configured to be 512MB.

>>> Bitnami ships with a 32bit version of Java for windows and we are

>> replacing

>>> it with a 64bit version.

>>>

>>> Passed in Java Options:

>>>

>>> -XX:MaxPermSize=64M

>>> -Xms3072M

>>> -Xmx6144M

>>> -XX:+UseParNewGC

>>> -XX:+UseConcMarkSweepGC

>>> -XX:CMSInitiatingOccupancyFraction=75

>>> -XX:+CMSClassUnloadingEnabled

>>> -XX:NewRatio=3

>>>

>>> -XX:MaxTenuringThreshold=8

>>>

>>> This is our use case:

>>>

>>> We have what we call a database core which remains fairly static and

>>> contains the imported contents of a table from SQL server. We then have

>>> user cores which contain the record ids of results from a text search

>>> outside of Solr. We then query for the data we want from the database

>> core

>>> and limit the results to the content of the user core. This allows us

to

>>> combine facet data from Solr with the search results from another

engine.

>>> We are creating the user cores on demand and removing them when the

user

>>> logs out.

>>>

>>> Our issue is the constant creation and removal of user cores combined

>> with

>>> the constant importing seems to push us over our PermGen limit. The

user

>>> cores are removed at the end of every session and as a test I made an

>>> application that would loop creating the user core, import a set of

data

>> to

>>> it, query the database core using it as a limiter and then remove the

>> user

>>> core. My expectation was in this scenario that all the permgen

associated

>>> with that user cores would be freed upon it's unload and allow permgen

to

>>> reclaim that memory during a garbage collection. This was not the case,

>> it

>>> would constantly go up until the application would exhaust the memory.

>>>

>>> I also investigated whether the there was a connection between the two

>>> cores left behind because I was joining them together in a query but

even

>>> unloading the database core after unloading all the user cores won't

>>> prevent the limit from being hit or any memory to be garbage collected

>> from

>>> Solr.

>>>

>>> Is this a known issue with creating and unloading a large number of

>> cores?

>>> Could it be configuration based for the core? Is there something other

>> than

>>> unloading that needs to happen to free the references?

>>>

>>> Thanks

>>>

>>> Notes: I've tried using tools to determine if it's a leak within Solr

>> such

>>> as Plumbr and my activities turned up nothing.

>>>

>>