sorry Shawn , a) Total docs solr is handling is 3 million . b) index size is only 5 GB
On Thu, Jun 4, 2015 at 9:35 PM, Shawn Heisey <apa...@elyograg.org> wrote: > On 6/4/2015 7:38 AM, Midas A wrote: > > On Thu, Jun 4, 2015 at 6:48 PM, Shawn Heisey <apa...@elyograg.org> > wrote: > > > >> On 6/4/2015 5:15 AM, Midas A wrote: > >>> I have some indexing issue . While indexing IOwait is high in solr > server > >>> and load also. > >> My first suspect here is that you don't have enough RAM for your index > >> size. > >> > >> * How many total docs is Solr handling (all cores)? > >> > > --30,00000 dos > > > >> * What is the total size on disk of all your cores? > >> > > -- 600 GB > > > >> * How much RAM does the machine have? > >> > > --48 GB > > > >> * What is the java max heap? > >> --30 GB(jvm) > > Is that 3 million docs or 30 million docs? The actual numbers are 3 > million, but you put a single comma in the number after the 30, so I am > not sure which you meant. Either way, those documents must be quite > large, to make a 600GB index. 30 million docs in my index would only be > about 30GB. > > With 48 GB of RAM, 30 GB allocated to Solr, and a 600GB index, you don't > have anywhere even close to enough RAM to cache your index effectively. > There's only 18GB of RAM left over for the OS disk cache. That's only 3 > percent of the index data that can fit in the OS disk cache. I would > imagine that you're going to need to be able to fit somewhere between 25 > and 50 percent of the index into RAM, which would mean that you're going > to want around 256GB of RAM for that index. 128GB *might* be enough. > Alternatively, you could work on making your index smaller -- but be > aware that to improve performance with low memory, you need to reduce > the *indexed* part, the *stored* part makes little difference. > > Another potential problem with a 30GB heap is related to garbage > collection tuning. If you haven't tuned your GC at all, then > performance will be terrible on a heap that large, especially when you > are indexing. The wiki page I linked on my previous reply contains a > link to my personal page, which covers GC tuning: > > https://wiki.apache.org/solr/ShawnHeisey > > Thanks, > Shawn > >
