Re: Index optimization takes too long

2018-11-04 Thread Toke Eskildsen
On Sat, 2018-11-03 at 21:41 -0700, Wei wrote: > Thanks everyone! I checked the system metrics during the optimization > process. CPU usage is quite low, there is no I/O wait, and memory > usage is not much different from before the docValues change. So I > wonder what could be the bottleneck.

Re: Index optimization takes too long

2018-11-03 Thread Wei
Thanks everyone! I checked the system metrics during the optimization process. CPU usage is quite low, there is no I/O wait, and memory usage is not much different from before the docValues change. So I wonder what could be the bottleneck. Thanks, Wei On Sat, Nov 3, 2018 at 1:38 PM Erick

Re: Index optimization takes too long

2018-11-03 Thread Erick Erickson
Going from my phone so it'll be terse. See uninvertingmergeuodateprocessor (or something like that). Also, there's an idea in SOLR-12259 IIRC, but that'll be in 7.6 at the earliest. On Sat, Nov 3, 2018, 07:13 Shawn Heisey On 11/3/2018 5:32 AM, Dave wrote: > > On a side note, does adding

Re: Index optimization takes too long

2018-11-03 Thread Shawn Heisey
On 11/3/2018 5:32 AM, Dave wrote: On a side note, does adding docvalues to an already indexed field, and then optimizing, prevent the need to reindex to take advantage of docvalues? I was under the impression you had to reindex the content. You must reindex when changing the schema to add

Re: Index optimization takes too long

2018-11-03 Thread Dave
On a side note, does adding docvalues to an already indexed field, and then optimizing, prevent the need to reindex to take advantage of docvalues? I was under the impression you had to reindex the content. > On Nov 3, 2018, at 4:41 AM, Deepak Goel wrote: > > I would start by monitoring the

Re: Index optimization takes too long

2018-11-03 Thread Deepak Goel
I would start by monitoring the hardware (CPU, Memory, Disk) & software (heap, threads) utilization's and seeing where the bottlenecks are. Or what is getting utilized the most. And then tune that parameter. I would also look at profiling the software. Deepak "The greatness of a nation can be

Re: Index optimization takes too long

2018-11-02 Thread Shawn Heisey
On 11/2/2018 5:00 PM, Wei wrote: After a recent schema change, it takes almost 40 minutes to optimize the index. The schema change is to enable docValues for all sort/facet fields, which increase the index size from 12G to 14G. Before the change it only takes 5 minutes to do the optimization.

Index optimization takes too long

2018-11-02 Thread Wei
Hello, After a recent schema change, it takes almost 40 minutes to optimize the index. The schema change is to enable docValues for all sort/facet fields, which increase the index size from 12G to 14G. Before the change it only takes 5 minutes to do the optimization. I have tried to increase