Re: seeing lots of autowarming messages in log during DIH indexing

2013-05-31 Thread geeky2 
the DIH is launched via a script - called by a cron like scheduler.

clean, commit and optimize are all true.

thx
mark



#!/bin/bash
SERVER=$1
PORT=$2
CLEAN=$3
COMMIT=$4
OPTIMIZE=$5
COREPATH=$6

echo SERVER: $SERVER
echo PORT: $PORT
echo CLEAN: $CLEAN
echo COMMIT: $COMMIT
echo OPTIMIZE: $OPTIMIZE
echo COREPATH: $COREPATH


if [ $# != 6 ]; then
echo USAGE: $0 [SERVER] [PORT] [CLEAN: true/false] [COMMIT:
true/false] [OPTIMIZE: true/false] [COREPATH] 
exit 1;
fi

...






--
View this message in context: 
http://lucene.472066.n3.nabble.com/seeing-lots-of-autowarming-messages-in-log-during-DIH-indexing-tp4064649p4067477.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: seeing lots of autowarming messages in log during DIH indexing

2013-05-25 Thread Tim Vaillancourt 

Interesting.

In your scenario would you use commit=true, or commit=false, and do you use 
auto soft/hard commits?

Secondly, if you did use auto-soft/hard commits, how would they affect this 
scenario? I'm guessing even with commit=false, the autoCommits would be 
triggered either by time or max docs, which opens a searcher anyways. A total 
guess though.

I'm interested in doing full-imports without committing/opening new searchers 
until it is complete.

Cheers!

Tim

On 20/05/13 03:59 PM, shreejay wrote:

geeky2 wrote

you mean i would add this switch to my script that kicks of the
dataimport?

exmaple:


OUTPUT=$(curl -v
http://${SERVER}.intra.searshc.com:${PORT}/solrpartscat/${CORE}/dataimport
-F command=full-import -F clean=${CLEAN} -F commit=${COMMIT} -F
optimize=${OPTIMIZE} -F openSearcher=false)

Yes. Thats correct



geeky2 wrote

what needs to be done _AFTER_ the DIH finishes (if anything)?

eg, does this need to be turned back on after the DIH has finished?

Yes. You need to open the searcher to be able to search. Just run another
commit with openSearcher = true , once your indexing process finishes.





--
View this message in context: 
http://lucene.472066.n3.nabble.com/seeing-lots-of-autowarming-messages-in-log-during-DIH-indexing-tp4064649p4064768.html
Sent from the Solr - User mailing list archive at Nabble.com.

seeing lots of autowarming messages in log during DIH indexing

2013-05-20 Thread geeky2 
hello,

we are tracking down some performance issues with our DIH process.

not sure if this is related - but i am seeing tons of the messages below in
the logs during re-indexing of the core.

what do these messages mean?


2013-05-18 19:37:30,623 INFO  [org.apache.solr.update.UpdateHandler]
(pool-11-thread-1) end_commit_flush
2013-05-18 19:37:30,623 INFO  [org.apache.solr.search.SolrIndexSearcher]
(pool-10-thread-1) autowarming Searcher@5b8d745 main from Searcher@1fb355af
main
   
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
2013-05-18 19:37:30,624 INFO  [org.apache.solr.search.SolrIndexSearcher]
(pool-10-thread-1) autowarming result for Searcher@5b8d745 main
   
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
2013-05-18 19:37:30,624 INFO  [org.apache.solr.search.SolrIndexSearcher]
(pool-10-thread-1) autowarming Searcher@5b8d745 main from Searcher@1fb355af
main
   
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
2013-05-18 19:37:30,625 INFO  [org.apache.solr.search.SolrIndexSearcher]
(pool-10-thread-1) autowarming result for Searcher@5b8d745 main
   
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
2013-05-18 19:37:30,625 INFO  [org.apache.solr.search.SolrIndexSearcher]
(pool-10-thread-1) autowarming Searcher@5b8d745 main from Searcher@1fb355af
main
   
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=3,evictions=0,size=3,warmupTime=1,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
2013-05-18 19:37:30,628 INFO  [org.apache.solr.search.SolrIndexSearcher]
(pool-10-thread-1) autowarming result for Searcher@5b8d745 main
   
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=3,evictions=0,size=3,warmupTime=3,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
2013-05-18 19:37:30,628 INFO  [org.apache.solr.search.SolrIndexSearcher]
(pool-10-thread-1) autowarming Searcher@5b8d745 main from Searcher@1fb355af
main
   
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
2013-05-18 19:37:30,628 INFO  [org.apache.solr.search.SolrIndexSearcher]
(pool-10-thread-1) autowarming result for Searcher@5b8d745 main
   
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}

thx
mark




--
View this message in context: 
http://lucene.472066.n3.nabble.com/seeing-lots-of-autowarming-messages-in-log-during-DIH-indexing-tp4064649.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: seeing lots of autowarming messages in log during DIH indexing

2013-05-20 Thread Shreejay 
Every time a commit is done a new searcher is opened. In the solr config file 
caches are defined with a parameter called autowarm. Autowarm basically tries 
to copy the cache values from previous searcher into the current one. If you 
are doing a bulk update and do not care for searching till your indexing is 
over then you can specify openSearcher=false while doing a commit. That should 
speed up your indexing a lot.  

-- 
Shreejay Nair
Sent from my mobile device. Please excuse brevity and typos.


On Monday, May 20, 2013 at 7:16, geeky2 wrote:

 hello,
 
 we are tracking down some performance issues with our DIH process.
 
 not sure if this is related - but i am seeing tons of the messages below in
 the logs during re-indexing of the core.
 
 what do these messages mean?
 
 
 2013-05-18 19:37:30,623 INFO [org.apache.solr.update.UpdateHandler]
 (pool-11-thread-1) end_commit_flush
 2013-05-18 19:37:30,623 INFO [org.apache.solr.search.SolrIndexSearcher]
 (pool-10-thread-1) autowarming Searcher@5b8d745 main from Searcher@1fb355af
 main
 fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
 2013-05-18 19:37:30,624 INFO [org.apache.solr.search.SolrIndexSearcher]
 (pool-10-thread-1) autowarming result for Searcher@5b8d745 main
 fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
 2013-05-18 19:37:30,624 INFO [org.apache.solr.search.SolrIndexSearcher]
 (pool-10-thread-1) autowarming Searcher@5b8d745 main from Searcher@1fb355af
 main
 filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
 2013-05-18 19:37:30,625 INFO [org.apache.solr.search.SolrIndexSearcher]
 (pool-10-thread-1) autowarming result for Searcher@5b8d745 main
 filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
 2013-05-18 19:37:30,625 INFO [org.apache.solr.search.SolrIndexSearcher]
 (pool-10-thread-1) autowarming Searcher@5b8d745 main from Searcher@1fb355af
 main
 queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=3,evictions=0,size=3,warmupTime=1,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
 2013-05-18 19:37:30,628 INFO [org.apache.solr.search.SolrIndexSearcher]
 (pool-10-thread-1) autowarming result for Searcher@5b8d745 main
 queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=3,evictions=0,size=3,warmupTime=3,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
 2013-05-18 19:37:30,628 INFO [org.apache.solr.search.SolrIndexSearcher]
 (pool-10-thread-1) autowarming Searcher@5b8d745 main from Searcher@1fb355af
 main
 documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
 2013-05-18 19:37:30,628 INFO [org.apache.solr.search.SolrIndexSearcher]
 (pool-10-thread-1) autowarming result for Searcher@5b8d745 main
 documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
 
 thx
 mark
 
 
 
 
 --
 View this message in context: 
 http://lucene.472066.n3.nabble.com/seeing-lots-of-autowarming-messages-in-log-during-DIH-indexing-tp4064649.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Re: seeing lots of autowarming messages in log during DIH indexing

2013-05-20 Thread geeky2 
you mean i would add this switch to my script that kicks of the dataimport?

exmaple:


OUTPUT=$(curl -v
http://${SERVER}.intra.searshc.com:${PORT}/solrpartscat/${CORE}/dataimport
-F command=full-import -F clean=${CLEAN} -F commit=${COMMIT} -F
optimize=${OPTIMIZE} -F openSearcher=false)


what needs to be done _AFTER_ the DIH finishes (if anything)?

eg, does this need to be turned back on after the DIH has finished?





--
View this message in context: 
http://lucene.472066.n3.nabble.com/seeing-lots-of-autowarming-messages-in-log-during-DIH-indexing-tp4064649p4064695.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: seeing lots of autowarming messages in log during DIH indexing

2013-05-20 Thread shreejay 
geeky2 wrote
 you mean i would add this switch to my script that kicks of the
 dataimport?
 
 exmaple:
 
 
 OUTPUT=$(curl -v
 http://${SERVER}.intra.searshc.com:${PORT}/solrpartscat/${CORE}/dataimport
 -F command=full-import -F clean=${CLEAN} -F commit=${COMMIT} -F
 optimize=${OPTIMIZE} -F openSearcher=false)

Yes. Thats correct



geeky2 wrote
 what needs to be done _AFTER_ the DIH finishes (if anything)?
 
 eg, does this need to be turned back on after the DIH has finished?

Yes. You need to open the searcher to be able to search. Just run another
commit with openSearcher = true , once your indexing process finishes.





--
View this message in context: 
http://lucene.472066.n3.nabble.com/seeing-lots-of-autowarming-messages-in-log-during-DIH-indexing-tp4064649p4064768.html
Sent from the Solr - User mailing list archive at Nabble.com.