Re: Solr vs Sphinx
Hi guys, I work now for serveral month on solr and really you provide quick answer ... and you're very nice to work with. But I've got huge issue that I couldn't fixe after lot of post. My indexation take one two days to be done. For 8G of data indexed and 1,5M of docs (ok I've plenty of links in my table but it takes such a long time). Second I've to do update every 20mn but every update represent maybe 20 000docs and when I use the replication I must replicate all the new index folder optimized because Ive too much datas updated and too much segment needs to be generate and I have to merge datas. So I lost my cache and my CPU goes mad. And I can't have more than 20request/sec. Fergus McMenemie-2 wrote: > >>Something that would be interesting is to share solr configs for >>various types of indexing tasks. From a solr configuration aimed at >>indexing web pages to one doing large amounts of text to one that >>indexes specific structured data. I could see those being posted on >>the wiki and helping folks who say "I want to do X, is there an >>example?". >> >>I think most folks start with the example Solr install and tweak from >>there, which probably isn't the best path... >> >>Eric > > Yep a solr "cookbook" with lots of different example recipes. However > these would need to be very actively maintained to ensure they always > represented best practice. While using cocoon I made extensive use > of the examples section of the cocoon website. However most of the, > massive number of, examples represent obsolete cocoon practise. Or > there were four or five examples doing the same thing in different > ways with no text explaining the pros/cons of the different approaches. > This held me, as a newcomer, back and gave a bad impression of cocoon. > > I was wondering about a performance hints page. I was caught by an > issue indexing CSV content where the use of &overwrite=false made > an almost 3x difference to my indexing speed. Still do not really > know why! > >> >>On May 15, 2009, at 8:09 AM, Mark Miller wrote: >> >>> In the spirit of good defaults: >>> >>> I think we should change the Solr highlighter to highlight phrase >>> queries by default, as well as prefix,range,wildcard constantscore >>> queries. Its awkward to have to tell people you have to turn those >>> on. I'd certainly prefer to have to turn them off if I have some >>> limitation rather than on. > > Yep I agree, all whizzy new features should ideally be on by default > unless there is a significant performance penalty. It is not enough > that to issue a default solrconfig.xml with the feature on, it has to > be on by default inside the code. > >>> >>> - Mark >> >>- >>Eric Pugh | Principal | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com >>Free/Busy: http://tinyurl.com/eric-cal > > Fergus > > -- View this message in context: http://www.nabble.com/Solr-vs-Sphinx-tp23524676p23852364.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: replication solr 1.4
what would be the url to ping to replicate like http://slave_host:port/solr/replication?command=enablepoll thanks -- View this message in context: http://www.nabble.com/replication-solr-1.4-tp23777206p23777272.html Sent from the Solr - User mailing list archive at Nabble.com.
replication solr 1.4
Hi guys, I didnt ennoy you for ages now ... hope everybody is fine ... I've an issue with my replication I was wondering ... after a while replication doesnt work anymore ... we have a script which enable or not replication every 2hours and this morning it didnt pull anything and it's maybe because the version is too far away ? is it possible ??? is it possible to tell it to just take the last one ? cuz now it's working but because i clicked on replication. thanks a lot, Johanna Replicatable Index Version:1239958162858, Generation: 1741 Poll Interval 00:00:30 Local Index Index Version: 1239958162819, Generation: 1702 Location: /data/solr/video/data/index Size: 8.86 GB Times Replicated Since Startup: 896 Previous Replication Done At: Thu May 28 14:39:51 CEST 2009 Config Files Replicated At: Config Files Replicated: Times Config Files Replicated Since Startup: Next Replication Cycle At: Thu May 28 14:34:31 CEST 2009 Current Replication Status Start Time: Fri May 29 11:49:50 CEST 2009 Files Downloaded: 0 / 12 Downloaded: 1.66 GB / 8.87 GB [18.0%] Downloading File: _5io.prx, Downloaded: 1.66 GB / 2.22 GB [74.0%] Time Elapsed: 122s, Estimated Time Remaining: 528s, Speed: 13.96 MB/s Controls Cores: [video ][user ][group ] Current Time: Fri May 29 11:51:52 CEST 2009 Server Start At: Thu May 28 10:43:30 CEST 2009 Return to Admin Page -- View this message in context: http://www.nabble.com/replication-solr-1.4-tp23777206p23777206.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: fieldType without tokenizer
hi I tried but Ive an error : May 12 15:48:51 solr-test jsvc.exec[2583]: May 12, 2009 3:48:51 PM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error loading class 'solr.KeywordTokenizer' ^Iat org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:310) ^Iat org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:325) ^Iat org.apache.solr.util.plugin.AbstractPluginLoader.create(AbstractPluginLoader.java:84) ^Iat org.apache.solr.util.plugin.AbstractPluginLoader.load(AbstractPluginLoader.java:141) ^Iat org.apache.solr.schema.IndexSchema.readAnalyzer(IndexSchema.java:804) ^Iat org.apache.solr.schema.IndexSchema.access$100(IndexSchema.java:58) ^Iat org.apache.solr.schema.IndexSchema$1.create(IndexSchema.java:425) ^Iat org.apache.solr.schema.IndexSchema$1.create(IndexSchema.java:443) ^Iat org.apache.solr.util.plugin.AbstractPluginLoader.load(AbstractPluginLoader.java:141) ^Iat org.apache.solr.schema.IndexSchema.readSchema(IndexSchema.java:452) ^Iat org.apache.solr.schema.In with : Shalin Shekhar Mangar wrote: > > On Mon, May 4, 2009 at 9:28 PM, sunnyfr wrote: > >> >> Hi, >> >> I would like to create a field without tokenizer but I've an error, >> > > You can use KeywordTokenizer which does not do any tokenization. > > -- > Regards, > Shalin Shekhar Mangar. > > -- View this message in context: http://www.nabble.com/fieldType-without-tokenizer-tp23371300p23502994.html Sent from the Solr - User mailing list archive at Nabble.com.
fieldType without tokenizer
Hi, I would like to create a field without tokenizer but I've an error, I tried : But I've : May 4 17:49:41 solr-test jsvc.exec[5786]: May 4, 2009 5:49:41 PM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: analyzer without class or tokenizer & filter list ^Iat org.apache.solr.schema.IndexSchema.readAnalyzer(IndexSchema.java:808) ^Iat org.apache.solr.schema.IndexSchema.access$100(IndexSchema.java:58) ^Iat org.apache.solr.schema.IndexSchema$1.create(IndexSchema.java:425) ^Iat org.apache.solr.schema.IndexSchema$1.create(IndexSchema.java:443) ^Iat org.apache.solr.util.plugin.AbstractPluginLoader.load(AbstractPluginLoader.java:141) ^Iat org.apache.solr.schema.IndexSchema.readSchema(IndexSchema.java:452) ^Iat org.apache.solr.schema.IndexSchema.(IndexSchema.java:95) ^Iat org.apache.solr.core.CoreContainer.create(CoreContainer.java:344) ^Iat org.apache.solr.core.CoreContainer.load(CoreContainer.java:222) ^Iat org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:107) ^Iat org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:69) Thanks for your help ! -- View this message in context: http://www.nabble.com/fieldType-without-tokenizer-tp23371300p23371300.html Sent from the Solr - User mailing list archive at Nabble.com.
autoSuggest
Hi, I would like to know how work /autoSuggest. I do have result when I hit : /autoSuggest?terms=true&indent=true&terms.fl=title&terms.rows=5&terms.lower=simp&omitHeader=true I've: 74 129 2 2 1 How can I ask it to suggest first expression which are more frequent in the database ? How can I look for even for two words, ie: I look for "bara" ... make it suggesting "barack obama" ??? thanks a lot, -- View this message in context: http://www.nabble.com/autoSuggest-tp23367848p23367848.html Sent from the Solr - User mailing list archive at Nabble.com.
/replication?command=isReplicating
Hi, Just to know if there is a quick way to get the information without hiting replication?command=details like =isReplicating Thanks, -- View this message in context: http://www.nabble.com/-replication-command%3DisReplicating-tp23295869p23295869.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: boost qf weight between 0 and 10
How can I get the weight of a field and use it in bf ?? thanks a lot sunnyfr wrote: > > Hi Hoss, > thanks for this answser, and is there a way to get the weight of a field ? > like that and use it in the bf? queryWeight > > > 0.14232224 = (MATCH) weight(text:chien^0.2 in 9412049), product of: > 0.0813888 = queryWeight(text:chien^0.2), product of: > 0.2 = boost > 6.5946517 = idf(docFreq=55585, numDocs=14951742) > 0.061708186 = queryNorm > > > thanks > > > hossman wrote: >> >> >> : I don't get really, I try to boost a field according to another one but >> I've >> : a huge weight when I'm using qf boost like : >> : >> : /select?qt=dismax&fl=*&q="obama >> : meeting"&debugQuery=true&qf=title&bf=product(title,stat_views) >> >> bf is a boost function -- you are using a product fucntion to multiply >> the >> "title" field by the stat_views" field ... this doesn't make sense to me? >> >> i'm assuming the "title" field contains text (the rest of your score >> explanation confirms this). when you try to do a math function on a >> string based field it deals with the "ordinal" value -- the higher the >> string is lexigraphically compared to all other docs ,the higher the >> ordinal value. >> >> i have no idea what's in your stat_views field -- but i can't imagine any >> way in which multipling it by the ordinal value of your text field would >> make sense... >> >> : 5803675.5 = (MATCH) >> FunctionQuery(product(ord(title),sint(stat_views))), >> : product of: >> : 9.5142968E7 = product(ord(title)=1119329,sint(stat_views)=85) >> : 1.0 = boost >> : 0.06099952 = queryNorm >> >> : But this is not equilibrate between this boost in qf and bf, how can I >> do ? >> >> when it comes to function query, you're on your own to figure out an >> appropriate query boost to blanace the scores out -- when you use a >> product function the scores are going to get huge like this unless you >> balance it somehow (and that ord(title) is just making this massively >> worse) >> >> >> -Hoss >> >> >> > > -- View this message in context: http://www.nabble.com/boost-qf-weight-between-0-and-10-tp22081396p23293956.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: boost qf weight between 0 and 10
Hi Hoss, thanks for this answser, and is there a way to get the weight of a field ? like that and use it in the bf? queryWeight 0.14232224 = (MATCH) weight(text:chien^0.2 in 9412049), product of: 0.0813888 = queryWeight(text:chien^0.2), product of: 0.2 = boost 6.5946517 = idf(docFreq=55585, numDocs=14951742) 0.061708186 = queryNorm thanks hossman wrote: > > > : I don't get really, I try to boost a field according to another one but > I've > : a huge weight when I'm using qf boost like : > : > : /select?qt=dismax&fl=*&q="obama > : meeting"&debugQuery=true&qf=title&bf=product(title,stat_views) > > bf is a boost function -- you are using a product fucntion to multiply the > "title" field by the stat_views" field ... this doesn't make sense to me? > > i'm assuming the "title" field contains text (the rest of your score > explanation confirms this). when you try to do a math function on a > string based field it deals with the "ordinal" value -- the higher the > string is lexigraphically compared to all other docs ,the higher the > ordinal value. > > i have no idea what's in your stat_views field -- but i can't imagine any > way in which multipling it by the ordinal value of your text field would > make sense... > > : 5803675.5 = (MATCH) > FunctionQuery(product(ord(title),sint(stat_views))), > : product of: > : 9.5142968E7 = product(ord(title)=1119329,sint(stat_views)=85) > : 1.0 = boost > : 0.06099952 = queryNorm > > : But this is not equilibrate between this boost in qf and bf, how can I > do ? > > when it comes to function query, you're on your own to figure out an > appropriate query boost to blanace the scores out -- when you use a > product function the scores are going to get huge like this unless you > balance it somehow (and that ord(title) is just making this massively > worse) > > > -Hoss > > > -- View this message in context: http://www.nabble.com/boost-qf-weight-between-0-and-10-tp22081396p23257545.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: autowarmcount how to check if cache has been warmed up
Ok I've an error :
Apr 24 09:28:41 search-01 jsvc.exec[8033]: :907) ^Iat
java.lang.Thread.run(Thread.java:619)
Apr 24 09:29:20 search-01 jsvc.exec[8033]: Apr 24, 2009 9:29:20 AM
org.apache.solr.common.SolrException log SEVERE: Error during auto-warming
of
key:org.apache.solr.search.queryresult...@cb278b97:java.lang.OutOfMemoryError:
Java heap space ^Iat
org.apache.lucene.search.FieldCacheImpl$10.createValue(FieldCacheImpl.java:410)
^Iat
org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:71)
^Iat
org.apache.lucene.search.FieldCacheImpl.getStringIndex(FieldCacheImpl.java:359)
^Iat
org.apache.solr.search.function.ReverseOrdFieldSource.getValues(ReverseOrdFieldSource.java:55)
^Iat
org.apache.solr.search.function.ReciprocalFloatFunction.getValues(ReciprocalFloatFunction.java:56)
^Iat
org.apache.solr.search.function.FunctionQuery$AllScorer.(FunctionQuery.java:105)
^Iat
org.apache.solr.search.function.FunctionQuery$FunctionWeight.scorer(FunctionQuery.java:81)
^Iat
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:231)
^Iat org.apache.lucene.search.IndexSearcher.doSearch(IndexSearc
Apr 24 09:29:20 search-01 jsvc.exec[8033]:
ndexSearcher.search(IndexSearcher.java:250) ^Iat
org.apache.lucene.search.Searcher.search(Searcher.java:126) ^Iat
org.apache.lucene.search.Searcher.search(Searcher.java:105) ^Iat
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1072)
^Iat
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:928)
^Iat
org.apache.solr.search.SolrIndexSearcher.access$000(SolrIndexSearcher.java:59)
^Iat
org.apache.solr.search.SolrIndexSearcher$3.regenerateItem(SolrIndexSearcher.java:336)
^Iat org.apache.solr.search.FastLRUCache.warm(FastLRUCache.java:150) ^Iat
org.apache.solr.search.SolrIndexSearcher.warm(SolrIndexSearcher.java:1646)
^Iat org.apache.solr.core.SolrCore$3.call(SolrCore.java:1147) ^Iat
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) ^Iat
java.util.concurrent.FutureTask.run(FutureTask.java:138) ^Iat
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
^Iat java.util.concurrent.Threa
Apr 24 09:29:20 search-01 jsvc.exec[8033]: :907) ^Iat
java.lang.Thread.run(Thread.java:619)
Apr 24 09:30:00 search-01 jsvc.exec[8033]: Apr 24, 2009 9:30:00 AM
org.apache.solr.common.SolrException log SEVERE: Error during auto-warming
of
key:org.apache.solr.search.queryresult...@2b729b52:java.lang.OutOfMemoryError:
Java heap space
Apr 24 09:30:10 search-01 jsvc.exec[8033]: Apr 24, 2009 9:30:10 AM
org.apache.solr.update.DirectUpdateHandler2 commit INFO: start
commit(optimize=false,waitFlush=false,waitSearcher=true)
Apr 24 09:30:10 search-01 jsvc.exec[8033]: Apr 24, 2009 9:30:10 AM
org.apache.solr.core.SolrCore getSearcher INFO: [video] PERFORMANCE WARNING:
Overlapping onDeckSearchers=2
Apr 24 09:30:10 search-01 jsvc.exec[8033]: Apr 24, 2009 9:30:10 AM
org.apache.solr.search.SolrIndexSearcher INFO: Opening
searc...@21df6b4f main
Apr 24 09:30:11 search-01 jsvc.exec[8033]: Apr 24, 2009 9:30:11 AM
org.apache.solr.update.DirectUpdateHandler2 commit INFO: end_commit_flush
Shalin Shekhar Mangar wrote:
>
> OK, lets try this:
>
> 1. Before a commit, check the stats page, see if the size is more than 5
> 2. Then call commit, and verify that the size is more than 5
>
> If the original size was > 5, then you should have size > 5 after
> autowarming too.
>
> On Wed, Apr 22, 2009 at 2:57 PM, sunnyfr wrote:
>
>>
>> still the same ?
>>
>> Seems done :
>> lookups : 0
>> hits : 0
>> hitratio : 0.00
>> inserts : 0
>> evictions : 0
>> size : 5
>> warmupTime : 20973
>> cumulative_lookups : 0
>> cumulative_hits : 0
>> cumulative_hitratio : 0.00
>> cumulative_inserts : 0
>> cumulative_evictions : 0
>>
>>
>> Apr 22 11:09:29 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:29 AM
>> org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming
>> searc...@48b6c333 main from searc...@79e79d96 main
>>
>> ^IfieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>> Apr 22 11:09:29 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:29 AM
>> org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming result
>> for
>> searc...@48b6c333 main
>>
>> ^IfieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>> Apr 22 11:09:29 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:29 AM
>> org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming
>> searc...@48b6c333 main from searc...@79e79d96
Re: autowarmcount how to check if cache has been warmed up
It looks like it doesnt warm up, no?
sunnyfr wrote:
>
> still the same ?
>
> Seems done :
> lookups : 0
> hits : 0
> hitratio : 0.00
> inserts : 0
> evictions : 0
> size : 5
> warmupTime : 20973
> cumulative_lookups : 0
> cumulative_hits : 0
> cumulative_hitratio : 0.00
> cumulative_inserts : 0
> cumulative_evictions : 0
>
>
> Apr 22 11:09:29 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:29 AM
> org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming
> searc...@48b6c333 main from searc...@79e79d96 main
> ^IfieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> Apr 22 11:09:29 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:29 AM
> org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming result for
> searc...@48b6c333 main
> ^IfieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> Apr 22 11:09:29 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:29 AM
> org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming
> searc...@48b6c333 main from searc...@79e79d96 main
> ^IfilterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> Apr 22 11:09:29 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:29 AM
> org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming result for
> searc...@48b6c333 main
> ^IfilterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> Apr 22 11:09:29 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:29 AM
> org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming
> searc...@48b6c333 main from searc...@79e79d96 main
> ^IqueryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=5,warmupTime=3055,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> Apr 22 11:09:50 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:50 AM
> org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming result for
> searc...@48b6c333 main
> ^IqueryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=5,warmupTime=20973,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> Apr 22 11:09:50 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:50 AM
> org.apache.solr.core.QuerySenderListener newSearcher INFO:
> QuerySenderListener sending requests to searc...@48b6c333 main
> Apr 22 11:09:50 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:50 AM
> org.apache.solr.core.SolrCore execute INFO: [video] webapp=null path=null
> params={start=0&q=solr&rows=100} hits=164 status=0 QTime=0
> Apr 22 11:09:50 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:50 AM
> org.apache.solr.core.SolrCore execute INFO: [video] webapp=null path=null
> params={start=0&q=rocks&rows=100} hits=167581 status=0 QTime=51
> Apr 22 11:09:51 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:51 AM
> org.apache.solr.core.SolrCore execute INFO: [video] webapp=null path=null
> params={sort=id+desc&q=anything} hits=8419 status=0 QTime=50
> Apr 22 11:09:51 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:51 AM
> org.apache.solr.core.QuerySenderListener newSearcher INFO:
> QuerySenderListener done.
> Apr 22 11:09:51 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:51 AM
> org.apache.solr.core.SolrCore registerSearcher INFO: [video] Registered
> new searcher searc...@48b6c333 main
> Apr 22 11:09:51 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:51 AM
> org.apache.solr.search.SolrIndexSearcher close INFO: Closing
> searc...@79e79d96 main
> ^IfieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> ^IfilterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> ^IqueryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=5,warmupTime=3055,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> Apr 22 11:09:51 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:51 AM
> org.apache.solr.handler.dataimport.SolrWriter readIndexerProperties INFO:
> Read dataimport.properties
> Apr 22 11:09:
Re: autowarmcount how to check if cache has been warmed up
still the same ?
Seems done :
lookups : 0
hits : 0
hitratio : 0.00
inserts : 0
evictions : 0
size : 5
warmupTime : 20973
cumulative_lookups : 0
cumulative_hits : 0
cumulative_hitratio : 0.00
cumulative_inserts : 0
cumulative_evictions : 0
Apr 22 11:09:29 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:29 AM
org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming
searc...@48b6c333 main from searc...@79e79d96 main
^IfieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Apr 22 11:09:29 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:29 AM
org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming result for
searc...@48b6c333 main
^IfieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Apr 22 11:09:29 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:29 AM
org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming
searc...@48b6c333 main from searc...@79e79d96 main
^IfilterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Apr 22 11:09:29 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:29 AM
org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming result for
searc...@48b6c333 main
^IfilterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Apr 22 11:09:29 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:29 AM
org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming
searc...@48b6c333 main from searc...@79e79d96 main
^IqueryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=5,warmupTime=3055,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Apr 22 11:09:50 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:50 AM
org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming result for
searc...@48b6c333 main
^IqueryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=5,warmupTime=20973,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Apr 22 11:09:50 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:50 AM
org.apache.solr.core.QuerySenderListener newSearcher INFO:
QuerySenderListener sending requests to searc...@48b6c333 main
Apr 22 11:09:50 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:50 AM
org.apache.solr.core.SolrCore execute INFO: [video] webapp=null path=null
params={start=0&q=solr&rows=100} hits=164 status=0 QTime=0
Apr 22 11:09:50 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:50 AM
org.apache.solr.core.SolrCore execute INFO: [video] webapp=null path=null
params={start=0&q=rocks&rows=100} hits=167581 status=0 QTime=51
Apr 22 11:09:51 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:51 AM
org.apache.solr.core.SolrCore execute INFO: [video] webapp=null path=null
params={sort=id+desc&q=anything} hits=8419 status=0 QTime=50
Apr 22 11:09:51 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:51 AM
org.apache.solr.core.QuerySenderListener newSearcher INFO:
QuerySenderListener done.
Apr 22 11:09:51 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:51 AM
org.apache.solr.core.SolrCore registerSearcher INFO: [video] Registered new
searcher searc...@48b6c333 main
Apr 22 11:09:51 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:51 AM
org.apache.solr.search.SolrIndexSearcher close INFO: Closing
searc...@79e79d96 main
^IfieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
^IfilterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
^IqueryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=5,warmupTime=3055,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Apr 22 11:09:51 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:51 AM
org.apache.solr.handler.dataimport.SolrWriter readIndexerProperties INFO:
Read dataimport.properties
Apr 22 11:09:51 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:51 AM
org.apache.solr.handler.dataimport.SolrWriter persist INFO: Wrote last
indexed time to dataimport.properties
Apr 22 11:09:51 search-01 jsvc.exec[31908]: Apr 22, 2009 11:09:51 AM
org.apache.solr.handler.dataimport.DocBuilder execute INFO: Time taken =
0:9:49.967
Apr 22 11:10:49 search-01 jsvc.exec[31908]: Apr 22, 2009 11:10:49 AM
org.apache.solr.core.SolrCore execute INFO
Re: autowarmcount how to check if cache has been warmed up
yes but let me check again ... but the delta-import was lidle and I think warmup was done is the log .. i will check it again now and let you know. Shalin Shekhar Mangar wrote: > > On Wed, Apr 22, 2009 at 2:05 PM, sunnyfr wrote: > >> >> thanks Shalin, >> >> How come just 5 if my autowarmCount=500 ? >> > > Maybe the cache did not have more than 5 items before the commit? > > Actually, the cumulative_inserts=0 so I think Solr was just started up? > > -- > Regards, > Shalin Shekhar Mangar. > > -- View this message in context: http://www.nabble.com/autowarmcount-how-to-check-if-cache-has-been-warmed-up-tp23156612p23172214.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: autowarmcount how to check if cache has been warmed up
thanks Shalin, How come just 5 if my autowarmCount=500 ? -- View this message in context: http://www.nabble.com/autowarmcount-how-to-check-if-cache-has-been-warmed-up-tp23156612p23172066.html Sent from the Solr - User mailing list archive at Nabble.com.
autowarmcount how to check if cache has been warmed up
Hi, Is it possible to have autowarmcount=500 with warmupTime=2751 and size=5, where can I check up if the cache is full or not cuz really there it looks empty still??? and commitment is done. solr1.4 thanks for your help, sunny name:queryResultCache class: org.apache.solr.search.FastLRUCache version:1.0 description:Concurrent LRU Cache(maxSize=14774644, initialSize=14774644, minSize=13297179, acceptableSize=14035911, cleanupThread=false, autowarmCount=500, regenerator=org.apache.solr.search.solrindexsearche...@6e4eeaaf) stats: lookups : 0 hits : 0 hitratio : 0.00 inserts : 0 evictions : 0 size : 5 warmupTime : 2751 cumulative_lookups : 0 cumulative_hits : 0 cumulative_hitratio : 0.00 cumulative_inserts : 0 cumulative_evictions : 0 -- View this message in context: http://www.nabble.com/autowarmcount-how-to-check-if-cache-has-been-warmed-up-tp23156612p23156612.html Sent from the Solr - User mailing list archive at Nabble.com.
underscores are parsed only as spaces
Hi, I don't get why and how to change this: underscores are parsed only as spaces, meaning that a search for user "ejekt_festival" will return zero results, while "ejekt festival" will return the user "ejekt_festival". Thanks for your help, -- View this message in context: http://www.nabble.com/underscores-are-parsed-only-as-spaces-tp23132245p23132245.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Sorting performance + replication of index between cores
Hi Christophe, Did you find a way to fix up your problem, cuz even with replication will have this problem, lot of update means clear cache and manage that. I've the same issue, I just wondering if I won't turn off servers during update ??? How did you fix that ? Thanks, sunny christophe-2 wrote: > > Hi, > > After fully reloading my index, using another field than a Data does not > help that much. > Using a warmup query avoids having the first request slow, but: > - Frequents commits means that the Searcher is reloaded frequently > and, as the warmup takes time, the clients must wait. > - Having warmup slows down the index process (I guess this is > because after a commit, the Searchers are recreated) > > So I'm considering, as suggested, to have two instances: one for > indexing and one for searching. > I was wondering if there are simple ways to replicate the index in a > single Solr server running two cores ? Any such config already tested ? > I guess that the standard replication based on rsync can be simplified a > lot in this case as the two indexes are on the same server. > > Thanks > Christophe > > Beniamin Janicki wrote: >> :so you can send your updates anytime you want, and as long as you only >> :commit every 5 minutes (or commit on a master as often as you want, but >> :only run snappuller/snapinstaller on your slaves every 5 minutes) your >> :results will be at most 5minutes + warming time stale. >> >> This is what I do as well ( commits are done once per 5 minutes ). I've >> got >> master - slave configuration. Master has turned off all caches (commented >> in >> solrconfig.cml) and setup only 2 maxWarmingSearchers. Index size has 5GB >> ,Xmx= 1GB and committing takes around 10 secs ( on default configuration >> with warming it took from 30 mins up to 2 hours). >> >> Slave caches are configured to have autowarmCount="0" and >> maxWarmingSearchers=1 , and I have new data 1 second after snapshoot is >> done. I haven't noticed any huge delays while serving search request. >> Try to use those values - may be they'll help in your case too. >> >> Ben Janicki >> >> >> -Original Message- >> From: Chris Hostetter [mailto:hossman_luc...@fucit.org] >> Sent: 22 October 2008 04:56 >> To: solr-user@lucene.apache.org >> Subject: Re: Sorting performance >> >> >> : The problem is that I will have hundreds of users doing queries, and a >> : continuous flow of document coming in. >> : So a delay in warming up a cache "could" be acceptable if I do it a few >> times >> : per day. But not on a too regular basis (right now, the first query >> that >> loads >> : the cache takes 150s). >> : >> : However: I'm not sure why it looks not to be a good idea to update the >> caches >> >> you can refresh the caches automaticly after updating, the "newSearcher" >> event is fired whenever a searcher is opened (but before it's used by >> clients) so you can configure warming queries for it -- it doesn't have >> to >> be done manually (or by the first user to use that reader) >> >> so you can send your updates anytime you want, and as long as you only >> commit every 5 minutes (or commit on a master as often as you want, but >> only run snappuller/snapinstaller on your slaves every 5 minutes) your >> results will be at most 5minutes + warming time stale. >> >> >> -Hoss >> >> > > -- View this message in context: http://www.nabble.com/Sorting-performance-tp20037712p23094174.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: synchronizing slave indexes in distributing collections
Hi, I would like to know where are you about your script which take the slave out of the load balancer ?? I've no choice to do that during update on the slave server. Thanks, Yu-Hui Jin wrote: > > Thanks, guys. > > Glad to know the scripts work very well in your experience. (well, indeed > they are quite simple.) So that's how I imagine we should do it except > that > you guys added a very good point -- that the monitoring system can invoke > a > script to take the slave out of the load balancer. I'd like to implement > this idea. > > > Cheers, > > -Hui > > On 8/17/07, Bill Au wrote: >> >> If snapinstaller fails to install the lastest snapshot, then chances are >> that it would be able to install any earlier snapshots as well. All it >> does >> is some very simple filesystem operations and then invoke the Solr server >> to >> do a commit. I agree with Chris that the best thing to do is to take it >> out >> of rotation and fix the underlying problem. >> >> Bill >> >> On 8/17/07, Chris Hostetter wrote: >> > >> > >> > : So looks like all we can do is it monitoring the logs and alarm >> people >> > to >> > : fix the issue and rerun the scripts, etc. whenever failures occur. Is >> > that >> > : the correct understanding? >> > >> > I have *never* seen snappuller or snapinstaller fail (except during an >> > initial rollout of Solr when i forgot to setup the neccessary ssh >> keys). >> > >> > I suppose we could at an option to snapinstaller to support explicitly >> > installing a snapshot by name ... then if you detect that salve Z >> didn't >> > load the latest snapshot, you could always tell the other slaves to >> > snapinstall whatever older version slave Z is still using -- but >> frankly >> > that seems a little silly -- not to mention that if you couldn't load >> the >> > snapshot into Z, odds are Z isn't responding to queries either. >> > >> > a better course of action might just be to have an automated system >> which >> > monitors the distribution status info on the master, and takes any >> slaves >> > that don't update it properly out of your load balances rotation (and >> > notifies people to look into it) >> > >> > >> > >> > -Hoss >> > >> > >> > > > > -- > Regards, > > -Hui > > -- View this message in context: http://www.nabble.com/synchronizing-slave-indexes-in-distributing-collections-tp12194297p23039732.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Random queries extremely slow
Hi Oleg Did you find a way to pass over this issue ?? thanks a lot, oleg_gnatovskiy wrote: > > Can you expand on this? Mirroring delay on what? > > > > zayhen wrote: >> >> Use multiple boxes, with a mirroring delaay from one to another, like a >> pipeline. >> >> 2009/1/22 oleg_gnatovskiy >> >>> >>> Well this probably isn't the cause of our random slow queries, but might >>> be >>> the cause of the slow queries after pulling a new index. Is there >>> anything >>> we could do to reduce the performance hit we take from this happening? >>> >>> >>> >>> Otis Gospodnetic wrote: >>> > >>> > Here is one example: pushing a large newly optimized index onto the >>> > server. >>> > >>> > Otis >>> > -- >>> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch >>> > >>> > >>> > >>> > - Original Message >>> >> From: oleg_gnatovskiy >>> >> To: solr-user@lucene.apache.org >>> >> Sent: Thursday, January 22, 2009 2:22:51 PM >>> >> Subject: Re: Random queries extremely slow >>> >> >>> >> >>> >> What are some things that could happen to force files out of the >>> cache >>> on >>> >> a >>> >> Linux machine? I don't know what kinds of events to look for... >>> >> >>> >> >>> >> >>> >> >>> >> yonik wrote: >>> >> > >>> >> > On Thu, Jan 22, 2009 at 1:46 PM, oleg_gnatovskiy >>> >> > wrote: >>> >> >> Hello. Our production servers are operating relatively smoothly >>> most >>> >> of >>> >> >> the >>> >> >> time running Solr with 19 million listings. However every once in >>> a >>> >> while >>> >> >> the same query that used to take 100 miliseconds takes 6000. >>> >> > >>> >> > Anything else happening on the system that may have forced some of >>> the >>> >> > index files out of operating system disk cache at these times? >>> >> > >>> >> > -Yonik >>> >> > >>> >> > >>> >> >>> >> -- >>> >> View this message in context: >>> >> >>> http://www.nabble.com/Random-queries-extremely-slow-tp21610568p21611240.html >>> >> Sent from the Solr - User mailing list archive at Nabble.com. >>> > >>> > >>> > >>> >>> -- >>> View this message in context: >>> http://www.nabble.com/Random-queries-extremely-slow-tp21610568p21611454.html >>> Sent from the Solr - User mailing list archive at Nabble.com. >>> >>> >> >> >> -- >> Alexander Ramos Jardim >> >> >> - >> RPG da Ilha >> > > -- View this message in context: http://www.nabble.com/Random-queries-extremely-slow-tp21610568p23039151.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: commit / new searcher delay?
Hi Hossman, I would love to know either how do you manage this ? thanks, Shalin Shekhar Mangar wrote: > > On Fri, Mar 6, 2009 at 8:47 AM, Steve Conover wrote: > >> That's exactly what I'm doing, but I'm explicitly replicating, and >> committing. Even under these circumstances, what could explain the >> delay after commit before the new index becomes available? >> > > How are you explicitly replicating? I mean, how do you make sure that the > slave has actually finished replication and the new index is available > now? > Are you using the script based replication or the new java based one? > > -- > Regards, > Shalin Shekhar Mangar. > > -- View this message in context: http://www.nabble.com/commit---new-searcher-delay--tp22342916p23036207.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: solr 1.4 memory jvm
do you have an idea? sunnyfr wrote: > > Hi Noble, > > Yes exactly that, > I would like to know how people do during a replication ? > Do they turn off servers and put a high autowarmCount which turn off the > slave for a while like for my case, 10mn to bring back the new index and > then autowarmCount maybe 10 minutes more. > > Otherwise I tried to put large number of mergefactor but I guess I've too > much update every 30mn something like 2000docs and almost all segment are > modified. > > What would you reckon? :( :) > > Thanks a lot Noble > > > Noble Paul നോബിള് नोब्ळ् wrote: >> >> So what I decipher from the numbers is w/o queries Solr replication is >> not performing too badly. The queries are inherently slow and you wish >> to optimize the query performance itself. >> am I correct? >> >> On Tue, Apr 7, 2009 at 7:50 PM, sunnyfr wrote: >>> >>> Hi, >>> >>> So I did two test on two servers; >>> >>> First server : with just replication every 20mn like you can notice: >>> http://www.nabble.com/file/p22930179/cpu_without_request.png >>> cpu_without_request.png >>> http://www.nabble.com/file/p22930179/cpu2_without_request.jpg >>> cpu2_without_request.jpg >>> >>> Second server : with one first replication and a second one during query >>> test: between 15:32pm and 15h41 >>> during replication (checked on .../admin/replication/index.jsp) my >>> respond >>> time query at the end was around 5000msec >>> after the replication I guess during commitment I couldn't get answer of >>> my >>> query for a long time, I refreshed my page few minutes after. >>> http://www.nabble.com/file/p22930179/cpu_with_request.png >>> cpu_with_request.png >>> http://www.nabble.com/file/p22930179/cpu2_with_request.jpg >>> cpu2_with_request.jpg >>> >>> Now without replication I kept going query on the second server, and I >>> can't >>> get better than >>> 1000msec repond time and 11request/second. >>> http://www.nabble.com/file/p22930179/cpu_.jpg cpu_.jpg >>> >>> This is my request : >>> select?fl=id&fq=status_published:1+AND+status_moderated:0+AND+status_personal:0+AND+status_private:0+AND+status_deleted:0+AND+status_error:0+AND+status_ready_web:1&json.nl=map&wt=json&start=0&version=1.2&bq=status_official:1^1.5+OR+status_creative:1^1+OR+language:en^0.5&bf=recip(rord(created),1,10,10)^3+pow(stat_views,0.1)^15+pow(stat_comments,0.1)^15&rows=100&qt=dismax&qf=title_en^0.8+title^0.2+description_en^0.3+description^0.2+tags^1+owner_login^0.5 >>> >>> Do you have advice ? >>> >>> Thanks Noble >>> >>> >>> -- >>> View this message in context: >>> http://www.nabble.com/solr-1.4-memory-jvm-tp22913742p22930179.html >>> Sent from the Solr - User mailing list archive at Nabble.com. >>> >>> >> >> >> >> -- >> --Noble Paul >> >> > > -- View this message in context: http://www.nabble.com/solr-1.4-memory-jvm-tp22913742p23035520.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Any tips for indexing large amounts of data?
ok but how people do for a frequent update for a large dabase and lot of query on it ? do they turn off the slave during the warmup ?? Noble Paul നോബിള് नोब्ळ् wrote: > > On Thu, Apr 9, 2009 at 8:51 PM, sunnyfr wrote: >> >> Hi Otis, >> How did you manage that? I've 8 core machine with 8GB of ram and 11GB >> index >> for 14M docs and 5 update every 30mn but my replication kill >> everything. >> My segments are merged too often sor full index replicate and cache lost >> and >> I've no idea what can I do now? >> Some help would be brilliant, >> btw im using Solr 1.4. >> > > sunnnyfr , whether the replication is full or delta , the caches are > lost completely. > > you can think of partitioning the index into separate Solrs and > updating one partition at a time and perform distributed search. > >> Thanks, >> >> >> Otis Gospodnetic wrote: >>> >>> Mike is right about the occasional slow-down, which appears as a pause >>> and >>> is due to large Lucene index segment merging. This should go away with >>> newer versions of Lucene where this is happening in the background. >>> >>> That said, we just indexed about 20MM documents on a single 8-core >>> machine >>> with 8 GB of RAM, resulting in nearly 20 GB index. The whole process >>> took >>> a little less than 10 hours - that's over 550 docs/second. The vanilla >>> approach before some of our changes apparently required several days to >>> index the same amount of data. >>> >>> Otis >>> -- >>> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch >>> >>> - Original Message >>> From: Mike Klaas >>> To: solr-user@lucene.apache.org >>> Sent: Monday, November 19, 2007 5:50:19 PM >>> Subject: Re: Any tips for indexing large amounts of data? >>> >>> There should be some slowdown in larger indices as occasionally large >>> segment merge operations must occur. However, this shouldn't really >>> affect overall speed too much. >>> >>> You haven't really given us enough data to tell you anything useful. >>> I would recommend trying to do the indexing via a webapp to eliminate >>> all your code as a possible factor. Then, look for signs to what is >>> happening when indexing slows. For instance, is Solr high in cpu, is >>> the computer thrashing, etc? >>> >>> -Mike >>> >>> On 19-Nov-07, at 2:44 PM, Brendan Grainger wrote: >>> >>>> Hi, >>>> >>>> Thanks for answering this question a while back. I have made some >>>> of the suggestions you mentioned. ie not committing until I've >>>> finished indexing. What I am seeing though, is as the index get >>>> larger (around 1Gb), indexing is taking a lot longer. In fact it >>>> slows down to a crawl. Have you got any pointers as to what I might >>>> be doing wrong? >>>> >>>> Also, I was looking at using MultiCore solr. Could this help in >>>> some way? >>>> >>>> Thank you >>>> Brendan >>>> >>>> On Oct 31, 2007, at 10:09 PM, Chris Hostetter wrote: >>>> >>>>> >>>>> : I would think you would see better performance by allowing auto >>>>> commit >>>>> : to handle the commit size instead of reopening the connection >>>>> all the >>>>> : time. >>>>> >>>>> if your goal is "fast" indexing, don't use autoCommit at all ... >>> just >>>>> index everything, and don't commit until you are completely done. >>>>> >>>>> autoCommitting will slow your indexing down (the benefit being >>>>> that more >>>>> results will be visible to searchers as you proceed) >>>>> >>>>> >>>>> >>>>> >>>>> -Hoss >>>>> >>>> >>> >>> >>> >>> >>> >>> >> >> -- >> View this message in context: >> http://www.nabble.com/Any-tips-for-indexing-large-amounts-of-data--tp13510670p22973205.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > > > -- > --Noble Paul > > -- View this message in context: http://www.nabble.com/Any-tips-for-indexing-large-amounts-of-data--tp13510670p22986152.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Performance when indexing or cold cache
Hi Walter, Did you find a way to sort out your issue, I would be very interested. Thanks a lot, Walter Underwood wrote: > > We've had some performance problems while Solr is indexing and also when > it > starts with a cold cache. I'm still digging through our own logs, but I'd > like to get more info about this, so any ideas or info are welcome. > > We have four Solr servers on dual CPU PowerPC machines, 2G of heap, about > 100-300 queries/second, 250K docs, Tomcat 6.0.10, not fronted by Apache. > We don't use facets, we sort by score. In general use, there are six > different request handlers called to build a page. Here is one, they > are all very similar. > > > > 0.01 > > exact^8.0 exact_alt^6.0 exact_base^8.0 title^4.0 title_alt^3.0 > title_base^4.0 phonetic_hi^1.0 > > > exact^12.0 exact_alt^9.0 exact_base^12.0 title^6.0 title_alt^4.0 > title_base^6.0 phonetic_hi^1.5 > > > popularity^2.0 > > > id,type,movieid,personid,genreid,score > > 1 > 100 > > > (pushstatus:A AND (type:movie OR type:person)) > > > > wunder > > > > -- View this message in context: http://www.nabble.com/Performance-when-indexing-or-cold-cache-tp13348420p22984912.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Any tips for indexing large amounts of data?
Hi Otis, How did you manage that? I've 8 core machine with 8GB of ram and 11GB index for 14M docs and 5 update every 30mn but my replication kill everything. My segments are merged too often sor full index replicate and cache lost and I've no idea what can I do now? Some help would be brilliant, btw im using Solr 1.4. Thanks, Otis Gospodnetic wrote: > > Mike is right about the occasional slow-down, which appears as a pause and > is due to large Lucene index segment merging. This should go away with > newer versions of Lucene where this is happening in the background. > > That said, we just indexed about 20MM documents on a single 8-core machine > with 8 GB of RAM, resulting in nearly 20 GB index. The whole process took > a little less than 10 hours - that's over 550 docs/second. The vanilla > approach before some of our changes apparently required several days to > index the same amount of data. > > Otis > -- > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > > - Original Message > From: Mike Klaas > To: solr-user@lucene.apache.org > Sent: Monday, November 19, 2007 5:50:19 PM > Subject: Re: Any tips for indexing large amounts of data? > > There should be some slowdown in larger indices as occasionally large > segment merge operations must occur. However, this shouldn't really > affect overall speed too much. > > You haven't really given us enough data to tell you anything useful. > I would recommend trying to do the indexing via a webapp to eliminate > all your code as a possible factor. Then, look for signs to what is > happening when indexing slows. For instance, is Solr high in cpu, is > the computer thrashing, etc? > > -Mike > > On 19-Nov-07, at 2:44 PM, Brendan Grainger wrote: > >> Hi, >> >> Thanks for answering this question a while back. I have made some >> of the suggestions you mentioned. ie not committing until I've >> finished indexing. What I am seeing though, is as the index get >> larger (around 1Gb), indexing is taking a lot longer. In fact it >> slows down to a crawl. Have you got any pointers as to what I might >> be doing wrong? >> >> Also, I was looking at using MultiCore solr. Could this help in >> some way? >> >> Thank you >> Brendan >> >> On Oct 31, 2007, at 10:09 PM, Chris Hostetter wrote: >> >>> >>> : I would think you would see better performance by allowing auto >>> commit >>> : to handle the commit size instead of reopening the connection >>> all the >>> : time. >>> >>> if your goal is "fast" indexing, don't use autoCommit at all ... > just >>> index everything, and don't commit until you are completely done. >>> >>> autoCommitting will slow your indexing down (the benefit being >>> that more >>> results will be visible to searchers as you proceed) >>> >>> >>> >>> >>> -Hoss >>> >> > > > > > > -- View this message in context: http://www.nabble.com/Any-tips-for-indexing-large-amounts-of-data--tp13510670p22973205.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Snapinstaller vs Solr Restart
Hi Otis, Ok about that, but still when it merges segments it changes names and I've no choice to replicate all the segment which is bad for the replication and cpu. ?? Thanks Otis Gospodnetic wrote: > > Lower your mergeFactor and Lucene will merge segments(i.e. fewer index > files) and purge deletes more often for you at the expense of somewhat > slower indexing. > > > Otis > -- > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > > > > - Original Message >> From: wojtekpia >> To: solr-user@lucene.apache.org >> Sent: Tuesday, January 6, 2009 5:18:26 PM >> Subject: Re: Snapinstaller vs Solr Restart >> >> >> I'm optimizing because I thought I should. I'll be updating my index >> somewhere between every 15 minutes, and every 2 hours. That means between >> 12 >> and 96 updates per day. That seems like a lot of index files (and it >> scared >> me a little), so that's my second reason for wanting to optimize nightly. >> >> I haven't benchmarked the performance hit for not optimizing. That'll be >> my >> next step. If the hit isn't too bad, I'll look into optimizing less >> frequently (weekly, ...). >> >> Thanks Otis! >> >> >> Otis Gospodnetic wrote: >> > >> > OK, so that question/answer seems to have hit the nail on the head. :) >> > >> > When you optimize your index, all index files get rewritten. This >> means >> > that everything that the OS cached up to that point goes out the window >> > and the OS has to slowly re-cache the hot parts of the index. If you >> > don't optimize, this won't happen. Do you really need to optimize? Or >> > maybe a more direct question: why are you optimizing? >> > >> > >> > Regarding autowarming, with such high fq hit rate, I'd make good use of >> fq >> > autowarming. The result cache rate is lower, but still decent. I >> > wouldn't turn off autowarming the way you have. >> > >> > >> >> -- >> View this message in context: >> http://www.nabble.com/Snapinstaller-vs-Solr-Restart-tp21315273p21320334.html >> Sent from the Solr - User mailing list archive at Nabble.com. > > > -- View this message in context: http://www.nabble.com/Snapinstaller-vs-Solr-Restart-tp21315273p22972780.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: solr 1.4 facet boost field according to another field
Do you have an idea ? sunnyfr wrote: > > Hi, > > I've title description and tag field ... According to where I find the > word searched, I would like to boost differently other field like nb_views > or rating. > > if word is find in title then nb_views^10 and rating^10 > if word is find in description then nb_views^2 and rating^2 > > Thanks a lot for your help, > -- View this message in context: http://www.nabble.com/solr-1.4-facet-boost-field-according-to-another-field-tp22913642p2294.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: solr 1.4 memory jvm
Hi Noble, Yes exactly that, I would like to know how people do during a replication ? Do they turn off servers and put a high autowarmCount which turn off the slave for a while like for my case, 10mn to bring back the new index and then autowarmCount maybe 10 minutes more. Otherwise I tried to put large number of mergefactor but I guess I've too much update every 30mn something like 2000docs and almost all segment are modified. What would you reckon? :( :) Thanks a lot Noble Noble Paul നോബിള് नोब्ळ् wrote: > > So what I decipher from the numbers is w/o queries Solr replication is > not performing too badly. The queries are inherently slow and you wish > to optimize the query performance itself. > am I correct? > > On Tue, Apr 7, 2009 at 7:50 PM, sunnyfr wrote: >> >> Hi, >> >> So I did two test on two servers; >> >> First server : with just replication every 20mn like you can notice: >> http://www.nabble.com/file/p22930179/cpu_without_request.png >> cpu_without_request.png >> http://www.nabble.com/file/p22930179/cpu2_without_request.jpg >> cpu2_without_request.jpg >> >> Second server : with one first replication and a second one during query >> test: between 15:32pm and 15h41 >> during replication (checked on .../admin/replication/index.jsp) my >> respond >> time query at the end was around 5000msec >> after the replication I guess during commitment I couldn't get answer of >> my >> query for a long time, I refreshed my page few minutes after. >> http://www.nabble.com/file/p22930179/cpu_with_request.png >> cpu_with_request.png >> http://www.nabble.com/file/p22930179/cpu2_with_request.jpg >> cpu2_with_request.jpg >> >> Now without replication I kept going query on the second server, and I >> can't >> get better than >> 1000msec repond time and 11request/second. >> http://www.nabble.com/file/p22930179/cpu_.jpg cpu_.jpg >> >> This is my request : >> select?fl=id&fq=status_published:1+AND+status_moderated:0+AND+status_personal:0+AND+status_private:0+AND+status_deleted:0+AND+status_error:0+AND+status_ready_web:1&json.nl=map&wt=json&start=0&version=1.2&bq=status_official:1^1.5+OR+status_creative:1^1+OR+language:en^0.5&bf=recip(rord(created),1,10,10)^3+pow(stat_views,0.1)^15+pow(stat_comments,0.1)^15&rows=100&qt=dismax&qf=title_en^0.8+title^0.2+description_en^0.3+description^0.2+tags^1+owner_login^0.5 >> >> Do you have advice ? >> >> Thanks Noble >> >> >> -- >> View this message in context: >> http://www.nabble.com/solr-1.4-memory-jvm-tp22913742p22930179.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > > > -- > --Noble Paul > > -- View this message in context: http://www.nabble.com/solr-1.4-memory-jvm-tp22913742p22966630.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Times Replicated Since Startup: 109 since yesterday afternoon?
Hi, Commit can't throw away the cache, so how people do? everytime they have update to do? they lost their cache ??? I still can't imagine how the search engine can works this way? How people do to have update on their website? Shalin Shekhar Mangar wrote: > > On Mon, Mar 30, 2009 at 10:31 PM, sunnyfr wrote: > >> >> So except commit/optimize or replicate with a time poll less often, I >> can't >> change this ??? >> So replication when you have loads of data updated every 30mn is not >> adviced. >> Or I must replicate once a day ??? or ..? >> >> > Yes, commits are expensive. It is not advised to do them very often since > the caches are thrown away. > > However, there is a lot of work going on to make commits cheaper. Expect > to > see some improvements in the coming months. > > -- > Regards, > Shalin Shekhar Mangar. > > -- View this message in context: http://www.nabble.com/Times-Replicated-Since-Startup%3A-109--since-yesterday-afternoon--tp22784943p22952509.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: solr 1.4 memory jvm
Do you have an idea? sunnyfr wrote: > > Hi, > > So I did two test on two servers; > > First server : with just replication every 20mn like you can notice: > http://www.nabble.com/file/p22930179/cpu_without_request.png > cpu_without_request.png > http://www.nabble.com/file/p22930179/cpu2_without_request.jpg > cpu2_without_request.jpg > > Second server : with one first replication and a second one during query > test: between 15:32pm and 15h41 > during replication (checked on .../admin/replication/index.jsp) my respond > time query at the end was around 5000msec > after the replication I guess during commitment I couldn't get answer of > my query for a long time, I refreshed my page few minutes after. > http://www.nabble.com/file/p22930179/cpu_with_request.png > cpu_with_request.png > http://www.nabble.com/file/p22930179/cpu2_with_request.jpg > cpu2_with_request.jpg > > Now without replication I kept going query on the second server, and I > can't get better than > 1000msec repond time and 11request/second. > http://www.nabble.com/file/p22930179/cpu_.jpg cpu_.jpg > > This is my request : > select?fl=id&fq=status_published:1+AND+status_moderated:0+AND+status_personal:0+AND+status_private:0+AND+status_deleted:0+AND+status_error:0+AND+status_ready_web:1&json.nl=map&wt=json&start=0&version=1.2&bq=status_official:1^1.5+OR+status_creative:1^1+OR+language:en^0.5&bf=recip(rord(created),1,10,10)^3+pow(stat_views,0.1)^15+pow(stat_comments,0.1)^15&rows=100&qt=dismax&qf=title_en^0.8+title^0.2+description_en^0.3+description^0.2+tags^1+owner_login^0.5 > > Do you have advice ? > > Thanks Noble > > > -- View this message in context: http://www.nabble.com/solr-1.4-memory-jvm-tp22913742p22952324.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: solr 1.4 memory jvm
Hi, So I did two test on two servers; First server : with just replication every 20mn like you can notice: http://www.nabble.com/file/p22930179/cpu_without_request.png cpu_without_request.png http://www.nabble.com/file/p22930179/cpu2_without_request.jpg cpu2_without_request.jpg Second server : with one first replication and a second one during query test: between 15:32pm and 15h41 during replication (checked on .../admin/replication/index.jsp) my respond time query at the end was around 5000msec after the replication I guess during commitment I couldn't get answer of my query for a long time, I refreshed my page few minutes after. http://www.nabble.com/file/p22930179/cpu_with_request.png cpu_with_request.png http://www.nabble.com/file/p22930179/cpu2_with_request.jpg cpu2_with_request.jpg Now without replication I kept going query on the second server, and I can't get better than 1000msec repond time and 11request/second. http://www.nabble.com/file/p22930179/cpu_.jpg cpu_.jpg This is my request : select?fl=id&fq=status_published:1+AND+status_moderated:0+AND+status_personal:0+AND+status_private:0+AND+status_deleted:0+AND+status_error:0+AND+status_ready_web:1&json.nl=map&wt=json&start=0&version=1.2&bq=status_official:1^1.5+OR+status_creative:1^1+OR+language:en^0.5&bf=recip(rord(created),1,10,10)^3+pow(stat_views,0.1)^15+pow(stat_comments,0.1)^15&rows=100&qt=dismax&qf=title_en^0.8+title^0.2+description_en^0.3+description^0.2+tags^1+owner_login^0.5 Do you have advice ? Thanks Noble -- View this message in context: http://www.nabble.com/solr-1.4-memory-jvm-tp22913742p22930179.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: solr 1.4 memory jvm
Hi Noble I turnd off autoWarming to zero. And yes it's during it replicate, it takes all the data index. Because it merges too much, too much update 2000docs every 30mn, it always merge my index. So the replication bring back all my data/index. which use a big part of the cpu like u can see on the graph, the first part>> http://www.nabble.com/file/p22925561/cpu_.jpg cpu_.jpg and on this graph and first part of the graph (blue part) it's just replication no request at all. normally i've 20 request per second what would you reckon ? Noble Paul നോബിള് नोब्ळ् wrote: > > hi sunnyfr, > > I wish to clarify something. > > you say that the performance is poor "during" the replication. > > I suspect that the performance is poor soon after the replication. The > reason being , replication is a low CPU activity. If you think > otherwise let me know how you found it out. > > If the perf is low soon after the replication is completed. I mean the > index files are downloaded and the searcher is getting opened, it is > understandable. That is the time when warming is done. have you setup > auto warming? > > On Mon, Apr 6, 2009 at 11:12 PM, sunnyfr wrote: >> >> Hi, >> >> Sorry I can't find and issue, during my replication my respond time query >> goes very slow. >> I'm using replication handler, is there a way to slow down debit or ??? >> >> 11G index size >> 8G ram >> 20 requests/sec >> Java HotSpot(TM) 64-Bit Server VM >> >> >> 10.0-b22 >> Java HotSpot(TM) 64-Bit Server VM >> 4 >> >> -Xms4G >> -Xmx5G >> -XX:ScavengeBeforeFullGC >> -XX:+UseConcMarkSweepGC >> -XX:+HeapDumpOnOutOfMemoryError >> -Xloggc:/data/solr/logs/gc.log >> -XX:+PrintGCDetails >> -XX:+PrintGCTimeStam >> - >> >> >> Is it a problem ?? >> 0.21 >> (error executing: uname -a) >> (error executing: ulimit -n) >> (error executing: uptime) >> >> Thanks >> >> -- >> View this message in context: >> http://www.nabble.com/solr-1.4-memory-jvm-tp22913742p22913742.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > > > -- > --Noble Paul > > -- View this message in context: http://www.nabble.com/solr-1.4-memory-jvm-tp22913742p22925561.html Sent from the Solr - User mailing list archive at Nabble.com.
solr 1.4 memory jvm
Hi, Sorry I can't find and issue, during my replication my respond time query goes very slow. I'm using replication handler, is there a way to slow down debit or ??? 11G index size 8G ram 20 requests/sec Java HotSpot(TM) 64-Bit Server VM 10.0-b22 Java HotSpot(TM) 64-Bit Server VM 4 -Xms4G -Xmx5G -XX:ScavengeBeforeFullGC -XX:+UseConcMarkSweepGC -XX:+HeapDumpOnOutOfMemoryError -Xloggc:/data/solr/logs/gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStam − Is it a problem ?? 0.21 (error executing: uname -a) (error executing: ulimit -n) (error executing: uptime) Thanks -- View this message in context: http://www.nabble.com/solr-1.4-memory-jvm-tp22913742p22913742.html Sent from the Solr - User mailing list archive at Nabble.com.
solr 1.4 indexation or request > memory
Hi I would like to know if it use less memory to facet or put weight to a field when I index it then when I make a dismax request. Thanks, -- View this message in context: http://www.nabble.com/solr-1.4-indexation-or-request-%3E-memory-tp22913679p22913679.html Sent from the Solr - User mailing list archive at Nabble.com.
solr 1.4 facet boost field according to another field
Hi, I've title description and tag field ... According to where I find the word searched, I would like to boost differently other field like nb_views or rating. if word is find in title then nb_views^10 and rating^10 if word is find in description then nb_views^2 and rating^2 Thanks a lot for your help, -- View this message in context: http://www.nabble.com/solr-1.4-facet-boost-field-according-to-another-field-tp22913642p22913642.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: JVM best tune? help ... solr1.4
This is my conf : http://www.nabble.com/file/p22847570/solrconfig.xml solrconfig.xml And this is my delta import: */20 * * * * /usr/bin/wget -q --output-document=/home/video_import.txt http:/master.com:8180/solr/video/dataimport?command=delta-import&optimize=false -- View this message in context: http://www.nabble.com/JVM-best-tune--help-...-solr1.4-tp22846336p22847570.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: JVM best tune? help ... solr1.4
I don't optimize at all. my delta-import&optimize=false I didn't turnd on optimize, I think it merges segment alone, because size increase too quickly? Noble Paul നോബിള് नोब्ळ् wrote: > > it is not a good idea to optimize the index everytime you commit. That > is why your downloads are taking so long > > 2009/4/2 Johanna Mortemousque : >> I've so many update, almost 2 000 every 20mn, that lucene merge my index >> folder, >> so everytime my slave replicate its a new index folder merged so every >> time >> it brings back 10G datas. >> >> And during this time my repond time of my request are very slow. >> What can I check? >> >> Thanks Paul >> >> 2009/4/2 Noble Paul നോബിള് नोब्ळ् >>> >>> slave would not show increased request times because of replication. >>> If it does should be some bug >>> >>> On Thu, Apr 2, 2009 at 6:00 PM, wrote: >>> > I think its the same problem, tune jvm for multi thread ... 20request >>> > seconde. >>> > no?? >>> > >>> > >>> > >>> > Noble Paul നോബിള് नोब्ळ् wrote: >>> >> >>> >> http://people.apache.org/~hossman/#threadhijack >>> >> >>> >> On Thu, Apr 2, 2009 at 5:47 PM, sunnyfr wrote: >>> >>> >>> >>> I think about the slave. >>> >>> When I start in multi thread 20 request second my cpu is very bad. >>> >>> I'm sure I don't manage properly my gc. I've 8G per slave it should >>> be >>> >>> fine. >>> >>> >>> >>> I wonder, I shouldn't put 7G to xmx jvm, I don't know, >>> >>> but slave is as well a little problem during replication from the >>> >>> master. >>> >>> >>> >>> >>> >>> >>> >>> Noble Paul നോബിള് नोब्ळ् wrote: >>> >>>> >>> >>>> If you are looking at the QTime on the master it is likely to be >>> >>>> skewed by ReplicationHandler becaus ethe files are downloaded using >>> a >>> >>>> request. On a slave it should not be a problem. >>> >>>> >>> >>>> I guess we must not add the qtimes of ReplicationHandler >>> >>>> --Noble >>> >>>> >>> >>>> On Thu, Apr 2, 2009 at 5:34 PM, sunnyfr >>> wrote: >>> >>>>> >>> >>>>> Hi, >>> >>>>> >>> >>>>> Just applied replication by requestHandler. >>> >>>>> And since this the Qtime went mad and can reach long time >> >>>>> name="QTime">9068 >>> >>>>> Without this replication Qtime can be around 1sec. >>> >>>>> >>> >>>>> I've 14Mdocs stores for 11G. so not a lot of data stores. >>> >>>>> I've servers with 8G and tomcat use 7G. >>> >>>>> I'm updating every 30mn which is about 50 000docs. >>> >>>>> Have a look as well at my cpu which are aswell quite full ? >>> >>>>> >>> >>>>> Have you an idea? Do I miss a patch ? >>> >>>>> Thanks a lot, >>> >>>>> >>> >>>>> Solr Specification Version: 1.3.0.2009.01.22.13.51.22 >>> >>>>> Solr Implementation Version: 1.4-dev exported - root - 2009-01-22 >>> >>>>> 13:51:22 >>> >>>>> >>> >>>>> http://www.nabble.com/file/p22846336/CPU.jpg CPU.jpg >>> >>>>> -- >>> >>>>> View this message in context: >>> >>>>> >>> >>>>> >>> http://www.nabble.com/JVM-best-tune--help-...-solr1.4-tp22846336p22846336.html >>> >>>>> Sent from the Solr - User mailing list archive at Nabble.com. >>> >>>>> >>> >>>>> >>> >>>> >>> >>>> >>> >>>> >>> >>>> -- >>> >>>> --Noble Paul >>> >>>> >>> >>>> >>> >>> >>> >>> -- >>> >>> View this message in context: >>> >>> >>> >>> >>> http://www.nabble.com/JVM-best-tune--help-...-solr1.4-tp22846336p22846546.html >>> >>> Sent from the Solr - User mailing list archive at Nabble.com. >>> >>> >>> >>> >>> >> >>> >> >>> >> >>> >> -- >>> >> --Noble Paul >>> >> >>> >> >>> > Quoted from: >>> > >>> > >>> http://www.nabble.com/JVM-best-tune--help-...-solr1.4-tp22846336p22846716.html >>> > >>> > >>> >>> >>> >>> -- >>> --Noble Paul >> >> > > > > -- > --Noble Paul > > -- View this message in context: http://www.nabble.com/JVM-best-tune--help-...-solr1.4-tp22846336p22847455.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: JVM best tune? help ... solr1.4
I think about the slave. When I start in multi thread 20 request second my cpu is very bad. I'm sure I don't manage properly my gc. I've 8G per slave it should be fine. I wonder, I shouldn't put 7G to xmx jvm, I don't know, but slave is as well a little problem during replication from the master. Noble Paul നോബിള് नोब्ळ् wrote: > > If you are looking at the QTime on the master it is likely to be > skewed by ReplicationHandler becaus ethe files are downloaded using a > request. On a slave it should not be a problem. > > I guess we must not add the qtimes of ReplicationHandler > --Noble > > On Thu, Apr 2, 2009 at 5:34 PM, sunnyfr wrote: >> >> Hi, >> >> Just applied replication by requestHandler. >> And since this the Qtime went mad and can reach long time > name="QTime">9068 >> Without this replication Qtime can be around 1sec. >> >> I've 14Mdocs stores for 11G. so not a lot of data stores. >> I've servers with 8G and tomcat use 7G. >> I'm updating every 30mn which is about 50 000docs. >> Have a look as well at my cpu which are aswell quite full ? >> >> Have you an idea? Do I miss a patch ? >> Thanks a lot, >> >> Solr Specification Version: 1.3.0.2009.01.22.13.51.22 >> Solr Implementation Version: 1.4-dev exported - root - 2009-01-22 >> 13:51:22 >> >> http://www.nabble.com/file/p22846336/CPU.jpg CPU.jpg >> -- >> View this message in context: >> http://www.nabble.com/JVM-best-tune--help-...-solr1.4-tp22846336p22846336.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > > > -- > --Noble Paul > > -- View this message in context: http://www.nabble.com/JVM-best-tune--help-...-solr1.4-tp22846336p22846546.html Sent from the Solr - User mailing list archive at Nabble.com.
JVM best tune? help ... solr1.4
Hi, Just applied replication by requestHandler. And since this the Qtime went mad and can reach long time 9068 Without this replication Qtime can be around 1sec. I've 14Mdocs stores for 11G. so not a lot of data stores. I've servers with 8G and tomcat use 7G. I'm updating every 30mn which is about 50 000docs. Have a look as well at my cpu which are aswell quite full ? Have you an idea? Do I miss a patch ? Thanks a lot, Solr Specification Version: 1.3.0.2009.01.22.13.51.22 Solr Implementation Version: 1.4-dev exported - root - 2009-01-22 13:51:22 http://www.nabble.com/file/p22846336/CPU.jpg CPU.jpg -- View this message in context: http://www.nabble.com/JVM-best-tune--help-...-solr1.4-tp22846336p22846336.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: autowarm static queries
Ok so It doesn't seems to work after a replication, my first request on my slave is always very long and next one very quick ??? do I have to set something else ? solr 0 100 anything id desc fast_warm 0 100 anything id desc anything recip(rord(created),1,10,10)^3+pow(stat_views,0.1)^15+pow(stat_comments,0.1)^15 status_published:1+AND+status_moderated:0+AND+status_personal:0+AND+status_private:0+AND+status_deleted:0+AND+status_error:0+AND+status_ready_web:1 status_official:1^1.5+OR+status_creative:1^1+OR+language:en^0.5 title^0.2+description^0.2+tags^1+owner_login^0.5 Shalin Shekhar Mangar wrote: > > On Thu, Apr 2, 2009 at 2:13 PM, sunnyfr wrote: >> >> Hi Hoss, >> >> Do I need autowarming > 0 to have newSearcher and firstSearcher fired? >> >> Thanks a lot, >> > > Did you mean autowarmCount > 0? > > No, firstSearcher and newSearcher are always executed if specified. > The autowarmCount can be anything, it does not matter. > > -- > Regards, > Shalin Shekhar Mangar. > > -- View this message in context: http://www.nabble.com/autowarm-partial-queries-in-solrconfig.xml-tp13167933p22844377.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: autowarm static queries
Hi Hoss, Do I need autowarming > 0 to have newSearcher and firstSearcher fired? Thanks a lot, hossman wrote: > > > : Subject: autowarm static queries > > A minor followup about terminology: > > "auto-warming" describes what Solr does when it opens a new cache, and > seeds it with key/val pairs based on the "top" keys from the old instance > of the cache. > > "static warming" describes what you can do using newSearcher and > firstSearcher event listeners to force explicit warming actions to be > taken when one of these events happens -- frequently it involves seeding > one or more caches with values from "static" queries hard coded in the > solrconfig.xml > > i'm not sure what it would mean to autowarm a static query. > > > -Hoss > > > -- View this message in context: http://www.nabble.com/autowarm-partial-queries-in-solrconfig.xml-tp13167933p22843453.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: how to improve concurrent request performance and stress testing
Thanks for all this help, But I guess it can't be optimal with a lot of update, my slave get back from the master 20 000docs updated every 20minutes, it's made to try to warmup have a big cache and everything go fast with that amount of update I guess ...? zqzuk wrote: > > Hi, > > try to firstly have a look at http://wiki.apache.org/solr/SolrCaching the > section on firstsearcher and warming. Search engines rely on caching, so > first searches will be slow. I think to be fair testing it is necessary to > warm up the search engine by sending most frequently used and/or most > costly queries, then start your stress testing. > > I used this tool http://code.google.com/p/httpstone/ to do stress testing. > It allows you to create multiple threads sending queries to a server > simultaneously, and records time taken to process each query in each > thread. > > Hope it helps. > > > > sunnyfr wrote: >> >> Hi, >> >> I'm trying as well to stress test solr. I would love some advice to >> manage it properly. >> I'm using solr 1.3 and tomcat55. >> Thanks a lot, >> >> >> zqzuk wrote: >>> >>> Hi, I am doing a stress testing of my solr application to see how many >>> concurrent requests it can handle and how long it takes. But I m not >>> sure if I have done it in proper way... responses seem to be very slow >>> >>> My configuration: >>> 1 Solr instance, using the default settings distributed in the example >>> code, while I made two changes: >>> true >>> 10 >>> As I thought the more searchers the more concurrent requests can be >>> dealt with? >>> >>> There are 1.1 million documents indexed, and the platform is winxp sp2, >>> duo core 1.8 GB machine with ram 2GB >>> >>> I used httpstone, a simple server load testing tool to create 100 >>> workers (so 100 threads) each issuing one same query to the server. To >>> deal with a single request of this query it took solr 2 seconds (with >>> facet counts), and 7 documents are returned. I was assuming that only >>> first request would take longer time and following requests should be >>> almost instantaneous as the query is the same. But strange that the >>> first response took as long as 20 seconds. >>> >>> It looked like that the 100 workers sent same request to solr and then >>> all of a sudden solr server went silent. Only after 20 seconds some of >>> these workers started to receive responses, but still very slow. >>> >>> clearly there I must have made something wrong with configuring solr >>> server... Could you give me some pointers on how to improve the >>> performance please? >>> >>> Many thanks! >>> >> >> > > -- View this message in context: http://www.nabble.com/how-to-improve-concurrent-request-performance-and-stress-testing-tp15299687p22827717.html Sent from the Solr - User mailing list archive at Nabble.com.
solr 1.4 IndexReaders are in read-only mode
Hi, How can I be sure about that my IndexReaders are in read-only mode? Thanks a lot , -- View this message in context: http://www.nabble.com/solr-1.4-IndexReaders-are-in-read-only-mode-tp22804955p22804955.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Times Replicated Since Startup: 109 since yesterday afternoon?
So except commit/optimize or replicate with a time poll less often, I can't change this ??? So replication when you have loads of data updated every 30mn is not adviced. Or I must replicate once a day ??? or ..? Shalin Shekhar Mangar wrote: > > On Mon, Mar 30, 2009 at 8:17 PM, sunnyfr wrote: > >> >> Hi, >> >> Can you explain me more about this replication script in solr 1.4. >> It does work but it always replicate everything from the master so it >> lost >> every cache everything to replicate it. >> I don't get really how it works ? >> > > That's the normal behavior of a commit. The caches contain the internal > document ids assigned by Lucene. When you call commit, the caches are > thrown > away since the commit changes the doc ids. This is the reason for > autowarming/regenerating the caches. > > -- > Regards, > Shalin Shekhar Mangar. > > -- View this message in context: http://www.nabble.com/Times-Replicated-Since-Startup%3A-109--since-yesterday-afternoon--tp22784943p22787805.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Times Replicated Since Startup: 109 since yesterday afternoon?
I've about 30 000 docs updated every 20mn. I just store id and text which is (title description) my index is about 11G -- View this message in context: http://www.nabble.com/Times-Replicated-Since-Startup%3A-109--since-yesterday-afternoon--tp22784943p22785606.html Sent from the Solr - User mailing list archive at Nabble.com.
Times Replicated Since Startup: 109 since yesterday afternoon?
Hi, Can you explain me more about this replication script in solr 1.4. It does work but it always replicate everything from the master so it lost every cache everything to replicate it. I don't get really how it works ? Thanks a lot, -- View this message in context: http://www.nabble.com/Times-Replicated-Since-Startup%3A-109--since-yesterday-afternoon--tp22784943p22784943.html Sent from the Solr - User mailing list archive at Nabble.com.
keepOptimizedOnly solr1.4 trunk version 27/03/2009
Hi, I would like to know more about keepOptimizedOnly, My problem is on the slaves's servers it's a bit slow after a replication and I would like to automatize an optimization after every commit. How can I do that ? Is it this option keepOptimizedOnly? Thanks a lot, -- View this message in context: http://www.nabble.com/keepOptimizedOnly-solr1.4-trunk-version-27-03-2009-tp22784884p22784884.html Sent from the Solr - User mailing list archive at Nabble.com.
solrReplication solr1.4 slave is slower during replication
Hi I would like to know if you leave your slave allowed for searching during a replication. Everytime a replication is applied ... poll is enable, and start to bring back files, my slave have a very batim perf, and can take 5sec to bring back the result, as soon it's done everything is back properly. Do you have the same problem? Do you turn off searching on this slave during this time?? thanks a lot, Johanna -- View this message in context: http://www.nabble.com/solrReplication-solr1.4-slave-is-slower-during-replication-tp22769716p22769716.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: "Unable to move index file" error during replication
Sorry but which one shoud I take??
where exactly ?
Noble Paul നോബിള് नोब्ळ् wrote:
>
> this fix is there in the trunk ,
> you may not need to apply the patch
>
> On Fri, Mar 27, 2009 at 6:02 AM, sunnyfr wrote:
>>
>> Hi,
>>
>> It doesn't seem to work for me, I changed as well this part below is it
>> ok??
>>> - List copiedfiles = new ArrayList();
>>> + Set filesToCopy = new HashSet();
>>
>> http://www.nabble.com/file/p22734005/ReplicationHandler.java
>> ReplicationHandler.java
>>
>> Thanks a lot,
>>
>>
>>
>>
>>
>> Noble Paul നോബിള് नोब्ळ् wrote:
>>>
>>> James thanks .
>>>
>>> If this is true the place to fix this is in
>>> ReplicationHandler#getFileList(). patch is attached.
>>>
>>>
>>> On Wed, Dec 24, 2008 at 4:04 PM, James Grant
>>> wrote:
>>>> I had the same problem. It turned out that the list of files from the
>>>> master
>>>> included duplicates. When the slave completes the download and tries to
>>>> move
>>>> the files into the index it comes across a file that does not exist
>>>> because
>>>> it has already been moved so it backs out the whole operation.
>>>>
>>>> My solution for now was to patch the copyindexFiles method of
>>>> org.apache.solr.handler.SnapPuller so that it normalises the list
>>>> before
>>>> moving the files. This isn't the best solution since it will still
>>>> download
>>>> the file twice but it was the easiest and smallest change to make. The
>>>> patch
>>>> is below
>>>>
>>>> Regards
>>>>
>>>> James
>>>>
>>>> --- src/java/org/apache/solr/handler/SnapPuller.java (revision
>>>> 727347)
>>>> +++ src/java/org/apache/solr/handler/SnapPuller.java (working copy)
>>>> @@ -470,7 +470,7 @@
>>>> */
>>>> private boolean copyIndexFiles(File snapDir, File indexDir) {
>>>> String segmentsFile = null;
>>>> - List copiedfiles = new ArrayList();
>>>> + Set filesToCopy = new HashSet();
>>>> for (Map f : filesDownloaded) {
>>>> String fname = (String) f.get(NAME);
>>>> // the segments file must be copied last
>>>> @@ -482,6 +482,10 @@
>>>> segmentsFile = fname;
>>>> continue;
>>>> }
>>>> + filesToCopy.add(fname);
>>>> + }
>>>> + List copiedfiles = new ArrayList();
>>>> + for (String fname: filesToCopy) {
>>>> if (!copyAFile(snapDir, indexDir, fname, copiedfiles)) return
>>>> false;
>>>> copiedfiles.add(fname);
>>>> }
>>>>
>>>>
>>>> Jaco wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> While testing out the new replication features, I'm running into some
>>>>> strange problem. On the slave, I keep getting an error like this after
>>>>> all
>>>>> files have been copied from the master to the temporary
>>>>> index.x
>>>>> directory:
>>>>>
>>>>> SEVERE: Unable to move index file from:
>>>>> D:\Data\solr\Slave\data\index.20081224110855\_21e.tvx to:
>>>>> D:\Data\Solr\Slave\data\index\_21e.tvx
>>>>>
>>>>> The replication then stops, index remains in original state, so the
>>>>> updates
>>>>> are not available at the slave.
>>>>>
>>>>> This is my replication config at the master:
>>>>>
>>>>> >>>> >
>>>>>
>>>>>
>>>>> commit
>>>>> schema.xml
>>>>>
>>>>>
>>>>>
>>>>> This is the replication config at the slave:
>>>>>
>>>>> >>>> >
>>>>>
>>>>>
>>>>> http://hostnamemaster:8080/solr/Master/replication
>>>>> 00:10:00
>>>>> true
>>>>>
>>>>>
>>>>>
>>>>> I'm running a Solr nightly build of 21.12.2008 in Tomcat 6 on Windows
>>>>> 2003.
>>>>> Initially I thought ther
Re: "Unable to move index file" error during replication
Hi,
It doesn't seem to work for me, I changed as well this part below is it ok??
> -List copiedfiles = new ArrayList();
> +Set filesToCopy = new HashSet();
http://www.nabble.com/file/p22734005/ReplicationHandler.java
ReplicationHandler.java
Thanks a lot,
Noble Paul നോബിള് नोब्ळ् wrote:
>
> James thanks .
>
> If this is true the place to fix this is in
> ReplicationHandler#getFileList(). patch is attached.
>
>
> On Wed, Dec 24, 2008 at 4:04 PM, James Grant
> wrote:
>> I had the same problem. It turned out that the list of files from the
>> master
>> included duplicates. When the slave completes the download and tries to
>> move
>> the files into the index it comes across a file that does not exist
>> because
>> it has already been moved so it backs out the whole operation.
>>
>> My solution for now was to patch the copyindexFiles method of
>> org.apache.solr.handler.SnapPuller so that it normalises the list before
>> moving the files. This isn't the best solution since it will still
>> download
>> the file twice but it was the easiest and smallest change to make. The
>> patch
>> is below
>>
>> Regards
>>
>> James
>>
>> --- src/java/org/apache/solr/handler/SnapPuller.java(revision 727347)
>> +++ src/java/org/apache/solr/handler/SnapPuller.java(working copy)
>> @@ -470,7 +470,7 @@
>> */
>> private boolean copyIndexFiles(File snapDir, File indexDir) {
>>String segmentsFile = null;
>> -List copiedfiles = new ArrayList();
>> +Set filesToCopy = new HashSet();
>>for (Map f : filesDownloaded) {
>> String fname = (String) f.get(NAME);
>> // the segments file must be copied last
>> @@ -482,6 +482,10 @@
>>segmentsFile = fname;
>>continue;
>> }
>> + filesToCopy.add(fname);
>> +}
>> +List copiedfiles = new ArrayList();
>> +for (String fname: filesToCopy) {
>> if (!copyAFile(snapDir, indexDir, fname, copiedfiles)) return false;
>> copiedfiles.add(fname);
>>}
>>
>>
>> Jaco wrote:
>>>
>>> Hello,
>>>
>>> While testing out the new replication features, I'm running into some
>>> strange problem. On the slave, I keep getting an error like this after
>>> all
>>> files have been copied from the master to the temporary index.x
>>> directory:
>>>
>>> SEVERE: Unable to move index file from:
>>> D:\Data\solr\Slave\data\index.20081224110855\_21e.tvx to:
>>> D:\Data\Solr\Slave\data\index\_21e.tvx
>>>
>>> The replication then stops, index remains in original state, so the
>>> updates
>>> are not available at the slave.
>>>
>>> This is my replication config at the master:
>>>
>>>
>>>
>>>
>>>commit
>>>schema.xml
>>>
>>>
>>>
>>> This is the replication config at the slave:
>>>
>>>
>>>
>>>
>>> http://hostnamemaster:8080/solr/Master/replication
>>>00:10:00
>>>true
>>>
>>>
>>>
>>> I'm running a Solr nightly build of 21.12.2008 in Tomcat 6 on Windows
>>> 2003.
>>> Initially I thought there was some problem with disk space, but this is
>>> not
>>> the case. Replication did run fine for intial version of index, but
>>> after
>>> that at some point it didn't work anymore. Any ideas what could be wrong
>>> here?
>>>
>>> Thanks very much in advance, bye,
>>>
>>> Jaco.
>>>
>>>
>>
>>
>
>
>
> --
> --Noble Paul
>
> Index: src/java/org/apache/solr/handler/ReplicationHandler.java
> ===
> --- src/java/org/apache/solr/handler/ReplicationHandler.java (revision
> 729282)
> +++ src/java/org/apache/solr/handler/ReplicationHandler.java (working
> copy)
> @@ -268,7 +268,7 @@
> List> result = new ArrayList Object>>();
> try {
>//get all the files in the commit
> - Collection files = commit.getFileNames();
> + Collection files = new
> HashSet(commit.getFileNames());
>for (String fileName : files) {
> File file = new File(core.getIndexDir(), fileName);
> Map fileMeta = getFileInfo(file);
>
>
--
View this message in context:
http://www.nabble.com/%22Unable-to-move-index-file%22-error-during-replication-tp21157722p22734005.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Size of my index directory increase considerably
Just applied this patch : http://www.nabble.com/Solr-Replication%3A-disk-space-consumed-on-slave-much-higher-than-on--master-td21579171.html#a21622876 It seems to work well now. Do I have to do something else ? Do you reckon something for my configuration ? Thanks a lot -- View this message in context: http://www.nabble.com/Size-of-my-index-directory-increase-considerably-tp22718590p22722075.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: replication requesthandler solr1.4 slow answer
sunnyfr wrote: > > Hi, > > Since I put this functionnality on, on my servers it takes sometimes a > long time to get a respond for a select > sometimes Qtime = 4sec some other 200msec ? > > Do you know why? and when I look at my servers graph, users part is very > used since I've applied this two patch. > Thanks for your help. > > I've applied two recent patch : > > http://www.nabble.com/Solr-Replication%3A-disk-space-consumed-on-slave-much-higher-than-on--master-td21579171.html#a21622876 > > AND > > http://www.nabble.com/%22Unable-to-move-index-file%22-error-during-replication-td21157722.html#a21157722 > > http://www.nabble.com/file/p22721744/cpu_.jpg > http://www.nabble.com/file/p22721754/cpu_.jpg cpu_.jpg -- View this message in context: http://www.nabble.com/replication-requesthandler-solr1.4-slow-answer-tp22721744p22721754.html Sent from the Solr - User mailing list archive at Nabble.com.
replication requesthandler solr1.4 slow answer
Hi, Since I put this functionnality on, on my servers it takes sometimes a long time to get a respond for a select sometimes Qtime = 4sec some other 200msec ? Do you know why? and when I look at my servers graph, users part is very used since I've applied this two patch. Thanks for your help. I've applied two recent patch : http://www.nabble.com/Solr-Replication%3A-disk-space-consumed-on-slave-much-higher-than-on--master-td21579171.html#a21622876 AND http://www.nabble.com/%22Unable-to-move-index-file%22-error-during-replication-td21157722.html#a21157722 http://www.nabble.com/file/p22721744/cpu_.jpg -- View this message in context: http://www.nabble.com/replication-requesthandler-solr1.4-slow-answer-tp22721744p22721744.html Sent from the Solr - User mailing list archive at Nabble.com.
replication requesthandler solr1.4
Hi, Since I put this functionnality on, on my servers it takes sometimes a long time to get a respond for a select sometimes Qtime = 4sec some other 200msec ? Do you know why? and when I look at my servers graph, users part is very used since I've applied this two patch. Thanks for your help. I've applied two recent patch : http://www.nabble.com/Solr-Replication%3A-disk-space-consumed-on-slave-much-higher-than-on--master-td21579171.html#a21622876 AND http://www.nabble.com/%22Unable-to-move-index-file%22-error-during-replication-td21157722.html#a21157722 http://www.nabble.com/file/p22721742/cpu_.jpg -- View this message in context: http://www.nabble.com/replication-requesthandler-solr1.4-tp22721742p22721742.html Sent from the Solr - User mailing list archive at Nabble.com.
Size of my index directory increase considerably
Hi, I don't understand how my index folder can pass from 11G to 45G? Is it a prob with my segment? For information I'm using solr 1.4, i've 14M of docs. The first full import or optimize low down size to 11G. I'm updating data (delta-import) every 30 mn for about 50 000docs updated every time. Maybe my conf are not optimized ? 10 1000 1000 2147483647 10 1000 1 r...@search-01:/data/solr/video/data# ls index/ _2de9.fdt _2de9.tis_2dea.tvd _2dek.tii _2dev.prx _2df6.fdx _2df6.tvd _2dfh.nrm _2dfs.fdt _2dfs.tis _2dft.frq _2dft.tvx _2dfv.fnm _2dfw.frq _2dfx.nrm _2dg4.frq _2dg4.tvx _2de9.fdx _2de9.tvd_2dea.tvf _2dek.tis _2dev.tii _2df6.fnm _2df6.tvf _2dfh.prx _2dfs.fdx _2dfs.tvd _2dft.nrm _2dfu.fnm _2dfv.frq _2dfw.nrm _2dfx.prx _2dg4.nrm segments.gen _2de9.fnm _2de9.tvf_2dea.tvx _2dev.fdt _2dev.tis _2df6.frq _2df6.tvx _2dfh.tii _2dfs.fnm _2dfs.tvf _2dft.prx _2dfu.frq _2dfv.nrm _2dfw.prx _2dfx.tii _2dg4.prx segments_23ik _2de9.frq _2de9.tvx_2dek.fnm _2dev.fdx _2dev.tvd _2df6.nrm _2dfh.fdt _2dfh.tis _2dfs.frq _2dfs.tvx _2dft.tii _2dfu.nrm _2dfv.prx _2dfw.tii _2dfx.tis _2dg4.tii segments_23il _2de9.nrm _2de9_a.del _2dek.frq _2dev.fnm _2dev.tvf _2df6.prx _2dfh.fdx _2dfh.tvd _2dfs.nrm _2dft.fdt _2dft.tis _2dfu.prx _2dfv.tii _2dfw.tis _2dg4.fdt _2dg4.tis _2de9.prx _2dea.fdt_2dek.nrm _2dev.frq _2dev.tvx _2df6.tii _2dfh.fnm _2dfh.tvf _2dfs.prx _2dft.fdx _2dft.tvd _2dfu.tii _2dfv.tis _2dfx.fnm _2dg4.fdx _2dg4.tvd _2de9.tii _2dea.fdx_2dek.prx _2dev.nrm _2df6.fdt _2df6.tis _2dfh.frq _2dfh.tvx _2dfs.tii _2dft.fnm _2dft.tvf _2dfu.tis _2dfw.fnm _2dfx.frq _2dg4.fnm _2dg4.tvf Thanks, Sunny -- View this message in context: http://www.nabble.com/Size-of-my-index-directory-increase-considerably-tp22718590p22718590.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: delta-import commit=false doesn't seems to work
Hi, Sorry I still don't know what should I do ??? I can see in my log which clearly optimize somewhere even if my command is deltaimport&optimize=false is it a parameter to add to the commit or to the snappuller or ??? Mar 24 23:02:44 search-01 jsvc.exec[22812]: Mar 24, 2009 11:02:44 PM org.apache.solr.handler.dataimport.SolrWriter persistStartTime INFO: Wrote last indexed time to dataimport.properties Mar 24 23:02:44 search-01 jsvc.exec[22812]: Mar 24, 2009 11:02:44 PM org.apache.solr.handler.dataimport.DocBuilder commit INFO: Full Import completed successfully Mar 24 23:02:44 search-01 jsvc.exec[22812]: Mar 24, 2009 11:02:44 PM org.apache.solr.update.DirectUpdateHandler2 commit INFO: start commit(optimize=true,waitFlush=false,waitSearcher=true) thanks a lot for your help sunnyfr wrote: > > Like you can see, I did that and I've no information in my DIH but you can > notice in my logs and even my segments > that and optimize is fired alone automaticly? > > > Noble Paul നോബിള് नोब्ळ् wrote: >> >> just hit the DIH without any command and you may be able to see the >> status of the last import. It can tell you whether a commit/optimize >> was performed >> >> On Fri, Mar 20, 2009 at 7:07 PM, sunnyfr wrote: >>> >>> Thanks I gave more information there : >>> http://www.nabble.com/Problem-for-replication-%3A-segment-optimized-automaticly-td22601442.html >>> >>> thanks a lot Paul >>> >>> >>> Noble Paul നോബിള് नोब्ळ् wrote: >>>> >>>> sorry, the whole thing was commented . I did not notice that. I'll >>>> look into that >>>> >>>> 2009/3/20 Noble Paul നോബിള് नोब्ळ् : >>>>> you have set autoCommit every x minutes . it must have invoked commit >>>>> automatically >>>>> >>>>> >>>>> On Thu, Mar 19, 2009 at 4:17 PM, sunnyfr wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> Even if I hit command=delta-import&commit=false&optimize=false >>>>>> I still have commit set in my logs and sometimes even optimize=true, >>>>>> >>>>>> About optimize I wonder if it comes from commitment too close and one >>>>>> is >>>>>> not >>>>>> done, but still I don't know really. >>>>>> >>>>>> Any idea? >>>>>> >>>>>> Thanks a lot, >>>>>> -- >>>>>> View this message in context: >>>>>> http://www.nabble.com/delta-import-commit%3Dfalse-doesn%27t-seems-to-work-tp22597630p22597630.html >>>>>> Sent from the Solr - User mailing list archive at Nabble.com. >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> --Noble Paul >>>>> >>>> >>>> >>>> >>>> -- >>>> --Noble Paul >>>> >>>> >>> >>> -- >>> View this message in context: >>> http://www.nabble.com/Re%3A-delta-import-commit%3Dfalse-doesn%27t-seems-to-work-tp22614216p22620439.html >>> Sent from the Solr - User mailing list archive at Nabble.com. >>> >>> >> >> >> >> -- >> --Noble Paul >> >> > > -- View this message in context: http://www.nabble.com/Re%3A-delta-import-commit%3Dfalse-doesn%27t-seems-to-work-tp22614216p22691417.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Optimize
thanks for your answer, then what fire merging because in my log i've optimize=true, if it's not optimization because I don't fire it, it must me marging how can I stop this?? Thanks a lot, Shalin Shekhar Mangar wrote: > > No, optimize is not automatic. You have to invoke it yourself just like > commits. > > Take a look at the following for examples: > http://wiki.apache.org/solr/UpdateXmlMessages > > On Thu, Oct 2, 2008 at 2:03 PM, sunnyfr wrote: > >> >> >> Hi, >> >> Can somebody explain me a bit how works optimize? >> I read the doc but didn't get really what fire optimize. >> >> Thanks a lot, >> -- >> View this message in context: >> http://www.nabble.com/Optimize-tp19775320p19775320.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > > -- > Regards, > Shalin Shekhar Mangar. > > -- View this message in context: http://www.nabble.com/Optimize-tp19775320p22684113.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Problem for replication : segment optimized automaticly
How can I stop this? Noble Paul നോബിള് नोब्ळ् wrote: > > if the DIH status does not say that it optimized, it is lucene > mergeing the segments > > On Mon, Mar 23, 2009 at 8:15 PM, sunnyfr wrote: >> >> I checked this out but It doesn't say nothing about optimizing. >> I'm sure it's lucene part about merging or I don't know ...?? >> >> >> Noble Paul നോബിള് नोब्ळ् wrote: >>> >>> the easiest way to find out what DIH did is to hit it's status >>> command. It will give you a brief description of what all it did >>> during last import >>> >>> On Mon, Mar 23, 2009 at 12:59 AM, Shalin Shekhar Mangar >>> wrote: >>>> Lucene will automatically merge segments when they exceed the >>>> mergeFactor. >>>> This may be one reason but I'm not sure. >>>> >>>> I checked DataImportHandler's code again. It won't optimize if >>>> optimize=false is specified. >>>> >>>> On Mon, Mar 23, 2009 at 12:43 AM, sunnyfr wrote: >>>> >>>>> >>>>> Do you have any idea ??? >>>>> :( >>>>> >>>>> cheer, >>>>> >>>>> >>>>> sunnyfr wrote: >>>>> > >>>>> > Hi everybody ... still me :) >>>>> > hoo happy day :) >>>>> > >>>>> > Just, I dont get where I miss something, I will try to be clear. >>>>> > >>>>> > this is my index folder (and we can notice the evolution according >>>>> to >>>>> the >>>>> > delta import every 30mn) : >>>>> > >>>>> > r...@search-01:/data/solr# ls video/data/index/ >>>>> > _2bel.fdt _2bel.fnm _2bel.nrm _2bel.tii _2bel.tvd _2bel.tvx >>>>> > _2bem.fdt _2bem.fnm _2bem.nrm _2bem.tii _2bem.tvd _2bem.tvx >>>>> > _2ben.frq _2ben.prx _2ben.tis _2beo.fdx segments.gen >>>>> > _2bel.fdx _2bel.frq _2bel.prx _2bel.tis _2bel.tvf _2bel_1.del >>>>> > _2bem.fdx _2bem.frq _2bem.prx _2bem.tis _2bem.tvf _2ben.fnm >>>>> > _2ben.nrm _2ben.tii _2beo.fdt _2beo.fnm segments_230x >>>>> > r...@search-01:/data/solr# ls video/data/index/ >>>>> > _2bel.fdt _2bel.frq _2bel.tii _2bel.tvf _2bem.fdt _2bem.frq >>>>> > _2bem.tii _2bem.tvf _2ben.frq _2ben.tii _2beo.fdx _2beo.nrm >>>>> > _2beo.tis _2beo.tvx >>>>> > _2bel.fdx _2bel.nrm _2bel.tis _2bel.tvx _2bem.fdx _2bem.nrm >>>>> > _2bem.tis _2bem.tvx _2ben.nrm _2ben.tis _2beo.fnm _2beo.prx >>>>> > _2beo.tvd segments.gen >>>>> > _2bel.fnm _2bel.prx _2bel.tvd _2bel_1.del _2bem.fnm _2bem.prx >>>>> > _2bem.tvd _2ben.fnm _2ben.prx _2beo.fdt _2beo.frq _2beo.tii >>>>> > _2beo.tvf segments_230x >>>>> > r...@search-01:/data/solr# ls video/data/index/ >>>>> > _2beo.fdt _2beo.fdx _2beo.fnm _2beo.frq _2beo.nrm _2beo.prx >>>>> > _2beo.tii _2beo.tis _2beo.tvd _2beo.tvf _2beo.tvx segments.gen >>>>> > segments_230y >>>>> > >>>>> > So as you can notice my segments increased which is perfect for my >>>>> > replication (faster to get JUST last segments) >>>>> > But like you can see in my last ls my segment has been optimized. >>>>> > >>>>> > Like I can notice in my log : >>>>> > Mar 19 15:42:37 search-01 jsvc.exec[23255]: Mar 19, 2009 3:42:37 PM >>>>> > org.apache.solr.handler.dataimport.DocBuilder commit INFO: Full >>>>> Import >>>>> > completed successfully without optimization >>>>> > Mar 19 15:42:37 search-01 jsvc.exec[23255]: Mar 19, 2009 3:42:37 PM >>>>> > org.apache.solr.update.DirectUpdateHandler2 commit INFO: start >>>>> > commit(optimize=true,waitFlush=false,waitSearcher=true) >>>>> > >>>>> > But I didn't fire any optimize, my delta-import is fired like : >>>>> > /solr/video/dataimport?command=delta-import&optimize=false >>>>> > >>>>> > Solrconfig.xml : autocommit turnd off >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > Maybe it comes from lucene parameters? >>>>> > >>>>> > false >>>>> > 50 >>>>> > 50 >>>>> > >>>>> > >>>>> > 2147483647 >>>>> > 1 >>>>> > >>>>> > Thanks a lot for your help, >>>>> > Sunny >>>>> > >>>>> > >>>>> > >>>>> > >>>>> >>>>> -- >>>>> View this message in context: >>>>> http://www.nabble.com/Problem-for-replication-%3A-segment-optimized-automaticly-tp22601442p22649412.html >>>>> Sent from the Solr - User mailing list archive at Nabble.com. >>>>> >>>>> >>>> >>>> >>>> -- >>>> Regards, >>>> Shalin Shekhar Mangar. >>>> >>> >>> >>> >>> -- >>> --Noble Paul >>> >>> >> >> -- >> View this message in context: >> http://www.nabble.com/Problem-for-replication-%3A-segment-optimized-automaticly-tp22601442p22661545.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > > > -- > --Noble Paul > > -- View this message in context: http://www.nabble.com/Problem-for-replication-%3A-segment-optimized-automaticly-tp22601442p22675729.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Problem for replication : segment optimized automaticly
Where do you see that I can maybe check in my code, is it in updateHandler2 ? thanks, Shalin Shekhar Mangar wrote: > > Lucene will automatically merge segments when they exceed the mergeFactor. > This may be one reason but I'm not sure. > > I checked DataImportHandler's code again. It won't optimize if > optimize=false is specified. > > On Mon, Mar 23, 2009 at 12:43 AM, sunnyfr wrote: > >> >> Do you have any idea ??? >> :( >> >> cheer, >> >> >> sunnyfr wrote: >> > >> > Hi everybody ... still me :) >> > hoo happy day :) >> > >> > Just, I dont get where I miss something, I will try to be clear. >> > >> > this is my index folder (and we can notice the evolution according to >> the >> > delta import every 30mn) : >> > >> > r...@search-01:/data/solr# ls video/data/index/ >> > _2bel.fdt _2bel.fnm _2bel.nrm _2bel.tii _2bel.tvd _2bel.tvx >> > _2bem.fdt _2bem.fnm _2bem.nrm _2bem.tii _2bem.tvd _2bem.tvx >> > _2ben.frq _2ben.prx _2ben.tis _2beo.fdx segments.gen >> > _2bel.fdx _2bel.frq _2bel.prx _2bel.tis _2bel.tvf _2bel_1.del >> > _2bem.fdx _2bem.frq _2bem.prx _2bem.tis _2bem.tvf _2ben.fnm >> > _2ben.nrm _2ben.tii _2beo.fdt _2beo.fnm segments_230x >> > r...@search-01:/data/solr# ls video/data/index/ >> > _2bel.fdt _2bel.frq _2bel.tii _2bel.tvf_2bem.fdt _2bem.frq >> > _2bem.tii _2bem.tvf _2ben.frq _2ben.tii _2beo.fdx _2beo.nrm >> > _2beo.tis _2beo.tvx >> > _2bel.fdx _2bel.nrm _2bel.tis _2bel.tvx_2bem.fdx _2bem.nrm >> > _2bem.tis _2bem.tvx _2ben.nrm _2ben.tis _2beo.fnm _2beo.prx >> > _2beo.tvd segments.gen >> > _2bel.fnm _2bel.prx _2bel.tvd _2bel_1.del _2bem.fnm _2bem.prx >> > _2bem.tvd _2ben.fnm _2ben.prx _2beo.fdt _2beo.frq _2beo.tii >> > _2beo.tvf segments_230x >> > r...@search-01:/data/solr# ls video/data/index/ >> > _2beo.fdt _2beo.fdx _2beo.fnm _2beo.frq _2beo.nrm _2beo.prx >> > _2beo.tii _2beo.tis _2beo.tvd _2beo.tvf _2beo.tvx segments.gen >> > segments_230y >> > >> > So as you can notice my segments increased which is perfect for my >> > replication (faster to get JUST last segments) >> > But like you can see in my last ls my segment has been optimized. >> > >> > Like I can notice in my log : >> > Mar 19 15:42:37 search-01 jsvc.exec[23255]: Mar 19, 2009 3:42:37 PM >> > org.apache.solr.handler.dataimport.DocBuilder commit INFO: Full Import >> > completed successfully without optimization >> > Mar 19 15:42:37 search-01 jsvc.exec[23255]: Mar 19, 2009 3:42:37 PM >> > org.apache.solr.update.DirectUpdateHandler2 commit INFO: start >> > commit(optimize=true,waitFlush=false,waitSearcher=true) >> > >> > But I didn't fire any optimize, my delta-import is fired like : >> > /solr/video/dataimport?command=delta-import&optimize=false >> > >> > Solrconfig.xml : autocommit turnd off >> > >> > >> > >> > >> > Maybe it comes from lucene parameters? >> > >> > false >> > 50 >> > 50 >> > >> > >> > 2147483647 >> > 1 >> > >> > Thanks a lot for your help, >> > Sunny >> > >> > >> > >> > >> >> -- >> View this message in context: >> http://www.nabble.com/Problem-for-replication-%3A-segment-optimized-automaticly-tp22601442p22649412.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > > -- > Regards, > Shalin Shekhar Mangar. > > -- View this message in context: http://www.nabble.com/Problem-for-replication-%3A-segment-optimized-automaticly-tp22601442p22661653.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Problem for replication : segment optimized automaticly
Obviously mine yes ... :s Shalin Shekhar Mangar wrote: > > Lucene will automatically merge segments when they exceed the mergeFactor. > This may be one reason but I'm not sure. > > I checked DataImportHandler's code again. It won't optimize if > optimize=false is specified. > > On Mon, Mar 23, 2009 at 12:43 AM, sunnyfr wrote: > >> >> Do you have any idea ??? >> :( >> >> cheer, >> >> >> sunnyfr wrote: >> > >> > Hi everybody ... still me :) >> > hoo happy day :) >> > >> > Just, I dont get where I miss something, I will try to be clear. >> > >> > this is my index folder (and we can notice the evolution according to >> the >> > delta import every 30mn) : >> > >> > r...@search-01:/data/solr# ls video/data/index/ >> > _2bel.fdt _2bel.fnm _2bel.nrm _2bel.tii _2bel.tvd _2bel.tvx >> > _2bem.fdt _2bem.fnm _2bem.nrm _2bem.tii _2bem.tvd _2bem.tvx >> > _2ben.frq _2ben.prx _2ben.tis _2beo.fdx segments.gen >> > _2bel.fdx _2bel.frq _2bel.prx _2bel.tis _2bel.tvf _2bel_1.del >> > _2bem.fdx _2bem.frq _2bem.prx _2bem.tis _2bem.tvf _2ben.fnm >> > _2ben.nrm _2ben.tii _2beo.fdt _2beo.fnm segments_230x >> > r...@search-01:/data/solr# ls video/data/index/ >> > _2bel.fdt _2bel.frq _2bel.tii _2bel.tvf_2bem.fdt _2bem.frq >> > _2bem.tii _2bem.tvf _2ben.frq _2ben.tii _2beo.fdx _2beo.nrm >> > _2beo.tis _2beo.tvx >> > _2bel.fdx _2bel.nrm _2bel.tis _2bel.tvx_2bem.fdx _2bem.nrm >> > _2bem.tis _2bem.tvx _2ben.nrm _2ben.tis _2beo.fnm _2beo.prx >> > _2beo.tvd segments.gen >> > _2bel.fnm _2bel.prx _2bel.tvd _2bel_1.del _2bem.fnm _2bem.prx >> > _2bem.tvd _2ben.fnm _2ben.prx _2beo.fdt _2beo.frq _2beo.tii >> > _2beo.tvf segments_230x >> > r...@search-01:/data/solr# ls video/data/index/ >> > _2beo.fdt _2beo.fdx _2beo.fnm _2beo.frq _2beo.nrm _2beo.prx >> > _2beo.tii _2beo.tis _2beo.tvd _2beo.tvf _2beo.tvx segments.gen >> > segments_230y >> > >> > So as you can notice my segments increased which is perfect for my >> > replication (faster to get JUST last segments) >> > But like you can see in my last ls my segment has been optimized. >> > >> > Like I can notice in my log : >> > Mar 19 15:42:37 search-01 jsvc.exec[23255]: Mar 19, 2009 3:42:37 PM >> > org.apache.solr.handler.dataimport.DocBuilder commit INFO: Full Import >> > completed successfully without optimization >> > Mar 19 15:42:37 search-01 jsvc.exec[23255]: Mar 19, 2009 3:42:37 PM >> > org.apache.solr.update.DirectUpdateHandler2 commit INFO: start >> > commit(optimize=true,waitFlush=false,waitSearcher=true) >> > >> > But I didn't fire any optimize, my delta-import is fired like : >> > /solr/video/dataimport?command=delta-import&optimize=false >> > >> > Solrconfig.xml : autocommit turnd off >> > >> > >> > >> > >> > Maybe it comes from lucene parameters? >> > >> > false >> > 50 >> > 50 >> > >> > >> > 2147483647 >> > 1 >> > >> > Thanks a lot for your help, >> > Sunny >> > >> > >> > >> > >> >> -- >> View this message in context: >> http://www.nabble.com/Problem-for-replication-%3A-segment-optimized-automaticly-tp22601442p22649412.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > > -- > Regards, > Shalin Shekhar Mangar. > > -- View this message in context: http://www.nabble.com/Problem-for-replication-%3A-segment-optimized-automaticly-tp22601442p22661570.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Problem for replication : segment optimized automaticly
I checked this out but It doesn't say nothing about optimizing. I'm sure it's lucene part about merging or I don't know ...?? Noble Paul നോബിള് नोब्ळ् wrote: > > the easiest way to find out what DIH did is to hit it's status > command. It will give you a brief description of what all it did > during last import > > On Mon, Mar 23, 2009 at 12:59 AM, Shalin Shekhar Mangar > wrote: >> Lucene will automatically merge segments when they exceed the >> mergeFactor. >> This may be one reason but I'm not sure. >> >> I checked DataImportHandler's code again. It won't optimize if >> optimize=false is specified. >> >> On Mon, Mar 23, 2009 at 12:43 AM, sunnyfr wrote: >> >>> >>> Do you have any idea ??? >>> :( >>> >>> cheer, >>> >>> >>> sunnyfr wrote: >>> > >>> > Hi everybody ... still me :) >>> > hoo happy day :) >>> > >>> > Just, I dont get where I miss something, I will try to be clear. >>> > >>> > this is my index folder (and we can notice the evolution according to >>> the >>> > delta import every 30mn) : >>> > >>> > r...@search-01:/data/solr# ls video/data/index/ >>> > _2bel.fdt _2bel.fnm _2bel.nrm _2bel.tii _2bel.tvd _2bel.tvx >>> > _2bem.fdt _2bem.fnm _2bem.nrm _2bem.tii _2bem.tvd _2bem.tvx >>> > _2ben.frq _2ben.prx _2ben.tis _2beo.fdx segments.gen >>> > _2bel.fdx _2bel.frq _2bel.prx _2bel.tis _2bel.tvf _2bel_1.del >>> > _2bem.fdx _2bem.frq _2bem.prx _2bem.tis _2bem.tvf _2ben.fnm >>> > _2ben.nrm _2ben.tii _2beo.fdt _2beo.fnm segments_230x >>> > r...@search-01:/data/solr# ls video/data/index/ >>> > _2bel.fdt _2bel.frq _2bel.tii _2bel.tvf _2bem.fdt _2bem.frq >>> > _2bem.tii _2bem.tvf _2ben.frq _2ben.tii _2beo.fdx _2beo.nrm >>> > _2beo.tis _2beo.tvx >>> > _2bel.fdx _2bel.nrm _2bel.tis _2bel.tvx _2bem.fdx _2bem.nrm >>> > _2bem.tis _2bem.tvx _2ben.nrm _2ben.tis _2beo.fnm _2beo.prx >>> > _2beo.tvd segments.gen >>> > _2bel.fnm _2bel.prx _2bel.tvd _2bel_1.del _2bem.fnm _2bem.prx >>> > _2bem.tvd _2ben.fnm _2ben.prx _2beo.fdt _2beo.frq _2beo.tii >>> > _2beo.tvf segments_230x >>> > r...@search-01:/data/solr# ls video/data/index/ >>> > _2beo.fdt _2beo.fdx _2beo.fnm _2beo.frq _2beo.nrm _2beo.prx >>> > _2beo.tii _2beo.tis _2beo.tvd _2beo.tvf _2beo.tvx segments.gen >>> > segments_230y >>> > >>> > So as you can notice my segments increased which is perfect for my >>> > replication (faster to get JUST last segments) >>> > But like you can see in my last ls my segment has been optimized. >>> > >>> > Like I can notice in my log : >>> > Mar 19 15:42:37 search-01 jsvc.exec[23255]: Mar 19, 2009 3:42:37 PM >>> > org.apache.solr.handler.dataimport.DocBuilder commit INFO: Full Import >>> > completed successfully without optimization >>> > Mar 19 15:42:37 search-01 jsvc.exec[23255]: Mar 19, 2009 3:42:37 PM >>> > org.apache.solr.update.DirectUpdateHandler2 commit INFO: start >>> > commit(optimize=true,waitFlush=false,waitSearcher=true) >>> > >>> > But I didn't fire any optimize, my delta-import is fired like : >>> > /solr/video/dataimport?command=delta-import&optimize=false >>> > >>> > Solrconfig.xml : autocommit turnd off >>> > >>> > >>> > >>> > >>> > Maybe it comes from lucene parameters? >>> > >>> > false >>> > 50 >>> > 50 >>> > >>> > >>> > 2147483647 >>> > 1 >>> > >>> > Thanks a lot for your help, >>> > Sunny >>> > >>> > >>> > >>> > >>> >>> -- >>> View this message in context: >>> http://www.nabble.com/Problem-for-replication-%3A-segment-optimized-automaticly-tp22601442p22649412.html >>> Sent from the Solr - User mailing list archive at Nabble.com. >>> >>> >> >> >> -- >> Regards, >> Shalin Shekhar Mangar. >> > > > > -- > --Noble Paul > > -- View this message in context: http://www.nabble.com/Problem-for-replication-%3A-segment-optimized-automaticly-tp22601442p22661545.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Problem for replication : segment optimized automaticly
Do you have any idea ??? :( cheer, sunnyfr wrote: > > Hi everybody ... still me :) > hoo happy day :) > > Just, I dont get where I miss something, I will try to be clear. > > this is my index folder (and we can notice the evolution according to the > delta import every 30mn) : > > r...@search-01:/data/solr# ls video/data/index/ > _2bel.fdt _2bel.fnm _2bel.nrm _2bel.tii _2bel.tvd _2bel.tvx > _2bem.fdt _2bem.fnm _2bem.nrm _2bem.tii _2bem.tvd _2bem.tvx > _2ben.frq _2ben.prx _2ben.tis _2beo.fdx segments.gen > _2bel.fdx _2bel.frq _2bel.prx _2bel.tis _2bel.tvf _2bel_1.del > _2bem.fdx _2bem.frq _2bem.prx _2bem.tis _2bem.tvf _2ben.fnm > _2ben.nrm _2ben.tii _2beo.fdt _2beo.fnm segments_230x > r...@search-01:/data/solr# ls video/data/index/ > _2bel.fdt _2bel.frq _2bel.tii _2bel.tvf_2bem.fdt _2bem.frq > _2bem.tii _2bem.tvf _2ben.frq _2ben.tii _2beo.fdx _2beo.nrm > _2beo.tis _2beo.tvx > _2bel.fdx _2bel.nrm _2bel.tis _2bel.tvx_2bem.fdx _2bem.nrm > _2bem.tis _2bem.tvx _2ben.nrm _2ben.tis _2beo.fnm _2beo.prx > _2beo.tvd segments.gen > _2bel.fnm _2bel.prx _2bel.tvd _2bel_1.del _2bem.fnm _2bem.prx > _2bem.tvd _2ben.fnm _2ben.prx _2beo.fdt _2beo.frq _2beo.tii > _2beo.tvf segments_230x > r...@search-01:/data/solr# ls video/data/index/ > _2beo.fdt _2beo.fdx _2beo.fnm _2beo.frq _2beo.nrm _2beo.prx > _2beo.tii _2beo.tis _2beo.tvd _2beo.tvf _2beo.tvx segments.gen > segments_230y > > So as you can notice my segments increased which is perfect for my > replication (faster to get JUST last segments) > But like you can see in my last ls my segment has been optimized. > > Like I can notice in my log : > Mar 19 15:42:37 search-01 jsvc.exec[23255]: Mar 19, 2009 3:42:37 PM > org.apache.solr.handler.dataimport.DocBuilder commit INFO: Full Import > completed successfully without optimization > Mar 19 15:42:37 search-01 jsvc.exec[23255]: Mar 19, 2009 3:42:37 PM > org.apache.solr.update.DirectUpdateHandler2 commit INFO: start > commit(optimize=true,waitFlush=false,waitSearcher=true) > > But I didn't fire any optimize, my delta-import is fired like : > /solr/video/dataimport?command=delta-import&optimize=false > > Solrconfig.xml : autocommit turnd off > > > > > Maybe it comes from lucene parameters? > > false > 50 > 50 > > > 2147483647 > 1 > > Thanks a lot for your help, > Sunny > > > > -- View this message in context: http://www.nabble.com/Problem-for-replication-%3A-segment-optimized-automaticly-tp22601442p22649412.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: delta-import commit=false doesn't seems to work
Like you can see, I did that and I've no information in my DIH but you can notice in my logs and even my segments that and optimize is fired alone automaticly? Noble Paul നോബിള് नोब्ळ् wrote: > > just hit the DIH without any command and you may be able to see the > status of the last import. It can tell you whether a commit/optimize > was performed > > On Fri, Mar 20, 2009 at 7:07 PM, sunnyfr wrote: >> >> Thanks I gave more information there : >> http://www.nabble.com/Problem-for-replication-%3A-segment-optimized-automaticly-td22601442.html >> >> thanks a lot Paul >> >> >> Noble Paul നോബിള് नोब्ळ् wrote: >>> >>> sorry, the whole thing was commented . I did not notice that. I'll >>> look into that >>> >>> 2009/3/20 Noble Paul നോബിള് नोब्ळ् : >>>> you have set autoCommit every x minutes . it must have invoked commit >>>> automatically >>>> >>>> >>>> On Thu, Mar 19, 2009 at 4:17 PM, sunnyfr wrote: >>>>> >>>>> Hi, >>>>> >>>>> Even if I hit command=delta-import&commit=false&optimize=false >>>>> I still have commit set in my logs and sometimes even optimize=true, >>>>> >>>>> About optimize I wonder if it comes from commitment too close and one >>>>> is >>>>> not >>>>> done, but still I don't know really. >>>>> >>>>> Any idea? >>>>> >>>>> Thanks a lot, >>>>> -- >>>>> View this message in context: >>>>> http://www.nabble.com/delta-import-commit%3Dfalse-doesn%27t-seems-to-work-tp22597630p22597630.html >>>>> Sent from the Solr - User mailing list archive at Nabble.com. >>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> --Noble Paul >>>> >>> >>> >>> >>> -- >>> --Noble Paul >>> >>> >> >> -- >> View this message in context: >> http://www.nabble.com/Re%3A-delta-import-commit%3Dfalse-doesn%27t-seems-to-work-tp22614216p22620439.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > > > -- > --Noble Paul > > -- View this message in context: http://www.nabble.com/Re%3A-delta-import-commit%3Dfalse-doesn%27t-seems-to-work-tp22614216p22625149.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: how can I check field which are indexed but not stored?
Cool I was just having a look on it but it doesn't seem to show up field which are not stored just tried : /admin/luke?id=8582006&fl=description but it doesn't seems to work :( It find this id but show up stored field. Did I do a mistake ? thanks a lot Markus Jelsma - Buyways B.V. wrote: > > > > On Fri, 2009-03-20 at 03:41 -0700, sunnyfr wrote: > >> Hi >> >> I've an issue, I've some data which come up but I've applied a filtre on >> it >> and it shouldnt, when I check in my database mysql I've obviously the >> document which has been updated so I will like to see how it is in solr. >> >> if I do : /solr/video/select?q=id:8582006 I will just see field which has >> been stored. Is there a way to see how data are indexed for other field >> of >> my schema which are not stored but indexed. > > > /solr/admin/luke > will show you a lot of information concering stored and indexed fields. > > Hope this is what you meant. > > >> >> Like a bit in the console dataimporthandler, which with verbose activated >> I >> can see every field of my schema. >> >> Otherwise what would you reckon in this case, a document which has not >> been >> updated ? how can I sort it out? >> >> Thanks a lot guys for your excellent help > > -- View this message in context: http://www.nabble.com/how-can-I-check-field-which-are-indexed-but-not-stored--tp22617914p22621773.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: delta-import commit=false doesn't seems to work
Thanks I gave more information there : http://www.nabble.com/Problem-for-replication-%3A-segment-optimized-automaticly-td22601442.html thanks a lot Paul Noble Paul നോബിള് नोब्ळ् wrote: > > sorry, the whole thing was commented . I did not notice that. I'll > look into that > > 2009/3/20 Noble Paul നോബിള് नोब्ळ् : >> you have set autoCommit every x minutes . it must have invoked commit >> automatically >> >> >> On Thu, Mar 19, 2009 at 4:17 PM, sunnyfr wrote: >>> >>> Hi, >>> >>> Even if I hit command=delta-import&commit=false&optimize=false >>> I still have commit set in my logs and sometimes even optimize=true, >>> >>> About optimize I wonder if it comes from commitment too close and one is >>> not >>> done, but still I don't know really. >>> >>> Any idea? >>> >>> Thanks a lot, >>> -- >>> View this message in context: >>> http://www.nabble.com/delta-import-commit%3Dfalse-doesn%27t-seems-to-work-tp22597630p22597630.html >>> Sent from the Solr - User mailing list archive at Nabble.com. >>> >>> >> >> >> >> -- >> --Noble Paul >> > > > > -- > --Noble Paul > > -- View this message in context: http://www.nabble.com/Re%3A-delta-import-commit%3Dfalse-doesn%27t-seems-to-work-tp22614216p22620439.html Sent from the Solr - User mailing list archive at Nabble.com.
how can I check field which are indexed but not stored?
Hi I've an issue, I've some data which come up but I've applied a filtre on it and it shouldnt, when I check in my database mysql I've obviously the document which has been updated so I will like to see how it is in solr. if I do : /solr/video/select?q=id:8582006 I will just see field which has been stored. Is there a way to see how data are indexed for other field of my schema which are not stored but indexed. Like a bit in the console dataimporthandler, which with verbose activated I can see every field of my schema. Otherwise what would you reckon in this case, a document which has not been updated ? how can I sort it out? Thanks a lot guys for your excellent help -- View this message in context: http://www.nabble.com/how-can-I-check-field-which-are-indexed-but-not-stored--tp22617914p22617914.html Sent from the Solr - User mailing list archive at Nabble.com.
Problem for replication : segment optimized automaticly
Hi everybody ... still me :) hoo happy day :) Just, I dont get where I miss something, I will try to be clear. this is my index folder (and we can notice the evolution according to the delta import every 30mn) : r...@search-01:/data/solr# ls video/data/index/ _2bel.fdt _2bel.fnm _2bel.nrm _2bel.tii _2bel.tvd _2bel.tvx _2bem.fdt _2bem.fnm _2bem.nrm _2bem.tii _2bem.tvd _2bem.tvx _2ben.frq _2ben.prx _2ben.tis _2beo.fdx segments.gen _2bel.fdx _2bel.frq _2bel.prx _2bel.tis _2bel.tvf _2bel_1.del _2bem.fdx _2bem.frq _2bem.prx _2bem.tis _2bem.tvf _2ben.fnm _2ben.nrm _2ben.tii _2beo.fdt _2beo.fnm segments_230x r...@search-01:/data/solr# ls video/data/index/ _2bel.fdt _2bel.frq _2bel.tii _2bel.tvf_2bem.fdt _2bem.frq _2bem.tii _2bem.tvf _2ben.frq _2ben.tii _2beo.fdx _2beo.nrm _2beo.tis _2beo.tvx _2bel.fdx _2bel.nrm _2bel.tis _2bel.tvx_2bem.fdx _2bem.nrm _2bem.tis _2bem.tvx _2ben.nrm _2ben.tis _2beo.fnm _2beo.prx _2beo.tvd segments.gen _2bel.fnm _2bel.prx _2bel.tvd _2bel_1.del _2bem.fnm _2bem.prx _2bem.tvd _2ben.fnm _2ben.prx _2beo.fdt _2beo.frq _2beo.tii _2beo.tvf segments_230x r...@search-01:/data/solr# ls video/data/index/ _2beo.fdt _2beo.fdx _2beo.fnm _2beo.frq _2beo.nrm _2beo.prx _2beo.tii _2beo.tis _2beo.tvd _2beo.tvf _2beo.tvx segments.gen segments_230y So as you can notice my segments increased which is perfect for my replication (faster to get JUST last segments) But like you can see in my last ls my segment has been optimized. Like I can notice in my log : Mar 19 15:42:37 search-01 jsvc.exec[23255]: Mar 19, 2009 3:42:37 PM org.apache.solr.handler.dataimport.DocBuilder commit INFO: Full Import completed successfully without optimization Mar 19 15:42:37 search-01 jsvc.exec[23255]: Mar 19, 2009 3:42:37 PM org.apache.solr.update.DirectUpdateHandler2 commit INFO: start commit(optimize=true,waitFlush=false,waitSearcher=true) But I didn't fire any optimize, my delta-import is fired like : /solr/video/dataimport?command=delta-import&optimize=false Solrconfig.xml : autocommit turnd off Maybe it comes from lucene parameters? false 50 50 2147483647 1 Thanks a lot for your help, Sunny -- View this message in context: http://www.nabble.com/Problem-for-replication-%3A-segment-optimized-automaticly-tp22601442p22601442.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: delta-import commit=false doesn't seems to work
So I've several searcher opened for just on delta import hit and the param
commit=false
no cronjob set
-->
32
2147483647
1
1000
false
32
25
2147483647
1
Huge thanks for your help
Mar 19 11:38:32 search-01 jsvc.exec[17650]: Mar 19, 2009 11:38:32 AM
org.apache.solr.search.SolrIndexSearcher INFO: Opening
searc...@695f main
Mar 19 11:38:32 search-01 jsvc.exec[17650]: Mar 19, 2009 11:38:32 AM
org.apache.solr.update.DirectUpdateHandler2 commit INFO: end_commit_flush
Mar 19 11:38:32 search-01 jsvc.exec[17650]: Mar 19, 2009 11:38:32 AM
org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming
searc...@695f main from searc...@7118e99d main
^IfilterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Mar 19 11:38:32 search-01 jsvc.exec[17650]: Mar 19, 2009 11:38:32 AM
org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming result for
searc...@695f main
^IfilterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Mar 19 11:38:32 search-01 jsvc.exec[17650]: Mar 19, 2009 11:38:32 AM
org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming
searc...@695f main from searc...@7118e99d main
^IqueryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=2,size=1,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=238}
Mar 19 11:38:32 search-01 jsvc.exec[17650]: Mar 19, 2009 11:38:32 AM
org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming result for
searc...@695f main
^IqueryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=238}
Mar 19 11:38:32 search-01 jsvc.exec[17650]: Mar 19, 2009 11:38:32 AM
org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming
searc...@695f main from searc...@7118e99d main
^IdocumentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=38,size=2,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=4522}
Mar 19 11:38:32 search-01 jsvc.exec[17650]: Mar 19, 2009 11:38:32 AM
org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming result for
searc...@695f main
^IdocumentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=4522}
Mar 19 11:38:32 search-01 jsvc.exec[17650]: Mar 19, 2009 11:38:32 AM
org.apache.solr.core.QuerySenderListener newSearcher INFO:
QuerySenderListener sending requests to searc...@695f main
Mar 19 11:38:32 search-01 jsvc.exec[17650]: Mar 19, 2009 11:38:32 AM
org.apache.solr.core.SolrCore execute INFO: [video] webapp=null path=null
params={start=0&q=solr&rows=10} hits=163 status=0 QTime=2
Mar 19 11:38:32 search-01 jsvc.exec[17650]: Mar 19, 2009 11:38:32 AM
org.apache.solr.core.SolrCore execute INFO: [video] webapp=null path=null
params={start=0&q=rocks&rows=10} hits=160584 status=0 QTime=10
Mar 19 11:38:32 search-01 jsvc.exec[17650]: Mar 19, 2009 11:38:32 AM
org.apache.solr.core.SolrCore execute INFO: [video] webapp=null path=null
params={q=static+newSearcher+warming+query+from+solrconfig.xml} hits=0
status=0 QTime=2
Mar 19 11:38:32 search-01 jsvc.exec[17650]: Mar 19, 2009 11:38:32 AM
org.apache.solr.core.QuerySenderListener newSearcher INFO:
QuerySenderListener done.
Mar 19 11:38:32 search-01 jsvc.exec[17650]: Mar 19, 2009 11:38:32 AM
org.apache.solr.core.SolrCore registerSearcher INFO: [video] Registered new
searcher searc...@695f main
Mar 19 11:38:32 search-01 jsvc.exec[17650]: Mar 19, 2009 11:38:32 AM
org.apache.solr.search.SolrIndexSearcher close INFO: Closing
searc...@7118e99d main
^IfilterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
^IqueryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=2,size=1,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=240}
^IdocumentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=38,size=2,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=4560}
sunnyfr wrote:
>
> Hi,
>
> Even if I hit command=delta-import&commit=false&optimize=false
> I still have commit set in my logs and sometimes even optimize=true,
>
> About optimize I wonder if it comes from commitme
delta-import commit=false doesn't seems to work
Hi, Even if I hit command=delta-import&commit=false&optimize=false I still have commit set in my logs and sometimes even optimize=true, About optimize I wonder if it comes from commitment too close and one is not done, but still I don't know really. Any idea? Thanks a lot, -- View this message in context: http://www.nabble.com/delta-import-commit%3Dfalse-doesn%27t-seems-to-work-tp22597630p22597630.html Sent from the Solr - User mailing list archive at Nabble.com.
which parameter fire optimize
Hi I've in my log optimize=true after a commit but I didnt allow it in my solrconfig ??? /data/solr/video/bin/snapshooter /data/solr/video/bin -c true Do you have an idea where it comes from?? Thanks a lot, -- View this message in context: http://www.nabble.com/which-parameter-fire-optimize-tp22590500p22590500.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: optimize after a commit don't know why?
Maybe I miss something in solrconfig.xml ??? sunnyfr wrote: > > Hi > > I've a little problem with optimization which is very interesting but > juste one time per day otherwise replication take ages to bring back index > hard link. > > So my cron is every 30mn : > /solr/user/dataimport?command=delta-import&optimize=false > otherwise i've cron for optimizing every six hours : > 0 */6 * * * /data/solr/user/bin/optimize > > But I don't get why in my logs I've : > Mar 18 05:35:34 search-01 jsvc.exec[12612]: Mar 18, 2009 5:35:34 AM > org.apache.solr.handler.dataimport.SolrWriter persistStartTime INFO: Wrote > last indexed time to dataimport.properties > Mar 18 05:35:34 search-01 jsvc.exec[12612]: Mar 18, 2009 5:35:34 AM > org.apache.solr.handler.dataimport.DocBuilder commit INFO: Full Import > completed successfully > Mar 18 05:35:34 search-01 jsvc.exec[12612]: Mar 18, 2009 5:35:34 AM > org.apache.solr.update.DirectUpdateHandler2 commit INFO: start > commit(optimize=true,waitFlush=false,waitSearcher=true) > > > I've a commit which is not automatic in solrconfig but by cronjob every > 30mn > */30 * * * * /data/solr/user/bin/commit > > Any idea would be excellent, thanks a lot, > > > > > -- View this message in context: http://www.nabble.com/optimize-after-a-commit-don%27t-know-why--tp22576374p22588737.html Sent from the Solr - User mailing list archive at Nabble.com.
optimize after a commit don't know why?
Hi I've a little problem with optimization which is very interesting but juste one time per day otherwise replication take ages to bring back index hard link. So my cron is every 30mn : /solr/user/dataimport?command=delta-import&optimize=false&commit=false otherwise i've cron for optimizing every six hours : 0 */6 * * * /data/solr/user/bin/optimize But I don't get why in my logs I've : Mar 18 05:35:34 search-01 jsvc.exec[12612]: Mar 18, 2009 5:35:34 AM org.apache.solr.handler.dataimport.SolrWriter persistStartTime INFO: Wrote last indexed time to dataimport.properties Mar 18 05:35:34 search-01 jsvc.exec[12612]: Mar 18, 2009 5:35:34 AM org.apache.solr.handler.dataimport.DocBuilder commit INFO: Full Import completed successfully Mar 18 05:35:34 search-01 jsvc.exec[12612]: Mar 18, 2009 5:35:34 AM org.apache.solr.update.DirectUpdateHandler2 commit INFO: start commit(optimize=true,waitFlush=false,waitSearcher=true) Mar 18 05:35:36 search-01 jsvc.exec[12612]: Mar 18, 2009 5:35:35 AM org.apache.solr.update.DirectUpdateHandler2 commit INFO: start commit(optimize=false,waitFlush=false,waitSearcher=true) I've a commit which is not automatic in solrconfig but by cronjob every 30mn */30 * * * * /data/solr/user/bin/commit Any idea would be excellent, thanks a lot, -- View this message in context: http://www.nabble.com/optimize-after-a-commit-don%27t-know-why--tp22576374p22576374.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Is optimize always a commit?
Hi If I want to commit without optimize. Because Ive that : > start commit(optimize=true,waitFlush=false,waitSearcher=true) but I don't want to optimize otherwise my replication will take every time the full index folder. Thanks a lot guys for ur help, ryantxu wrote: > > yes. optimize also commits > > Maximilian Hütter wrote: >> Hi, >> >> maybe this is a stupid question, but is a optimize always a commit? >> In the log it looks like it: >> >> start commit(optimize=true,waitFlush=false,waitSearcher=true) >> >> I just wanted to be sure. >> >> Best regards, >> >> Max >> >> > > > -- View this message in context: http://www.nabble.com/Is-optimize-always-a-commit--tp15498266p22562206.html Sent from the Solr - User mailing list archive at Nabble.com.
rsync snappuller slowdown Qtime
Hi, Noticing a relevant latency during search, I tried to turn off cronjob and test it manually. And it was obvious how during snappuller on a slave server, the query time was a lot longer than the rest of the time. Even snapinstaller didn't affect the query time. without any action around 200msec with snappuller 3-6sec .. Do you have any idea? Thanks a lot, -- View this message in context: http://www.nabble.com/rsync-snappuller-slowdown-Qtime-tp22497625p22497625.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Re[2]: the time factor
Hi Hoss, How come if bq doesn't influence what matches -- that's q -- bq only influence the scores of existing matches if they also match the bq when I put : as bq=(country:FR)^2 (status_official:1 status_new:1)^2.5 Ive no result if I put just bq=(country:FR)^2 Or bq=(status_official:1 status_new:1)^2.5 or even bq=(country:FR)^2 OR (status_official:1 status_new:1)^2.5 I will have one result. I don't want OR but AND, if the book as this status AND are FR then boost it more but not OR. Because the book which come up is a book which is not french but still its should come up. Thanks a lot hossman wrote: > > > : I'm not quite understanding how boost query works though. How does it > : "influence" the score exactly? Does it just simply append to the "q" > : param? From the wiki: > > Esentially yes, but documents must match the at least one clause of > the "q", matching the "bq" is optional (and when it happens, will result > in a score increase accordingly) > > : If this is how it works, it sounds like the bq will be used first > : to get a result set, then the result set will be sorted by q > : (relevance)? > > no. bq doesn't influence what matches -- that's q -- bq only influence > the scores of existing matches if they also match the bq. > > > > -Hoss > > > -- View this message in context: http://www.nabble.com/the-time-factor-tp17204792p22413452.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: order of word in the request
Thanks Yonik, Yonik Seeley-2 wrote: > > On Thu, Feb 26, 2009 at 11:25 AM, sunnyfr wrote: >> How can I tell it to put a lot of more weight for the book which has >> exactly >> the same title. > > A sloppy phrase query should work. > See the "pf" param in the dismax query parser. > > -Yonik > http://www.lucidimagination.com > > -- View this message in context: http://www.nabble.com/order-of-word-in-the-request-tp7783p22241361.html Sent from the Solr - User mailing list archive at Nabble.com.
order of word in the request
Hi guys, I look for the parameter or the way to boost the order of the word in the query. Let's imagine people look for "rich & famous" book ... so in the search they will just write rich & famous and let's imagine a book with a better rating and lot of views is like famous & very rich is there, it will come up first. How can I tell it to put a lot of more weight for the book which has exactly the same title. Thanks a lot, -- View this message in context: http://www.nabble.com/order-of-word-in-the-request-tp7783p7783.html Sent from the Solr - User mailing list archive at Nabble.com.
dismax + and -
Hi How come if i put in my query q=+wow-kill wow-kill dismax I will have books which contain wow and kill instead of books which have wow in the title without kill??? Thanks a lot, -- View this message in context: http://www.nabble.com/dismax-%2B-and---tp4770p4770.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: boost several boolean in bq
I've actually added (status_official:1 OR status_creative:1)^2.5 sunnyfr wrote: > > Hi > > I dont get where I'm wrong. > I would like to boost some type of my books. > > So If I do : &bq=status_official:0^1.5+status_creative:0^1.5 > I've one result > > If I do: &bq=status_official:1^1.5+status_creative:1^1.5 > Nothing, I think the result should still come up even if it doesn't have > this status > > If I do : &bq=status_official:1^1 or just &bq=status_creative:1^1.5 > just like that > I've one result which is wierd according to the previous test..?? > > Any idea > Thanks a lot for your help, > Sunny > -- View this message in context: http://www.nabble.com/boost-several-boolean-in-bq-tp0325p0469.html Sent from the Solr - User mailing list archive at Nabble.com.
boost several boolean in bq
Hi I dont get where I'm wrong. I would like to boost some type of my books. So If I do : &bq=status_official:0^1.5+status_creative:0^1.5 I've one result If I do: &bq=status_official:1^1.5+status_creative:1^1.5 Nothing, I think the result should still come up even if it doesn't have this status If I do : &bq=status_official:1^1 or just &bq=status_creative:1^1.5 just like that I've one result which is wierd according to the previous test..?? Any idea Thanks a lot for your help, Sunny -- View this message in context: http://www.nabble.com/boost-several-boolean-in-bq-tp0325p0325.html Sent from the Solr - User mailing list archive at Nabble.com.
boost type:true
Hello everybody, Little question : status_official:true^1,5 How come this doesn't show up datas and if I remove status_official then it will show up data. I tried to add status_official:false^1 but nothing come up and if I remove this param I've some value. I would like to boost some status ... Help would be very appreciated, Thanks Sunny -- View this message in context: http://www.nabble.com/boost-type%3Atrue-tp22219684p22219684.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: show up every parameter in my dismax query
yes thanks a lot Koji, Koji Sekiguchi-2 wrote: > > sunnyfr wrote: >> Hi >> >> Sorry I dont remember what is the parameter which show up every >> parameters >> stores in my solrconfig.xml file for the dismax query ? thanks a >> lot, >> >> >> > echoParams=all ? > > regards, > > Koji > > > > -- View this message in context: http://www.nabble.com/show-up-every-parameter-in-my-dismax-query-tp22181063p22184676.html Sent from the Solr - User mailing list archive at Nabble.com.
show up every parameter in my dismax query
Hi Sorry I dont remember what is the parameter which show up every parameters stores in my solrconfig.xml file for the dismax query ? thanks a lot, -- View this message in context: http://www.nabble.com/show-up-every-parameter-in-my-dismax-query-tp22181063p22181063.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: delta-import or update for one special ID
0 PM
org.apache.solr.core.SolrDeletionPolicy updateCommits INFO: last commit =
1228743261094
Feb 19 15:24:10 search-07 jsvc.exec[3233]: Feb 19, 2009 3:24:10 PM
org.apache.solr.handler.dataimport.DataImportHandler$1 upload INFO: Indexing
stopped at docCount = 1
Feb 19 15:24:10 search-07 jsvc.exec[3233]: Feb 19, 2009 3:24:10 PM
org.apache.solr.handler.dataimport.SolrWriter readIndexerProperties INFO:
Read dataimport.properties
Feb 19 15:24:10 search-07 jsvc.exec[3233]: Feb 19, 2009 3:24:10 PM
org.apache.solr.handler.dataimport.SolrWriter persistStartTime INFO: Wrote
last indexed time to dataimport.properties
Feb 19 15:24:10 search-07 jsvc.exec[3233]: Feb 19, 2009 3:24:10 PM
org.apache.solr.handler.dataimport.DocBuilder commit INFO: Full Import
completed successfully
Feb 19 15:24:10 search-07 jsvc.exec[3233]: Feb 19, 2009 3:24:10 PM
org.apache.solr.update.DirectUpdateHandler2 commit INFO: start
commit(optimize=true,waitFlush=false,waitSearcher=true)
Feb 19 15:25:12 search-07 /USR/SBIN/CRON[4329]: (root) CMD ([ -x
/usr/lib/sysstat/sa1 ] && { [ -r "$DEFAULT" ] && . "$DEFAULT" ; [ "$ENABLED"
= "true" ] && exec /usr/lib/sysstat/sa1 $SA1_OPTIONS 1 1 ; })
Feb 19 15:25:13 search-07 jsvc.exec[3233]: Feb 19, 2009 3:25:13 PM
org.apache.solr.update.DirectUpdateHandler2 commit INFO: start
commit(optimize=false,waitFlush=false,waitSearcher=true)
Feb 19 15:25:13 search-07 jsvc.exec[3233]: Feb 19, 2009 3:25:13 PM
org.apache.solr.search.SolrIndexSearcher INFO: Opening
searc...@6d3136e5 main
Feb 19 15:25:13 search-07 jsvc.exec[3233]: Feb 19, 2009 3:25:13 PM
org.apache.solr.update.DirectUpdateHandler2 commit INFO: end_commit_flush
Feb 19, 2009 3:25:13 PM org.apache.solr.search.SolrIndexSearcher warm INFO:
autowarming searc...@6d3136e5 main from searc...@5fa13338 main
^IfilterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Feb 19 15:25:13 search-07 jsvc.exec[3233]: Feb 19, 2009 3:25:13 PM
org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming result for
searc...@6d3136e5 main
^IfilterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Feb 19 15:25:13 search-07 jsvc.exec[3233]: Feb 19, 2009 3:25:13 PM
org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming
searc...@6d3136e5 main from searc...@5fa13338 main
^IqueryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=3,evictions=0,size=3,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Feb 19 15:25:13 search-07 jsvc.exec[3233]: Feb 19, 2009 3:25:13 PM
org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming result for
searc...@6d3136e5 main
^IqueryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Feb 19 15:25:13 search-07 jsvc.exec[3233]: Feb 19, 2009 3:25:13 PM
org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming
searc...@6d3136e5 main from searc...@5fa13338 main
^IdocumentCache{lookups=0,hits=0,hitratio=0.00,inserts=29,evictions=0,size=29,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Feb 19 15:25:13 search-07 jsvc.exec[3233]: Feb 19, 2009 3:25:13 PM
org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming result for
searc...@6d3136e5 main
^IdocumentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Feb 19 15:25:13 search-07 jsvc.exec[3233]: Feb 19, 2009 3:25:13 PM
org.apache.solr.core.QuerySenderListener newSearcher INFO:
QuerySenderListener sending requests to searc...@6d3136e5 main
Feb 19 15:25:13 search-07 jsvc.exec[3233]: Feb 19, 2009 3:25:13 PM
org.apache.solr.core.SolrCore execute INFO: [user] webapp=null path=null
params={start=0&q=solr&rows=10} hits=9 status=0 QTime=1 Feb 19, 2009
3:25:13 PM org.apache.solr.core.SolrCore execute INFO: [user] webapp=null
path=null params={start=0&q=rocks&rows=10} hits=16983 status=0 QTime=2
But when I look for it, it doesn't appear :
:8180/solr/book/select?q=id:9327553
response>
−
0
12
−
id:9327553
Maybe the warmup, but I've no warmup maybe just 30 so almost nothing.
Noble Paul നോബിള് नोब्ळ् wrote:
>
> I was referring to the DIH debug page.
>
> But apparently, in some cases it seems to be working for you. can you
> elaborate , when does it work and when it doesn't?
>
> On Thu, Feb 19, 2009 at 5:38 PM, sunnyfr wrote:
>>
>> Hi,
>> Thanks Paul
>>
Re: delta-import or update for one special ID
It looks like books which have no link with entity are not took in consideration ??? part of my data-config.xml: Is it normal ? sunnyfr wrote: > > Hi, > Thanks Paul > > I did that : > book/dataimport?command=full-import&clean=false&start=9327553&rows=1&debugQuery=true > > but it stays stuck like that : > > 0:2:10.979 > 1 > 0 > 0 > 0 > 2009-02-19 13:06:23 > > > but with another Id it works and I've no error ...? and if I make the > request straight in MySql database i've the row which come up properly. > > what else can I do ... check ?? > > thanks a lot, > > > Noble Paul നോബിള് नोब्ळ् wrote: >> >> the start and rows is supposed to work . If you put it into debug you >> may see what is happening >> >> On Thu, Feb 19, 2009 at 3:47 PM, sunnyfr wrote: >>> >>> Hi >>> >>> I looked for a book that I couldn't find in solr's databe. >>> How can I update just this one by the command in the Url ... I tried : >>> But it doesn't seems to work ?? >>> >>> dataimport?command=full-import&clean=false&start=11289500&rows=100 >>> >>> Is there another way ??? maybe the book can't be updated but how can I >>> check >>> why ?? >>> >>> Thanks a lot >>> -- >>> View this message in context: >>> http://www.nabble.com/delta-import-or-update-for-one-special-ID-tp22097433p22097433.html >>> Sent from the Solr - User mailing list archive at Nabble.com. >>> >>> >> >> >> >> -- >> --Noble Paul >> >> > > -- View this message in context: http://www.nabble.com/delta-import-or-update-for-one-special-ID-tp22097433p22099470.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: delta-import or update for one special ID
Hi, Thanks Paul I did that : book/dataimport?command=full-import&clean=false&start=9327553&rows=1&debugQuery=true but it stays stuck like that : 0:2:10.979 1 0 0 0 2009-02-19 13:06:23 but with another Id it works and I've no error ...? and if I make the request straight in MySql database i've the row which come up properly. what else can I do ... check ?? thanks a lot, Noble Paul നോബിള് नोब्ळ् wrote: > > the start and rows is supposed to work . If you put it into debug you > may see what is happening > > On Thu, Feb 19, 2009 at 3:47 PM, sunnyfr wrote: >> >> Hi >> >> I looked for a book that I couldn't find in solr's databe. >> How can I update just this one by the command in the Url ... I tried : >> But it doesn't seems to work ?? >> >> dataimport?command=full-import&clean=false&start=11289500&rows=100 >> >> Is there another way ??? maybe the book can't be updated but how can I >> check >> why ?? >> >> Thanks a lot >> -- >> View this message in context: >> http://www.nabble.com/delta-import-or-update-for-one-special-ID-tp22097433p22097433.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > > > -- > --Noble Paul > > -- View this message in context: http://www.nabble.com/delta-import-or-update-for-one-special-ID-tp22097433p22099000.html Sent from the Solr - User mailing list archive at Nabble.com.
delta-import or update for one special ID
Hi I looked for a book that I couldn't find in solr's databe. How can I update just this one by the command in the Url ... I tried : But it doesn't seems to work ?? dataimport?command=full-import&clean=false&start=11289500&rows=100 Is there another way ??? maybe the book can't be updated but how can I check why ?? Thanks a lot -- View this message in context: http://www.nabble.com/delta-import-or-update-for-one-special-ID-tp22097433p22097433.html Sent from the Solr - User mailing list archive at Nabble.com.
bq type_:true for two types doesn't come up books.
Hi, I don't get: I added a bq boost, the point is i've some book which are normal, some which are type_roman or type_comedy and other type but I would like to boost both of this type for every books indexed. So if I do : &bq=type_roman:true^1,5+type_comedy:true^1,5 no video come up but if I do : &bq=type_roman:true^1,5+type_comedy:false^1,5 or just one type videos come up. I would like to boost if one or the other one is selected but it's clear that a book can't have both type. How can I manage this ? Thanks a lot, -- View this message in context: http://www.nabble.com/bq-type_%3Atrue-for-two-types-doesn%27t-come-up-books.-tp22083323p22083323.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: boost qf weight between 0 and 10
Obviously it should be qb and not bf it looks better. Is there everything in the wiki because I read it but I'm still a bit confused about it. sunnyfr wrote: > > Hi, > > I don't get really, I try to boost a field according to another one but > I've a huge weight when I'm using qf boost like : > > /select?qt=dismax&fl=*&q="obama > meeting"&debugQuery=true&qf=title&bf=product(title,stat_views) > > I will have : > 5803681.0 = (MATCH) sum of: > 4.9400806 = weight(title:"obama meet" in 8216294), product of: > 0.98198587 = queryWeight(title:"obama meet"), product of: > 16.098255 = idf(title: obama=7654 meet=7344) > 0.06099952 = queryNorm > 5.0307045 = fieldWeight(title:"obama meet" in 8216294), product of: > 1.0 = tf(phraseFreq=1.0) > 16.098255 = idf(title: obama=7654 meet=7344) > 0.3125 = fieldNorm(field=title, doc=8216294) > 0.40961993 = weight(text:"obama meet"~100^0.2 in 8216294), product of: > 0.17883755 = queryWeight(text:"obama meet"~100^0.2), product of: > 0.2 = boost > 14.658932 = idf(text: obama=12446 meet=19052) > 0.06099952 = queryNorm > 2.2904582 = fieldWeight(text:"obama meet" in 8216294), product of: > 1.0 = tf(phraseFreq=1.0) > 14.658932 = idf(text: obama=12446 meet=19052) > 0.15625 = fieldNorm(field=text, doc=8216294) > 5803675.5 = (MATCH) FunctionQuery(product(ord(title),sint(stat_views))), > product of: > 9.5142968E7 = product(ord(title)=1119329,sint(stat_views)=85) > 1.0 = boost > 0.06099952 = queryNorm > > > > But this is not equilibrate between this boost in qf and bf, how can I do > ? > > Thanks a lot > > -- View this message in context: http://www.nabble.com/boost-qf-weight-between-0-and-10-tp22081396p22081479.html Sent from the Solr - User mailing list archive at Nabble.com.
boost qf weight between 0 and 10
Hi, I don't get really, I try to boost a field according to another one but I've a huge weight when I'm using qf boost like : /select?qt=dismax&fl=*&q="obama meeting"&debugQuery=true&qf=title&bf=product(title,stat_views) I will have : 5803681.0 = (MATCH) sum of: 4.9400806 = weight(title:"obama meet" in 8216294), product of: 0.98198587 = queryWeight(title:"obama meet"), product of: 16.098255 = idf(title: obama=7654 meet=7344) 0.06099952 = queryNorm 5.0307045 = fieldWeight(title:"obama meet" in 8216294), product of: 1.0 = tf(phraseFreq=1.0) 16.098255 = idf(title: obama=7654 meet=7344) 0.3125 = fieldNorm(field=title, doc=8216294) 0.40961993 = weight(text:"obama meet"~100^0.2 in 8216294), product of: 0.17883755 = queryWeight(text:"obama meet"~100^0.2), product of: 0.2 = boost 14.658932 = idf(text: obama=12446 meet=19052) 0.06099952 = queryNorm 2.2904582 = fieldWeight(text:"obama meet" in 8216294), product of: 1.0 = tf(phraseFreq=1.0) 14.658932 = idf(text: obama=12446 meet=19052) 0.15625 = fieldNorm(field=text, doc=8216294) 5803675.5 = (MATCH) FunctionQuery(product(ord(title),sint(stat_views))), product of: 9.5142968E7 = product(ord(title)=1119329,sint(stat_views)=85) 1.0 = boost 0.06099952 = queryNorm But this is not equilibrate between this boost in qf and bf, how can I do ? Thanks a lot -- View this message in context: http://www.nabble.com/boost-qf-weight-between-0-and-10-tp22081396p22081396.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: solr 1.4 - boost query from where it finds the word(s)
Hi Grant, It doesn't seems to work ? What's wrong with that I done? &bf=product(title^2,stat_views) Thanks Grant Ingersoll-6 wrote: > > You might be able to with FunctionQueries, especially the relatively > new and underpromoted ability that Yonik added to use them to multiply > in scoring instead of adding. > > See http://wiki.apache.org/solr/FunctionQuery > > > > On Feb 12, 2009, at 10:17 AM, sunnyfr wrote: > >> >> Hi Grant, >> >> Thanks for your quick answer. >> >> So there is not a real quick way to increase one field in particular >> according to another one if the text is find there, otherwise how >> can I do >> that in two queries ? >> >> thanks a lot, >> >> >> >> Grant Ingersoll-6 wrote: >>> >>> Hi Sunny, >>> >>> As with any relevance issue, one of the first thing I ask before >>> getting to a solution, is what is the problem you are seeing that >>> makes you want to change the way things work? >>> >>> That being said, the only way you would be able to do this is through >>> some custom components and I'm pretty sure it would involve having to >>> run at least two queries (the first which involves SpanQueries), >>> but I >>> might be missing something. >>> >>> >>> -Grant >>> >>> On Feb 12, 2009, at 5:00 AM, sunnyfr wrote: >>> >>>> >>>> Hi everybody, >>>> Wish you a nice day, >>>> >>>> I've a question, I would like to know if it's possible to boost >>>> differently >>>> some field according to where it find the word. >>>> >>>> Will try to make it more clear; >>>> >>>> I've a book core with title, description and tags. >>>> >>>> If word looked for is found in the title, I would like to boost >>>> differntly >>>> another field like number of view >>>> >>>>> found in the title then : nb_views^2 and rating^1 >>>>> found in the description then : nb_views^0.5 and rating^0.2 >>>>> found in the tag then : nb_views^1 and rating^0.5 >>>> >>>> How can I do that ? >>>> >>>> Even I would love to make something like if after : >>>> if nb_views between 0 and 50 then nb_views^1.3 if nb_views>100 >>>> then >>>> nb_views^2 >>>> >>>> Do you have an idea ? What would you reckon ? >>>> >>>> THANKS A LOT GUYS, >>>> >>>> -- >>>> View this message in context: >>>> http://www.nabble.com/solr-1.4---boost-query-from-where-it-finds-the-word%28s%29-tp21972988p21972988.html >>>> Sent from the Solr - User mailing list archive at Nabble.com. >>>> >>> >>> >>> >> >> -- >> View this message in context: >> http://www.nabble.com/solr-1.4---boost-query-from-where-it-finds-the-word%28s%29-tp21973015p21978064.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> > > > -- View this message in context: http://www.nabble.com/solr-1.4---boost-query-from-where-it-finds-the-word%28s%29-tp21973015p22080404.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: solr 1.4 - boost query from where it finds the word(s)
Sorry, which function is it ?? thanks, Grant Ingersoll-6 wrote: > > You might be able to with FunctionQueries, especially the relatively > new and underpromoted ability that Yonik added to use them to multiply > in scoring instead of adding. > > See http://wiki.apache.org/solr/FunctionQuery > > > > On Feb 12, 2009, at 10:17 AM, sunnyfr wrote: > >> >> Hi Grant, >> >> Thanks for your quick answer. >> >> So there is not a real quick way to increase one field in particular >> according to another one if the text is find there, otherwise how >> can I do >> that in two queries ? >> >> thanks a lot, >> >> >> >> Grant Ingersoll-6 wrote: >>> >>> Hi Sunny, >>> >>> As with any relevance issue, one of the first thing I ask before >>> getting to a solution, is what is the problem you are seeing that >>> makes you want to change the way things work? >>> >>> That being said, the only way you would be able to do this is through >>> some custom components and I'm pretty sure it would involve having to >>> run at least two queries (the first which involves SpanQueries), >>> but I >>> might be missing something. >>> >>> >>> -Grant >>> >>> On Feb 12, 2009, at 5:00 AM, sunnyfr wrote: >>> >>>> >>>> Hi everybody, >>>> Wish you a nice day, >>>> >>>> I've a question, I would like to know if it's possible to boost >>>> differently >>>> some field according to where it find the word. >>>> >>>> Will try to make it more clear; >>>> >>>> I've a book core with title, description and tags. >>>> >>>> If word looked for is found in the title, I would like to boost >>>> differntly >>>> another field like number of view >>>> >>>>> found in the title then : nb_views^2 and rating^1 >>>>> found in the description then : nb_views^0.5 and rating^0.2 >>>>> found in the tag then : nb_views^1 and rating^0.5 >>>> >>>> How can I do that ? >>>> >>>> Even I would love to make something like if after : >>>> if nb_views between 0 and 50 then nb_views^1.3 if nb_views>100 >>>> then >>>> nb_views^2 >>>> >>>> Do you have an idea ? What would you reckon ? >>>> >>>> THANKS A LOT GUYS, >>>> >>>> -- >>>> View this message in context: >>>> http://www.nabble.com/solr-1.4---boost-query-from-where-it-finds-the-word%28s%29-tp21972988p21972988.html >>>> Sent from the Solr - User mailing list archive at Nabble.com. >>>> >>> >>> >>> >> >> -- >> View this message in context: >> http://www.nabble.com/solr-1.4---boost-query-from-where-it-finds-the-word%28s%29-tp21973015p21978064.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> > > > -- View this message in context: http://www.nabble.com/solr-1.4---boost-query-from-where-it-finds-the-word%28s%29-tp21973015p22080195.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: delete snapshot??
How can I remove from time to time, because for the script snapcleaner I just have the option to delete last day ??? thanks a lot Noble and sorry again for all this question, Noble Paul നോബിള് नोब्ळ् wrote: > > The hardlinks will prevent the unused files from getting cleaned up. > So the diskspace is consumed for unused index files also. You may need > to delete unused snapshots from time to time > --Noble > > On Tue, Feb 17, 2009 at 5:24 AM, sunnyfr wrote: >> >> Hi Noble, >> >> I maybe don't get something >> Ok if it's hard link but how come i've not space left on device error and >> 30G shown on the data folder ?? >> sorry I'm quite new >> >> 6.0G/data/solr/book/data/snapshot.20090216214502 >> 35M /data/solr/book/data/snapshot.20090216195003 >> 12M /data/solr/book/data/snapshot.20090216195502 >> 12K /data/solr/book/data/spellchecker2 >> 36M /data/solr/book/data/snapshot.20090216185502 >> 37M /data/solr/book/data/snapshot.20090216203502 >> 6.0M/data/solr/book/data/index >> 12K /data/solr/book/data/snapshot.20090216204002 >> 5.8G/data/solr/book/data/snapshot.20090216172020 >> 12K /data/solr/book/data/spellcheckerFile >> 28K /data/solr/book/data/snapshot.20090216200503 >> 40K /data/solr/book/data/snapshot.20090216194002 >> 24K /data/solr/book/data/snapshot.2009021622 >> 32K /data/solr/book/data/snapshot.20090216184502 >> 20K /data/solr/book/data/snapshot.20090216191004 >> 1.1M/data/solr/book/data/snapshot.20090216213502 >> 1.1M/data/solr/book/data/snapshot.20090216201502 >> 1.1M/data/solr/book/data/snapshot.20090216213005 >> 24K /data/solr/book/data/snapshot.20090216191502 >> 1.1M/data/solr/book/data/snapshot.20090216212503 >> 107M/data/solr/book/data/snapshot.20090216212002 >> 14M /data/solr/book/data/snapshot.20090216190502 >> 32K /data/solr/book/data/snapshot.20090216201002 >> 2.3M/data/solr/book/data/snapshot.20090216204502 >> 28K /data/solr/book/data/snapshot.20090216184002 >> 5.8G/data/solr/book/data/snapshot.20090216181425 >> 44K /data/solr/book/data/snapshot.20090216190001 >> 20K /data/solr/book/data/snapshot.20090216183401 >> 1.1M/data/solr/book/data/snapshot.20090216203002 >> 44K /data/solr/book/data/snapshot.20090216194502 >> 36K /data/solr/book/data/snapshot.20090216185004 >> 12K /data/solr/book/data/snapshot.20090216182720 >> 12K /data/solr/book/data/snapshot.20090216214001 >> 5.8G/data/solr/book/data/snapshot.20090216175106 >> 1.1M/data/solr/book/data/snapshot.20090216202003 >> 5.8G/data/solr/book/data/snapshot.20090216173224 >> 12K /data/solr/book/data/spellchecker1 >> 1.1M/data/solr/book/data/snapshot.20090216202502 >> 30G /data/solr/book/data >> thanks a lot, >> >> >> Noble Paul നോബിള് नोब्ळ् wrote: >>> >>> they are just hardlinks. they do not consume space on disk >>> >>> On Mon, Feb 16, 2009 at 10:34 PM, sunnyfr wrote: >>>> >>>> Hi, >>>> >>>> Ok but can I use it more often then every day like every three hours, >>>> because snapshot are quite big. >>>> >>>> Thanks a lot, >>>> >>>> >>>> Bill Au wrote: >>>>> >>>>> The --delete option of the rsync command deletes extraneous files from >>>>> the >>>>> destination directory. It does not delete Solr snapshots. To do that >>>>> you >>>>> can use the snapcleaner on the master and/or slave. >>>>> >>>>> Bill >>>>> >>>>> On Fri, Feb 13, 2009 at 10:15 AM, sunnyfr >>>>> wrote: >>>>> >>>>>> >>>>>> root 26834 16.2 0.0 19412 824 ?S16:05 0:08 >>>>>> rsync >>>>>> -Wa >>>>>> --delete rsync://##.##.##.##:18180/solr/snapshot.20090213160051/ >>>>>> /data/solr/books/data/snapshot.20090213160051-wip >>>>>> >>>>>> Hi obviously it can't delete them because the adress is bad it >>>>>> shouldnt >>>>>> be >>>>>> : >>>>>> rsync://##.##.##.##:18180/solr/snapshot.20090213160051/ >>>>>> but: >>>>>> rsync://##.##.##.##:18180/solr/books/snapshot.20090213160051/ >>>>>> >>>>>> Where should I change this, I checked my sc
Re: delete snapshot??
Hi Noble, I maybe don't get something Ok if it's hard link but how come i've not space left on device error and 30G shown on the data folder ?? sorry I'm quite new 6.0G/data/solr/book/data/snapshot.20090216214502 35M /data/solr/book/data/snapshot.20090216195003 12M /data/solr/book/data/snapshot.20090216195502 12K /data/solr/book/data/spellchecker2 36M /data/solr/book/data/snapshot.20090216185502 37M /data/solr/book/data/snapshot.20090216203502 6.0M/data/solr/book/data/index 12K /data/solr/book/data/snapshot.20090216204002 5.8G/data/solr/book/data/snapshot.20090216172020 12K /data/solr/book/data/spellcheckerFile 28K /data/solr/book/data/snapshot.20090216200503 40K /data/solr/book/data/snapshot.20090216194002 24K /data/solr/book/data/snapshot.2009021622 32K /data/solr/book/data/snapshot.20090216184502 20K /data/solr/book/data/snapshot.20090216191004 1.1M/data/solr/book/data/snapshot.20090216213502 1.1M/data/solr/book/data/snapshot.20090216201502 1.1M/data/solr/book/data/snapshot.20090216213005 24K /data/solr/book/data/snapshot.20090216191502 1.1M/data/solr/book/data/snapshot.20090216212503 107M/data/solr/book/data/snapshot.20090216212002 14M /data/solr/book/data/snapshot.20090216190502 32K /data/solr/book/data/snapshot.20090216201002 2.3M/data/solr/book/data/snapshot.20090216204502 28K /data/solr/book/data/snapshot.20090216184002 5.8G/data/solr/book/data/snapshot.20090216181425 44K /data/solr/book/data/snapshot.20090216190001 20K /data/solr/book/data/snapshot.20090216183401 1.1M/data/solr/book/data/snapshot.20090216203002 44K /data/solr/book/data/snapshot.20090216194502 36K /data/solr/book/data/snapshot.20090216185004 12K /data/solr/book/data/snapshot.20090216182720 12K /data/solr/book/data/snapshot.20090216214001 5.8G/data/solr/book/data/snapshot.20090216175106 1.1M/data/solr/book/data/snapshot.20090216202003 5.8G/data/solr/book/data/snapshot.20090216173224 12K /data/solr/book/data/spellchecker1 1.1M/data/solr/book/data/snapshot.20090216202502 30G /data/solr/book/data thanks a lot, Noble Paul നോബിള് नोब्ळ् wrote: > > they are just hardlinks. they do not consume space on disk > > On Mon, Feb 16, 2009 at 10:34 PM, sunnyfr wrote: >> >> Hi, >> >> Ok but can I use it more often then every day like every three hours, >> because snapshot are quite big. >> >> Thanks a lot, >> >> >> Bill Au wrote: >>> >>> The --delete option of the rsync command deletes extraneous files from >>> the >>> destination directory. It does not delete Solr snapshots. To do that >>> you >>> can use the snapcleaner on the master and/or slave. >>> >>> Bill >>> >>> On Fri, Feb 13, 2009 at 10:15 AM, sunnyfr wrote: >>> >>>> >>>> root 26834 16.2 0.0 19412 824 ?S16:05 0:08 rsync >>>> -Wa >>>> --delete rsync://##.##.##.##:18180/solr/snapshot.20090213160051/ >>>> /data/solr/books/data/snapshot.20090213160051-wip >>>> >>>> Hi obviously it can't delete them because the adress is bad it shouldnt >>>> be >>>> : >>>> rsync://##.##.##.##:18180/solr/snapshot.20090213160051/ >>>> but: >>>> rsync://##.##.##.##:18180/solr/books/snapshot.20090213160051/ >>>> >>>> Where should I change this, I checked my script.conf on the slave >>>> server >>>> but >>>> it seems good. >>>> >>>> Because files can be very big and my server in few hours is getting >>>> full. >>>> >>>> So actually snapcleaner is not necessary on the master ? what about the >>>> slave? >>>> >>>> Thanks a lot, >>>> Sunny >>>> -- >>>> View this message in context: >>>> http://www.nabble.com/delete-snapshot---tp21998333p21998333.html >>>> Sent from the Solr - User mailing list archive at Nabble.com. >>>> >>>> >>> >>> >> >> -- >> View this message in context: >> http://www.nabble.com/delete-snapshot---tp21998333p22041332.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > > > -- > --Noble Paul > > -- View this message in context: http://www.nabble.com/delete-snapshot---tp21998333p22048398.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: delete snapshot??
Hi Noble, I maybe don't get something Ok if it's hard link but how come i've not space left on device error and 30G shown on the data folder ?? sorry I'm quite new 6.0G/data/solr/book/data/snapshot.20090216214502 35M /data/solr/book/data/snapshot.20090216195003 12M /data/solr/book/data/snapshot.20090216195502 12K /data/solr/book/data/spellchecker2 36M /data/solr/book/data/snapshot.20090216185502 37M /data/solr/book/data/snapshot.20090216203502 6.0M/data/solr/book/data/index 12K /data/solr/book/data/snapshot.20090216204002 5.8G/data/solr/book/data/snapshot.20090216172020 12K /data/solr/book/data/spellcheckerFile 28K /data/solr/book/data/snapshot.20090216200503 40K /data/solr/book/data/snapshot.20090216194002 24K /data/solr/book/data/snapshot.2009021622 32K /data/solr/book/data/snapshot.20090216184502 20K /data/solr/book/data/snapshot.20090216191004 1.1M/data/solr/book/data/snapshot.20090216213502 1.1M/data/solr/book/data/snapshot.20090216201502 1.1M/data/solr/book/data/snapshot.20090216213005 24K /data/solr/book/data/snapshot.20090216191502 1.1M/data/solr/book/data/snapshot.20090216212503 107M/data/solr/book/data/snapshot.20090216212002 14M /data/solr/book/data/snapshot.20090216190502 32K /data/solr/book/data/snapshot.20090216201002 2.3M/data/solr/book/data/snapshot.20090216204502 28K /data/solr/book/data/snapshot.20090216184002 5.8G/data/solr/book/data/snapshot.20090216181425 44K /data/solr/book/data/snapshot.20090216190001 20K /data/solr/book/data/snapshot.20090216183401 1.1M/data/solr/book/data/snapshot.20090216203002 44K /data/solr/book/data/snapshot.20090216194502 36K /data/solr/book/data/snapshot.20090216185004 12K /data/solr/book/data/snapshot.20090216182720 12K /data/solr/book/data/snapshot.20090216214001 5.8G/data/solr/book/data/snapshot.20090216175106 1.1M/data/solr/book/data/snapshot.20090216202003 5.8G/data/solr/book/data/snapshot.20090216173224 12K /data/solr/book/data/spellchecker1 1.1M/data/solr/book/data/snapshot.20090216202502 30G /data/solr/book/data thanks a lot, Noble Paul നോബിള് नोब्ळ् wrote: > > they are just hardlinks. they do not consume space on disk > > On Mon, Feb 16, 2009 at 10:34 PM, sunnyfr wrote: >> >> Hi, >> >> Ok but can I use it more often then every day like every three hours, >> because snapshot are quite big. >> >> Thanks a lot, >> >> >> Bill Au wrote: >>> >>> The --delete option of the rsync command deletes extraneous files from >>> the >>> destination directory. It does not delete Solr snapshots. To do that >>> you >>> can use the snapcleaner on the master and/or slave. >>> >>> Bill >>> >>> On Fri, Feb 13, 2009 at 10:15 AM, sunnyfr wrote: >>> >>>> >>>> root 26834 16.2 0.0 19412 824 ?S16:05 0:08 rsync >>>> -Wa >>>> --delete rsync://##.##.##.##:18180/solr/snapshot.20090213160051/ >>>> /data/solr/books/data/snapshot.20090213160051-wip >>>> >>>> Hi obviously it can't delete them because the adress is bad it shouldnt >>>> be >>>> : >>>> rsync://##.##.##.##:18180/solr/snapshot.20090213160051/ >>>> but: >>>> rsync://##.##.##.##:18180/solr/books/snapshot.20090213160051/ >>>> >>>> Where should I change this, I checked my script.conf on the slave >>>> server >>>> but >>>> it seems good. >>>> >>>> Because files can be very big and my server in few hours is getting >>>> full. >>>> >>>> So actually snapcleaner is not necessary on the master ? what about the >>>> slave? >>>> >>>> Thanks a lot, >>>> Sunny >>>> -- >>>> View this message in context: >>>> http://www.nabble.com/delete-snapshot---tp21998333p21998333.html >>>> Sent from the Solr - User mailing list archive at Nabble.com. >>>> >>>> >>> >>> >> >> -- >> View this message in context: >> http://www.nabble.com/delete-snapshot---tp21998333p22041332.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > > > -- > --Noble Paul > > -- View this message in context: http://www.nabble.com/delete-snapshot---tp21998333p22048391.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: delete snapshot??
Hi Noble, But how come i've space error ?? :( thanks a lot, Feb 16 18:28:34 search-07 jsvc.exec[8872]: ataImporter.java:361) Caused by: java.io.IOException: No space left on device ^Iat java.io.RandomAccessFile.writeBytes(Native Method) ^Iat java.io.RandomAccessFile.write(RandomAccessFile.java:466) ^Iat org.apache.lucene.store.FSDirectory$FSIndexOutput.flushBuffer(FSDirectory.java:679) ^Iat org.apache.lucene.store.BufferedIndexOutput.flushBuffer(BufferedIndexOutput.java:96) ^Iat org.apache.lucene.store.BufferedIndexOutput.flush(BufferedIndexOutput.java:85) ^Iat org.apache.lucene.store.BufferedIndexOutput.seek(BufferedIndexOutput.java:124) ^Iat org.apache.lucene.store.FSDirectory$FSIndexOutput.seek(FSDirectory.java:704) ^Iat org.apache.lucene.index.TermInfosWriter.close(TermInfosWriter.java:220) ^Iat org.apache.lucene.index.FormatPostingsFieldsWriter.finish(FormatPostingsFieldsWriter.java:70) ^Iat org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:494) ^Iat org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:141) ^Iat org.apache.lucene.index.IndexW Noble Paul നോബിള് नोब्ळ् wrote: > > they are just hardlinks. they do not consume space on disk > > On Mon, Feb 16, 2009 at 10:34 PM, sunnyfr wrote: >> >> Hi, >> >> Ok but can I use it more often then every day like every three hours, >> because snapshot are quite big. >> >> Thanks a lot, >> >> >> Bill Au wrote: >>> >>> The --delete option of the rsync command deletes extraneous files from >>> the >>> destination directory. It does not delete Solr snapshots. To do that >>> you >>> can use the snapcleaner on the master and/or slave. >>> >>> Bill >>> >>> On Fri, Feb 13, 2009 at 10:15 AM, sunnyfr wrote: >>> >>>> >>>> root 26834 16.2 0.0 19412 824 ?S16:05 0:08 rsync >>>> -Wa >>>> --delete rsync://##.##.##.##:18180/solr/snapshot.20090213160051/ >>>> /data/solr/books/data/snapshot.20090213160051-wip >>>> >>>> Hi obviously it can't delete them because the adress is bad it shouldnt >>>> be >>>> : >>>> rsync://##.##.##.##:18180/solr/snapshot.20090213160051/ >>>> but: >>>> rsync://##.##.##.##:18180/solr/books/snapshot.20090213160051/ >>>> >>>> Where should I change this, I checked my script.conf on the slave >>>> server >>>> but >>>> it seems good. >>>> >>>> Because files can be very big and my server in few hours is getting >>>> full. >>>> >>>> So actually snapcleaner is not necessary on the master ? what about the >>>> slave? >>>> >>>> Thanks a lot, >>>> Sunny >>>> -- >>>> View this message in context: >>>> http://www.nabble.com/delete-snapshot---tp21998333p21998333.html >>>> Sent from the Solr - User mailing list archive at Nabble.com. >>>> >>>> >>> >>> >> >> -- >> View this message in context: >> http://www.nabble.com/delete-snapshot---tp21998333p22041332.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > > > -- > --Noble Paul > > -- View this message in context: http://www.nabble.com/delete-snapshot---tp21998333p22044788.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: delete snapshot??
Hi, Ok but can I use it more often then every day like every three hours, because snapshot are quite big. Thanks a lot, Bill Au wrote: > > The --delete option of the rsync command deletes extraneous files from the > destination directory. It does not delete Solr snapshots. To do that you > can use the snapcleaner on the master and/or slave. > > Bill > > On Fri, Feb 13, 2009 at 10:15 AM, sunnyfr wrote: > >> >> root 26834 16.2 0.0 19412 824 ?S16:05 0:08 rsync >> -Wa >> --delete rsync://##.##.##.##:18180/solr/snapshot.20090213160051/ >> /data/solr/books/data/snapshot.20090213160051-wip >> >> Hi obviously it can't delete them because the adress is bad it shouldnt >> be >> : >> rsync://##.##.##.##:18180/solr/snapshot.20090213160051/ >> but: >> rsync://##.##.##.##:18180/solr/books/snapshot.20090213160051/ >> >> Where should I change this, I checked my script.conf on the slave server >> but >> it seems good. >> >> Because files can be very big and my server in few hours is getting full. >> >> So actually snapcleaner is not necessary on the master ? what about the >> slave? >> >> Thanks a lot, >> Sunny >> -- >> View this message in context: >> http://www.nabble.com/delete-snapshot---tp21998333p21998333.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > -- View this message in context: http://www.nabble.com/delete-snapshot---tp21998333p22041332.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: snapshot as big as the index folder?
It change a lot in few minute ?? is it normal ? thanks 5.8Gbook/data/snapshot.20090216153346 4.0Kbook/data/index 5.8Gbook/data/ r...@search-07:/data/solr# du -h book/data/ 5.8Gbook/data/snapshot.20090216153346 3.7Gbook/data/index 4.0Kbook/data/snapshot.20090216153759 9.4Gbook/data/ r...@search-07:/data/solr# du -h book/data/ 5.8Gvideo/data/snapshot.20090216153346 4.4Gbook/data/index 4.0Kbook/data/snapshot.20090216153759 11G book/data/ r...@search-07:/data/solr# du -h book/data/ 5.8Gbook/data/snapshot.20090216153346 5.8Gbook/data/index 4.0Kbook/data/snapshot.20090216154819 4.0Kbook/data/snapshot.20090216154820 15M book/data/snapshot.20090216153759 12G book/data/ sunnyfr wrote: > > Hi, > > Is it normal or did I miss something ?? > 5.8G book/data/snapshot.20090216153346 > 12K book/data/spellchecker2 > 4.0K book/data/index > 12K book/data/spellcheckerFile > 12K book/data/spellchecker1 > 5.8G book/data/ > > Last update ? > 92562 > 45492 > 0 > 2009-02-16 15:20:01 > 2009-02-16 15:20:01 > 2009-02-16 15:20:42 > 2009-02-16 15:20:42 > 13223 > − > > Indexing completed. Added/Updated: 13223 documents. Deleted 0 documents. > > 2009-02-16 15:33:50 > 2009-02-16 15:33:50 > 0:13:48.853 > > > Thanks a lot, > > -- View this message in context: http://www.nabble.com/snapshot-as-big-as-the-index-folder--tp22038427p22038656.html Sent from the Solr - User mailing list archive at Nabble.com.
snapshot as big as the index folder?
Hi, Is it normal or did I miss something ?? 5.8Gbook/data/snapshot.20090216153346 12K book/data/spellchecker2 4.0Kbook/data/index 12K book/data/spellcheckerFile 12K book/data/spellchecker1 5.8Gbook/data/ Last update ? 92562 45492 0 2009-02-16 15:20:01 2009-02-16 15:20:01 2009-02-16 15:20:42 2009-02-16 15:20:42 13223 − Indexing completed. Added/Updated: 13223 documents. Deleted 0 documents. 2009-02-16 15:33:50 2009-02-16 15:33:50 0:13:48.853 Thanks a lot, -- View this message in context: http://www.nabble.com/snapshot-as-big-as-the-index-folder--tp22038427p22038427.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: almost realtime updates with replication
Hi Noble, So ok I don't mind really if it miss one, if it get the last one it's good. I've was wondering as well if a snapshot is created even if no document has been update? Thanks a lot Noble, Wish you a very nice day, Noble Paul നോബിള് नोब्ळ् wrote: > > I guess , it should not be a problem > --Noble > > On Mon, Feb 16, 2009 at 3:28 PM, sunnyfr wrote: >> >> Hi Hoss, >> >> Is it a problem if the snappuller miss one snapshot before the last one >> ?? >> >> Cheer, >> Have a nice day, >> >> >> hossman wrote: >>> >>> : >>> : There are a couple queries that we would like to run almost realtime >>> so >>> : I would like to have it so our client sends an update on every new >>> : document and then have solr configured to do an autocommit every 5-10 >>> : seconds. >>> : >>> : reading the Wiki, it seems like this isn't possible because of the >>> : strain of snapshotting and pulling to the slaves at such a high rate. >>> : What I was thinking was for these few queries to just query the master >>> : and the rest can query the slave with the not realtime data, although >>> : I'm assuming this wouldn't work either because since a snapshot is >>> : created on every commit, we would still impact the performance too >>> much? >>> >>> there is no reason why a commit has to trigger a snapshot, that happens >>> only if you configure a postCommit hook to do so in your solrconfig.xml >>> >>> you can absolutely commit every 5 seconds, but have a seperate cron task >>> that runs snapshooter ever 5 minutes -- you could even continue to run >>> snapshooter on every commit, and get a new snapshot ever 5 seconds, but >>> only run snappuller on your slave machines ever 5 minutes (the >>> snapshots are hardlinks and don't take up a lot of space, and snappuller >>> only needs to fetch the most recent snapshot) >>> >>> your idea of querying the msater directly for these queries seems >>> perfectly fine to me ... just make sure the auto warm count on the >>> caches >>> on your master is very tiny so the new searchers are ready quickly after >>> each commit. >>> >>> >>> >>> >>> -Hoss >>> >>> >>> >> >> -- >> View this message in context: >> http://www.nabble.com/almost-realtime-updates-with-replication-tp12276614p22034406.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > > > -- > --Noble Paul > > -- View this message in context: http://www.nabble.com/almost-realtime-updates-with-replication-tp12276614p22037977.html Sent from the Solr - User mailing list archive at Nabble.com.
snapshot created if there is no documente updated/new?
Hi I would like to know if a snapshot is automaticly created even if there is no document update or added ? Thanks a lot, -- View this message in context: http://www.nabble.com/snapshot-created-if-there-is-no-documente-updated-new--tp22034462p22034462.html Sent from the Solr - User mailing list archive at Nabble.com.
