Re: is it possible to do automatic indexing in solr ?
you can use the task scheduler of windows or tomcat listener ,the related solution is posted on the solr wiki http://wiki.apache.org/solr/DataImportHandler#HTTPPostScheduler
Re: how to do schduling in solr ?
You can have a look at this page: http://wiki.apache.org/solr/DataImportHandler#HTTPPostScheduler this scheduler can post not only command like delta-import but also command like full-import
Why I can't take an full-import with entity name?
I am using solr1.3,I updated solr index throgh solr delta import every two
hours. but the delta import is database connection wasteful.
So i want to use full-import with entity name instead of delta import.
my db-data-config.xml file:
entity name=article pk=Article_ID query=select
Article_ID,Article_Title,Article_Abstract from Article_Detail
field name=Article_ID column=Article_ID /
/entity
entity name=delta_article pk=Article_ID rootEngity=false
query=select Article_ID,Article_Title,Article_Abstract from Article_Detail
where Article_IDgt;'${dataimporter.request.minID}' and Article_ID
lt;='{dataimporter.request.maxID}'
field name=Article_ID column=Article_ID /
/entity
then I uses
http://192.168.1.98:8081/solr/db_article/dataimport?command=full-importentity=delta_articlecommit=trueclean=falsemaxID=1000minID=10
but the solr will finish nearyly instant,and there is no any record
imported. but what the fact is there are many records meets the condtion of
maxID and minID.
the tomcat log:
信息: [db_article] webapp=/solr path=/dataimport
params={maxID=6737277clean=falsecommit=trueentity=delta_articlecommand=full-importminID=6736841}
status=0 QTime=0
2011-8-29 19:00:03 org.apache.solr.handler.dataimport.DataImporter
doFullImport
信息: Starting Full Import
2011-8-29 19:00:03 org.apache.solr.handler.dataimport.SolrWriter
readIndexerProperties
信息: Read dataimport.properties
2011-8-29 19:00:03 org.apache.solr.handler.dataimport.SolrWriter
persistStartTime
信息: Wrote last indexed time to dataimport.properties
2011-8-29 19:00:03 org.apache.solr.handler.dataimport.DocBuilder commit
信息: Full Import completed successfully
some body who can help or some advices?
Re: Solr request filter and indexing process
A ha,I have found the root cause , the Solr has return the result properly .The root cause is the SolrPHPClient, The SolrPHPClient uses file_get_contents function for connecting to Solr by default ,this function is not stable, usually returns http status error. thanks for everybody who gives me help.Good luck for you! 2011/8/2 Chris Hostetter hossman_luc...@fucit.org : thanks for the reply. This is tomcat log files on my Solr Server: : I found that : if the server returns status=0 and QTime=0, the SolrPhpClient : will throughs an Exception. But the same query String will not always return : status=0 and QTime=0. The Query String is valid, I have tested them in Solr I know nothing about PHP but if your client code is throwing an exception anytime status=0 and QTime=0 then it sounds like a bug in your client code -- there is no reason why those two numbers being 0 should be considered an error. It just means the request was processed in under a millisecond. -Hoss
Re: Solr request filter and indexing process
thanks for the reply. This is tomcat log files on my Solr Server:
I found that : if the server returns status=0 and QTime=0, the SolrPhpClient
will throughs an Exception. But the same query String will not always return
status=0 and QTime=0. The Query String is valid, I have tested them in Solr
Admin Page. I uses ajax to call the PHP script ,then PHP script calls
SolrPHPClient to send query request to Solr. I can't find where is the root
cause.
2011-8-1 20:41:34 org.apache.solr.core.SolrCore execute
信息: [db_article] webapp=/solr path=/select
params={start=0q=(+((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+))+AND+article_detail_id:(440189)
json.nl=mapwt=jsonrows=12} hits=0 status=0 QTime=0
2011-8-1 20:41:35 org.apache.solr.core.SolrCore execute
信息: [db_article] webapp=/solr path=/select
params={sort=extracted_time+descstart=0q=((+((+京沪高铁)+++AND+(+运营)+)+((+个税)+++AND+(+改革)+))+AND+%2B(++((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)++((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)))+AND+extracted_time:[2011-08-01T00:00:00Z-8HOURS+TO+2011-08-01T23:59:59Z-8HOURS]
json.nl=mapwt=jsonrows=3} hits=45 status=0 QTime=0
2011-8-1 20:41:35 org.apache.solr.core.SolrCore execute
信息: [db_article] webapp=/solr path=/select
params={start=0q=(+((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+))+AND+article_detail_id:(622324+||+622292+||+621918)
json.nl=mapwt=jsonrows=3} hits=3 status=0 QTime=0
2011-8-1 20:41:35 org.apache.solr.core.SolrCore execute
信息: [db_operation] webapp=/solr path=/select
params={sort=extracted_time+descstart=0q=((+((+京沪高铁)+++AND+(+运营)+)+((+个税)+++AND+(+改革)+))+AND+-(++((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)++((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)))+AND+-user_confirm_emotion_type:[-1+TO+-1]
json.nl=mapwt=jsonrows=9} hits=73 status=0 QTime=0
2011-8-1 20:41:36 org.apache.solr.core.SolrCore execute
信息: [db_article] webapp=/solr path=/select
params={start=0q=(+((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+))+AND+article_detail_id:(621452+||+602037+||+599542+||+599541+||+599532+||+599529+||+599527+||+599526+||+599525)
json.nl=mapwt=jsonrows=9} hits=0 status=0 QTime=0
2011-8-1 20:41:37 org.apache.solr.core.SolrCore execute
信息: [db_article] webapp=/solr path=/select
params={sort=extracted_time+descstart=0q=((+((+南海+陈光标+温家宝)+++AND+(+越南+菲律宾)+)+((+暴雨)+++AND+(+长江)+AND+(+-武汉))+((+动车)+++AND+(+追尾+灾难)+)+((+美国债务)+++AND+(+违约)+)+((+乐思软件)+++AND+(+招聘)+)+((+大运会)+++)+((+京沪高铁)+++AND+(+运营)+)+((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)+((+个税)+++AND+(+改革)+)+((+顺德农商银行+顺德农村商业银行+顺德农商行)+++AND+(+假钱+贿赂+反价+骗贷+黑幕+违约)+))+AND+%2B(++((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)++((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)))+AND+media_type_code:N+AND+extracted_time:[2011-08-01T00:00:00Z-8HOURS+TO+2011-08-01T23:59:59Z-8HOURS]
json.nl=mapwt=jsonrows=3} hits=136 status=0 QTime=10
2011-8-1 20:41:37 org.apache.solr.core.SolrCore execute
信息: [db_article] webapp=/solr path=/select
params={start=0q=(+((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+))+AND+article_detail_id:(625436+||+625430+||+624871)
json.nl=mapwt=jsonrows=3} hits=3 status=0 QTime=0
2011-8-1 20:41:37 org.apache.solr.core.SolrCore execute
信息: [db_operation] webapp=/solr path=/select
params={sort=extracted_time+descstart=0q=((+((+南海+陈光标+温家宝)+++AND+(+越南+菲律宾)+)+((+暴雨)+++AND+(+长江)+AND+(+-武汉))+((+动车)+++AND+(+追尾+灾难)+)+((+美国债务)+++AND+(+违约)+)+((+乐思软件)+++AND+(+招聘)+)+((+大运会)+++)+((+京沪高铁)+++AND+(+运营)+)+((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)+((+个税)+++AND+(+改革)+)+((+顺德农商银行+顺德农村商业银行+顺德农商行)+++AND+(+假钱+贿赂+反价+骗贷+黑幕+违约)+))+AND+-(++((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)++((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)))+AND+media_type_code:N+AND+-user_confirm_emotion_type:[-1+TO+-1]
json.nl=mapwt=jsonrows=9} hits=65 status=0 QTime=10
2011-8-1 20:41:37 org.apache.solr.core.SolrCore execute
信息: [db_article] webapp=/solr path=/select
params={start=0q=(+((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+))+AND+article_detail_id:(624595+||+623244+||+623111+||+602402+||+597554+||+596524+||+596148+||+595638+||+595636)
json.nl=mapwt=jsonrows=9} hits=0 status=0 QTime=0
2011-8-1 20:41:39 org.apache.solr.core.SolrCore execute
信息: [db_article] webapp=/solr path=/select
params={sort=extracted_time+descstart=0q=((+((+南海+陈光标+温家宝)+++AND+(+越南+菲律宾)+)+((+暴雨)+++AND+(+长江)+AND+(+-武汉))+((+动车)+++AND+(+追尾+灾难)+)+((+美国债务)+++AND+(+违约)+)+((+乐思软件)+++AND+(+招聘)+)+((+大运会)+++)+((+京沪高铁)+++AND+(+运营)+)+((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)+((+个税)+++AND+(+改革)+)+((+顺德农商银行+顺德农村商业银行+顺德农商行)+++AND+(+假钱+贿赂+反价+骗贷+黑幕+违约)+))+AND+%2B(++((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)++((+个税+起征点+南海)+++AND+(+走秀+过重+反对+贫富分化+开火)+)))+AND+media_type_code:F+AND+extracted_time:[2011-08-01T00:00:00Z-8HOURS+TO+2011-08-01T23:59:59Z-8HOURS]
json.nl=mapwt=jsonrows=3} hits=83 status=0 QTime=10
2011-8-1 20:41:39 org.apache.solr.core.SolrCore execute
信息: [db_article] webapp=/solr path=/select
Re: Solr request filter and indexing process
thank you for reply my message. But the developer machine is a new Dell PC with windows7 and 4G memory. If I add 10 document through SolrPHPClient's addDocuments function with XML format, It will takens about more than five seconds. This speed must be unacceptable to our user. I have already disabled the optimize operation after update, Have any other factors is related ? 2011/7/31 O. Klein kl...@octoweb.nl Might wanna check your resources, like memory and diskspace. -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-request-filter-and-indexing-process-tp3212907p3213512.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr request filter and indexing process
Thanks for reply ,the Solr server uses dataimporthandler, it can import 60 thousands records(about 4G index files) properly in about 30 minutes. So I think the memory is enough. Do you know some performance factors about SolrPhpClient ? 2011/7/31 O. Klein kl...@octoweb.nl I don't have much experience with Solr on windows, but have you given Solr enough java heap size etc? Im assuming you are running Solr stand alone. Maybe try and run it in Tomcat? -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-request-filter-and-indexing-process-tp3212907p3213650.html Sent from the Solr - User mailing list archive at Nabble.com.
Solr request filter and indexing process
Hello,Dear friends, I have got an problem in developing with solr. In My Application ,It must sends multiple query to solr server after the page is loaded. Then I found a problem: some request will return statusCode:0 and QTime:0, The solr has accepted the request, but It does not return a result document. If I send each request one by one manually ,It will return the result. But If I send the request frequently in a very short times, It will return nothing only statusCode:0 and QTime:0. I think this may be a stratege for solr. but i can't find any documents or discussions on the internet. so i want you can help me. edited on 2011-07-28 and now I have a new problem, I am developing on php, so I connect solr through solrPhpClient( an opensource project on google code). I find the speed of add many documents is very slow. when I add ten documents to an solr index, It must takes more than 5 minutes(Because of the commit process ) anybody can help me?
about the Solr request filter
Hello,Dear friends, I have got an problem in developing with solr. In My Application ,It must sends multiple query to solr server after the page is loaded. Then I found a problem: some request will return statusCode:0 and QTime:0, The solr has accepted the request, but It does not return a result document. If I send each request one by one manually ,It will return the result. But If I send the request frequently in a very short times, It will return nothing only statusCode:0 and QTime:0. I think this may be a stratege for solr. but i can't find any documents or discussions on the internet. so i want you can help me. -- Surely, 你永远是最棒的!
