Re: Performing DIH on predefined list of IDS

2015-02-21 Thread Shawn Heisey
On 2/21/2015 6:33 PM, Walter Underwood wrote: > Never do POST for a read-only request. Never. That only guarantees that you > cannot reproduce the problem by looking at the logs. > > If your design requires extremely long GET requests, you may need to re-think > your design. I agree with those

Re: Performing DIH on predefined list of IDS

2015-02-21 Thread Walter Underwood
Am an expert? Not sure, but I worked on an enterprise search spider and search engine for about a decade (Ultraseek Server) and I’ve done customer-facing search for another 6+ years. Let the server reject URLs it cannot handle. Great servers will return a 414, good servers will return a 400, br

Re: Performing DIH on predefined list of IDS

2015-02-21 Thread Shawn Heisey
On 2/21/2015 1:46 AM, steve wrote: > Careful with the GETs! There is a real, hard limit on the length of a GET url > (in the low hundreds of characters). That's why a POST is so much better for > complex queries; the limit is in the hundreds of MegaBytes. The limit on a GET command (including th

RE: Performing DIH on predefined list of IDS

2015-02-21 Thread steve
ot; > From: wun...@wunderwood.org > Subject: Re: Performing DIH on predefined list of IDS > Date: Sat, 21 Feb 2015 09:50:46 -0800 > To: solr-user@lucene.apache.org > > The HTTP protocol does not set a limit on GET URL size, but individual web > servers usually do. You should get a r

Re: Performing DIH on predefined list of IDS

2015-02-21 Thread Walter Underwood
gt; >> Date: Sat, 21 Feb 2015 01:42:03 -0700 >> From: osta...@gmail.com >> To: solr-user@lucene.apache.org >> Subject: Re: Performing DIH on predefined list of IDS >> >> Yes, you right, I am not using a DB. >> SolrEntityProcessor is using a GET method

RE: Performing DIH on predefined list of IDS

2015-02-21 Thread steve
And I'm familiar with the setup and configuration using Python, JavaScript, and PHP; not at all with Java. > Date: Sat, 21 Feb 2015 01:52:07 -0700 > From: osta...@gmail.com > To: solr-user@lucene.apache.org > Subject: RE: Performing DIH on predefined list of IDS > > Tha

RE: Performing DIH on predefined list of IDS

2015-02-21 Thread SolrUser1543
That's right, but I am not sure that if it is works with Get I will able to use Post without changing it. -- View this message in context: http://lucene.472066.n3.nabble.com/Performing-DIH-on-predefined-list-of-IDS-tp4187589p4187838.html Sent from the Solr - User mailing list archive at Nabble

RE: Performing DIH on predefined list of IDS

2015-02-21 Thread steve
> To: solr-user@lucene.apache.org > Subject: Re: Performing DIH on predefined list of IDS > > Yes, you right, I am not using a DB. > SolrEntityProcessor is using a GET method, so I will need to send > relatively big URL ( something like a hundreds of ids ) hope it will be >

Re: Performing DIH on predefined list of IDS

2015-02-21 Thread SolrUser1543
Yes, you right, I am not using a DB. SolrEntityProcessor is using a GET method, so I will need to send relatively big URL ( something like a hundreds of ids ) hope it will be possible. Any way I think it is the only method to perform reindex if I want to control it and be able to continue fr

Re: Performing DIH on predefined list of IDS

2015-02-20 Thread Shawn Heisey
On 2/20/2015 3:46 PM, Shawn Heisey wrote: > If the URL parameter is "idlist" then you can use > ${dih.request.idlist} in your SELECT statement. I realized after I sent this that you are not using a database ... the list would simply go in the query you send to the other server. I don't know wheth

Re: Performing DIH on predefined list of IDS

2015-02-20 Thread Shawn Heisey
On 2/20/2015 2:57 PM, SolrUser1543 wrote: > That's the reason that I want to run on predefined list of IDs. > In this case I will able to restart from any point and to know about filed > IDs. You can include information on a URL parameter and then use that URL parameter inside your dih config.

Re: Performing DIH on predefined list of IDS

2015-02-20 Thread SolrUser1543
My index has about 110 millions of documents. The index is split over several shards. May be the number it's not so big ,but each document is relatively large. The reason to perform the reindex is something like adding a new fields , or adding some update processor which can extract something fr

Re: Performing DIH on predefined list of IDS

2015-02-20 Thread Mikhail Khludnev
It's a little bit hard to get the overall context eg why do you live with OOME as usual, what's the reasoning to pull from one index to another, and what's added during this process. Make sure that you are aware of http://wiki.apache.org/solr/DataImportHandler#SolrEntityProcessor which queries oth

Re: Performing DIH on predefined list of IDS

2015-02-20 Thread Erick Erickson
Personally, I much prefer indexing from an independent SolrJ client to using DIH when I have to take explicit control of errors & etc. Here's an example: https://lucidworks.com/blog/indexing-with-solrj/ In your example, you seem to be assuming that the Lucene IDs (and here I'm assuming you're not