On 6/27/2018 8:59 PM, solrnoobie wrote:
> One last thing though, will a queue based system work for us? Or are there
> better suitable implementations?
Exactly how you write your indexing software is up to you. There is no
single approach that's the best. Examine your business needs and the
Thank you very much!
This helped a lot in our estimates.
One last thing though, will a queue based system work for us? Or are there
better suitable implementations?
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
On 6/26/2018 8:24 AM, solrnoobie wrote:
> - Each SP call will return 15 result sets.
> - Each document can contain 300-1000 child documents.
> - If the batch size is 1000, the child documents for each can contain
> 300-1000 documents so that will eat up the 4g's allocated to the
> application.
If
Thanks for the tip.
Although we have increased our application's heap to 4g and it is still not
enough.
I guess here are the things we think we did wrong:
- Each SP call will return 15 result sets.
- Each document can contain 300-1000 child documents.
- If the batch size is 1000, the child
On 6/26/2018 12:06 AM, solrnoobie wrote:
We are having errors such as heap space error in our indexing so we decided
to lower the batch size to 50. The problem with this is that sometimes it
really does not help since 1 document can contain 1000 child documents and
it will still have the heap
1. We have 5 nodes and 3 zookeepers (will autoscale if needed)
2. We use java with the help of solrj / spring data for indexing.
3. We see the exception in our application so this is probably our fault and
not solr's so I'm asking what is the best approach for documents with a lot
of child
Would you mind sharing details on
1. the Solr Cloud setup, how may nodes do you have at your disposal and how
many shards do you have setup ?
2. The indexing technology, what are you using? Core java/.net threads ? Or a
system like spark ?
3. Where do you see the exceptions? The indexer process
We are currently having problems in out current production setup in solr.
What we currently have is something like this:
- Solr 6.6.3 (cloud mode)
- 10 threads for indexing
- 900k total documents
- 500 documents per batch
So in each thread, the process will call a stored procedure with a lot
So we have to optimize our current implementation of our indexing. Our
current implementation is to index per batch and each batch will have a
query call from the database that will return multiple result sets and the
application will be responsible in assembling the document based on the
result
11000 documents. So i want to i ndex only new 1000 documents not all
11000. Can anybody suggest for this?Thanks in advance.
--
View this message in context:
http://lucene.472066.n3.nabble.com/Solr-delta-indexing-approach-tp4117068.html
Sent from the Solr - User mailing list archive at Nabble.com.
this message in context:
http://lucene.472066.n3.nabble.com/Solr-delta-indexing-approach-tp4117068.html
Sent from the Solr - User mailing list archive at Nabble.com.
-indexing-approach-tp4117068p4117077.html
Sent from the Solr - User mailing list archive at Nabble.com.
, 13 February 2014 4:45 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr delta indexing approach
Thanks Alex,
Yes my source system maintains the crettion last modificaiton system of each
document.
As per your inputs, can i assume that next time when solr starts indexing, it
scans all
in context:
http://lucene.472066.n3.nabble.com/Solr-delta-indexing-approach-tp4117068p4117077.html
Sent from the Solr - User mailing list archive at Nabble.com.
2014 4:45 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr delta indexing approach
Thanks Alex,
Yes my source system maintains the crettion last modificaiton system of
each document.
As per your inputs, can i assume that next time when solr starts indexing, it
scans all
OOTB available in solr or we need to do it?
Regards.
--
View this message in context:
http://lucene.472066.n3.nabble.com/Solr-delta-indexing-approach-tp4117068p4117087.html
Sent from the Solr - User mailing list archive at Nabble.com.
sense.
-Original Message-
From: Walter Underwood [mailto:wun...@wunderwood.org]
Sent: Thursday, 13 February 2014 5:26 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr delta indexing approach
Why write a Perl script for that?
touch new_timestamp
find . -newer timestamp | script
17 matches
Mail list logo
