Hi Erick,
Thanks a lot for your help. I will go through MongoDB.
On Mon, Aug 10, 2015 at 9:14 PM Erick Erickson
wrote:
> bq: I changed
> *2*
> to *100*. And apply simultaneous
> searching using 100 workers.
>
> Do not do this. This has nothing to do with the number of searcher
bq: I changed
*2*
to *100*. And apply simultaneous
searching using 100 workers.
Do not do this. This has nothing to do with the number of searcher
threads. And with
your update rate, especially if you continue to insist on adding
commit=true to every
update request, this will explode your memory
Hi,
I used solr 5.2.1 version. It is fast, I think. But again, I am stuck
on concurrent searching and threading. I changed
*2*
to *100*. And apply simultaneous
searching using 100 workers. It works fast but not upto the mark.
It increases searching from 1.5 to 0.5 seconds. But If I run only
On 8/7/2015 1:15 PM, Nitin Solanki wrote:
> I wrote a python script for indexing and using
> urllib and urllib2 for indexing data via http..
There are a number of Solr python clients. Using a client makes your
code much easier to write and understand.
https://wiki.apache.org/solr/SolPython
I ha
If you are using Python, then you can use urllib2, or "requests" which
is reportedly better, or better still something like pysolr, which makes
life simpler.
Here's a Pull Request that makes pysolr Zookeeper aware, which'll help
if you are using SolrCloud. I hope one day they will merge it:
https
Thanks Erick for your suggestion. I will remove commit = true and use solr
5.2 and then get back to you again. For further help. Thanks.
On Sat, Aug 8, 2015 at 4:07 AM Erick Erickson
wrote:
> bq: So, How much minimum concurrent threads should I run?
>
> I really can't answer that in the abstract
bq: So, How much minimum concurrent threads should I run?
I really can't answer that in the abstract, you'll simply have to
test.
I'd prefer SolrJ to post.jar. If you're not going to SolrJ, I'd imagine that
moving from Python to post.jar isn't all that useful.
But before you do anything, see wha
Hi Erick,
posting files to Solr via curl =>
Rather than posting files via curl. Which is better SolrJ or post.jar... I
don't use both things. I wrote a python script for indexing and using
urllib and urllib2 for indexing data via http.. I don't have any option to
use SolrJ Right no
bq: How much limitations does Solr has related to indexing and searching
simultaneously? It means that how many simultaneously calls, I made for
searching and indexing once?
None a-priori. It all depends on the hardware you're throwing at it. Obviously
indexing with 100 threads is going to eat up
Hi Erick,
You said that soft commit should be more than 3000 ms.
Actually, I need Real time searching and that's why I need soft commit fast.
commit=true => I made commit=true because , It reduces by indexed data size
from 1.5GB to 500MB on* each shard*. When I did commit=false then,
Hi, Upayavira
RAM = 28GB
CPU = 4 processes..
On Fri, Aug 7, 2015 at 8:53 PM Upayavira wrote:
> How many CPUs do you have? 100 concurrent indexing calls seems like
> rather a lot. You're gonna end up doing a lot of context switching,
> hence degraded performance. Dunno what othe
Your soft commit time of 3 seconds is quite aggressive,
I'd lengthen it to as long as possible.
Ugh, looked at your query more closely. Adding commit=true to every update
request is horrible performance wise. Let your autocommit process
handle the commits is the first thing I'd do. Second, I'd try
How many CPUs do you have? 100 concurrent indexing calls seems like
rather a lot. You're gonna end up doing a lot of context switching,
hence degraded performance. Dunno what others would say, but I'd aim for
approx one indexing thread per CPU.
Upayavira
On Fri, Aug 7, 2015, at 02:58 PM, Nitin So
Hello Everyone,
I have indexed 16 million documents in Solr
Cloud. Created 4 nodes and 8 shards with single replica.
I am trying to make concurrent indexing and searching on those indexed
documents. Trying to make 100 concurrent indexing calls along with 100
concurrent sea
14 matches
Mail list logo
