Re: load balancer for solr

2016-11-06 Thread Rallavagu
Hi Shawn, Curious, I suppose you have haproxy for Solr in Master/Slave configuration? Thanks. On 11/6/16 9:33 AM, Shawn Heisey wrote: On 11/6/2016 4:08 AM, Mugeesh Husain wrote: Please sugguest load balancer name ? I use haproxy. It is a software load balancer with pretty impressive

Re: indexing - offline

2016-10-20 Thread Rallavagu
Thanks Evan for quick response. On 10/20/16 10:19 AM, Tom Evans wrote: On Thu, Oct 20, 2016 at 5:38 PM, Rallavagu <rallav...@gmail.com> wrote: Solr 5.4.1 cloud with embedded jetty Looking for some ideas around offline indexing where an independent node will be indexed o

indexing - offline

2016-10-20 Thread Rallavagu
Solr 5.4.1 cloud with embedded jetty Looking for some ideas around offline indexing where an independent node will be indexed offline (not in the cloud) and added to the cloud to become leader so other cloud nodes will get replicated. Wonder if this is possible without interrupting the live

Queries to help warm up (mmap)

2016-10-06 Thread Rallavagu
Looking for clues/recommendations to help warm up during startup. Not necessarily Solr caches but mmap as well. I have used some like "q=:[* TO *]" for various fields and it seems to help with mmap population around 40-50%. Is there anything else that could help achieve 90% or more? Thanks.

Re: QuerySenderListener

2016-10-05 Thread Rallavagu
, Rallavagu <rallav...@gmail.com> wrote: Solr Cloud 5.4.1 with embedded jetty, jdk8 At the time of startup it appears that "QuerySenderListener" is run twice and this is causing "firstSearcher" and "newSearcher" to run twice as well. Any clues as to why QuerySenderListener is triggered twice? Thanks.

Re: QuerySenderListener

2016-10-05 Thread Rallavagu
It is a single core. On 10/5/16 6:58 PM, Erick Erickson wrote: How many cores? Is it possible you're seeing these from two different cores? Erick On Wed, Oct 5, 2016 at 11:44 AM, Rallavagu <rallav...@gmail.com> wrote: Solr Cloud 5.4.1 with embedded jetty, jdk8 At the time of s

QuerySenderListener

2016-10-05 Thread Rallavagu
Solr Cloud 5.4.1 with embedded jetty, jdk8 At the time of startup it appears that "QuerySenderListener" is run twice and this is causing "firstSearcher" and "newSearcher" to run twice as well. Any clues as to why QuerySenderListener is triggered twice? Thanks.

disable updates during startup

2016-10-04 Thread Rallavagu
Solr Cloud 5.4.1 with embedded Jetty - jdk 8 Is there a way to disable incoming updates (from leader) during startup until "firstSearcher" queries finished? I am noticing that firstSearcher queries keep on running at the time of startup and node shows up as "Recovering". Thanks

Re: slow updates/searches

2016-09-30 Thread Rallavagu
s increase as the number of words in a phrase increase Well, there's more work to do as the # of words increases, and if you have large slops there's more work yet. Best, Erick On Wed, Sep 28, 2016 at 5:54 PM, Rallavagu <rallav...@gmail.com> wrote: Thanks Erick. I have added queries fo

Re: slow updates/searches

2016-09-28 Thread Rallavagu
sorlcloud/ Best, Erick On Sat, Sep 24, 2016 at 11:35 AM, Rallavagu <rallav...@gmail.com> wrote: On 9/22/16 5:59 AM, Shawn Heisey wrote: On 9/22/2016 5:46 AM, Muhammad Zahid Iqbal wrote: Did you find any solution to slow searches? As far as I know jetty container default configuration is

Re: slow updates/searches

2016-09-24 Thread Rallavagu
On 9/22/16 5:59 AM, Shawn Heisey wrote: On 9/22/2016 5:46 AM, Muhammad Zahid Iqbal wrote: Did you find any solution to slow searches? As far as I know jetty container default configuration is bit slow for large production environment. This might be true for the default configuration that

Re: slow updates/searches

2016-09-19 Thread Rallavagu
, Rallavagu <rallav...@gmail.com> wrote: Comments in line... On 9/16/16 10:15 AM, Erick Erickson wrote: Well, the next thing I'd look at is CPU activity. If you're flooding the system with updates there'll be CPU contention. Monitoring does not suggest any high CPU but as you can see from

Re: slow updates/searches

2016-09-16 Thread Rallavagu
high is right after softCommit which is expected. Wondering what causes update threads wait and if it has any impact on search at all. I had couple of more CPUs added but I still see similar behavior. Thanks. Best, Erick On Fri, Sep 16, 2016 at 9:19 AM, Rallavagu <rallav...@g

Re: slow updates/searches

2016-09-16 Thread Rallavagu
s whether you're _also_ seeing stop-the-world GC pauses. In that case there are a number of JVM options that can be tuned Best, Erick On Fri, Sep 16, 2016 at 8:40 AM, Rallavagu <rallav...@gmail.com> wrote: Solr 5.4.1 with embedded jetty single shard - NRT Looking in logs, noticed th

slow updates/searches

2016-09-16 Thread Rallavagu
Solr 5.4.1 with embedded jetty single shard - NRT Looking in logs, noticed that there are high QTimes for Queries and round same time high response times for updates. These are not during "commit" or "softCommit" but when client application is sending updates. Wondering how updates could

Re: How to enable JMX to monitor Jetty

2016-09-12 Thread Rallavagu
I have modified modules/http.mod as following (for solr 5.4.1, Jetty 9). As you can see I have referred jetty-jmx.xml. # # Jetty HTTP Connector # [depend] server [xml] etc/jetty-http.xml etc/jetty-jmx.xml On 5/21/16 3:59 AM, Georg Sorst wrote: Hi list, how do I correctly enable JMX in

Re: ConcurrentUpdateSolrClient threads

2016-09-12 Thread Rallavagu
Any takers? On 9/9/16 9:03 AM, Rallavagu wrote: All, Running Solr 5.4.1 with embedded Jetty with frequent updates coming in and softCommit is set to 10 min. What I am noticing is occasional "slow" updates (takes 8 sec to 15 sec sometimes) and about the same time slow QTimes. Upon inv

ConcurrentUpdateSolrClient threads

2016-09-09 Thread Rallavagu
All, Running Solr 5.4.1 with embedded Jetty with frequent updates coming in and softCommit is set to 10 min. What I am noticing is occasional "slow" updates (takes 8 sec to 15 sec sometimes) and about the same time slow QTimes. Upon investigating, it appears that

Re: Default Field Cache

2016-09-01 Thread Rallavagu
gt;(); args.put(NAME, "fieldValueCache"); args.put("size", "1"); args.put("initialSize", "10"); args.put("showItems", "-1"); conf = new CacheConfig(FastLRUCache.class, args, null); } fieldValueCacheConfig = conf; Cheers On

Re: Default Field Cache

2016-08-31 Thread Rallavagu
Yeo wrote: If I didn't get your question wrong, what you have listed is already the default configuration that comes with your version of Solr. Regards, Edwin On 30 August 2016 at 07:49, Rallavagu <rallav...@gmail.com> wrote: Solr 5.4.1 Wondering what is the defau

Default Field Cache

2016-08-29 Thread Rallavagu
Solr 5.4.1 Wondering what is the default configuration for "fieldValueCache".

Re: Solr embedded jetty jstack

2016-08-29 Thread Rallavagu
worked. On 8/29/16 11:31 AM, Rallavagu wrote: I have run into a strange issue where "jstack -l " does not work. I have tried this as the user that solr (5.4.1) is running as. I get following error. $ jstack -l 24064 24064: Unable to open socket file: target process not responding or HotSpot

Solr embedded jetty jstack

2016-08-29 Thread Rallavagu
I have run into a strange issue where "jstack -l " does not work. I have tried this as the user that solr (5.4.1) is running as. I get following error. $ jstack -l 24064 24064: Unable to open socket file: target process not responding or HotSpot VM not loaded The -F option can be used when

Re: solr.NRTCachingDirectoryFactory

2016-08-26 Thread Rallavagu
is enough. If the only small ratio of documents is updated and a bottleneck is filterCache you can experiment with segmened filters which suite more for NRT. http://blog-archive.griddynamics.com/2014/01/segmented-filter-cache-in-solr.html On Fri, Aug 26, 2016 at 2:56 AM, Rallavagu <rallav...@gmail.

Re: solr.NRTCachingDirectoryFactory

2016-08-25 Thread Rallavagu
latency yet, tell them they don't know what they're talking about when they insist on the NRT interval being a few seconds ;). Best, Erick On Tue, Jul 26, 2016 at 7:20 AM, Rallavagu <rallav...@gmail.com> wrote: On 7/26/16 5:46 AM, Shawn Heisey wrote: On 7/22/2016 10:15 AM, Rallavagu

Re: solr.NRTCachingDirectoryFactory

2016-07-26 Thread Rallavagu
On 7/26/16 5:46 AM, Shawn Heisey wrote: On 7/22/2016 10:15 AM, Rallavagu wrote: As Erick indicated, these settings are incompatible with Near Real Time updates. With those settings, every time you commit and create a new searcher, Solr will execute up to 1000 queries

Re: solr.NRTCachingDirectoryFactory

2016-07-22 Thread Rallavagu
ng? Pivot Faceting? 'cause from everything you've said so far it's surprising that you're seeing queries take this long, something doesn't feel right but what it is I don't have a clue. Thanks Best, Erick On Fri, Jul 22, 2016 at 9:15 AM, Rallavagu <rallav...@gmail.com> wrote: On

Re: solr.NRTCachingDirectoryFactory

2016-07-22 Thread Rallavagu
Also, here is the link to screenshot. https://dl.dropboxusercontent.com/u/39813705/Screen%20Shot%202016-07-22%20at%2010.40.21%20AM.png Thanks On 7/21/16 11:22 PM, Shawn Heisey wrote: On 7/21/2016 11:25 PM, Rallavagu wrote: There is no other software running on the system and it is completely

Re: solr.NRTCachingDirectoryFactory

2016-07-22 Thread Rallavagu
4192 6004 2996 S 0.0 0.0 15:08.87 nslcd On 7/21/16 11:22 PM, Shawn Heisey wrote: On 7/21/2016 11:25 PM, Rallavagu wrote: There is no other software running on the system and it is completely dedicated to Solr. It is running on Linux. Here is the full version. Linux version 3.8.13-55.1.6.e

Re: solr.NRTCachingDirectoryFactory

2016-07-22 Thread Rallavagu
t; wrote: On 7/21/2016 11:25 PM, Rallavagu wrote: There is no other software running on the system and it is completely dedicated to Solr. It is running on Linux. Here is the full version. Linux version 3.8.13-55.1.6.el7uek.x86_64 (mockbu...@ca-build56.us.oracle.com) (gcc version 4.8.3 20140911 (Red

Re: solr.NRTCachingDirectoryFactory

2016-07-21 Thread Rallavagu
On 7/21/16 9:16 PM, Shawn Heisey wrote: On 7/21/2016 9:37 AM, Rallavagu wrote: I suspect swapping as well. But, for my understanding - are the index files from disk memory mapped automatically at the startup time? They are *mapped* at startup time, but they are not *read* at startup

Re: solr.NRTCachingDirectoryFactory

2016-07-21 Thread Rallavagu
used to due to faceting (debug=true). Since we have adjusted indexing and facet times are improved but basic query QTime is still high so wondering where can I look? Is there a way to debug (instrument) a query on Solr node? FWIW, Erick On Thu, Jul 21, 2016 at 8:18 AM, Rallavagu <rall

solr.NRTCachingDirectoryFactory

2016-07-21 Thread Rallavagu
Solr 5.4.1 with embedded jetty with cloud enabled We have a Solr deployment (approximately 3 million documents) with both write and search operations happening. We have a requirement to have updates available immediately (NRT). Configured with default "solr.NRTCachingDirectoryFactory" for

Re: Document Cache

2016-03-19 Thread Rallavagu
;false". What I am seeing is (from heap dump due to OutOfMemory error) that the LRUCache pertaining "Document Cache" occupies around 85% of available heap and that is causing OOM errors. So, trying to understand the behavior to address the OOM issues. Thanks Best, Erick

Solr5 Optimize

2016-03-19 Thread Rallavagu
All, Solr 5.4 with emdbedded Jetty (4G heap) Trying to understand behavior of "optimize" operation if not run explicitly. What is the frequency at which this operation is run, what are the storage requirements and how do we schedule it? Any comments/pointers would greatly help. Thanks in

Re: Solr5 Optimize

2016-03-19 Thread Rallavagu
ds is low so it shouldn't have much impact on query perf. It's theoretically possible that the background merge will merge down to one segment, so you still need at least as much free space on your disk and your index occupies. Best, Erick On Wed, Mar 16, 2016 at 10:07 AM, Rallavagu <rallav...

Re: Document Cache

2016-03-19 Thread Rallavagu
size of 1000 entries should take around 20MB (assuming single shard) Thanks, Emir On 18.03.2016 17:02, Rallavagu wrote: On 3/18/16 8:56 AM, Emir Arnautovic wrote: Problem starts with autowarmCount="5000" - that executes 5000 queries when new searcher is created and as queries ar

Re: Document Cache

2016-03-19 Thread Rallavagu
documents with all fields. I could not reproduce OOM. I understand that I need to reduce cache sizes but wondering what conditions could have caused OOM so I can keep a watch. Thanks Thanks, Emir On 18.03.2016 15:43, Rallavagu wrote: Thanks for the recommendations Shawn. Those are the

Document Cache

2016-03-19 Thread Rallavagu
Solr 5.4 embedded Jetty Is it the right assumption that whenever a document that is returned as a response to a query is cached in "Document Cache"? Essentially, if I request for any entry like /select?q=id: will it be cached in "Document Cache"? If yes, what is the TTL? Thanks in advance

Re: Document Cache

2016-03-19 Thread Rallavagu
the searcher due to long running query that might be potentially causing OOM? Was trying to reproduce but could not so far. Here is the filter cache config autowarmCount="1000"/> Query Results cache autowarmCount="5000"/> On 3/18/16 7:31 AM, Shawn Heisey wrote: On 3/

Re: Solr5 Optimize

2016-03-19 Thread Rallavagu
se Thanks. The third video down here: http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html is Mikes visualization of the automatic merging process. Best, Erick On Wed, Mar 16, 2016 at 9:40 AM, Rallavagu <rallav...@gmail.com> wrote: All, Solr 5.4 wi

Re: Document Cache

2016-03-18 Thread Rallavagu
tes. It seems that it is either configured to be huge or you have big documents and retrieving all fields or dont have lazy field loading set to true. Can you please share your document cache config and heap settings. Thanks, Emir On 17.03.2016 22:24, Rallavagu wrote: comments in line... On 3/17/16 2

Re: SolrCloud breaks and does not recover

2015-11-03 Thread Rallavagu
One another item to look into is to increase the zookeeper timeout in solr.xml of Solr. This would help with timeout caused by long GC pauses. On 11/3/15 9:12 AM, Björn Häuser wrote: Hi, thank you for your answer. 1> No OOM hit, the log does not contain any hind of that. Also solr wasn't

growth of tlog

2015-10-30 Thread Rallavagu
4.10.4 solr cloud, 3 zk quorum, jdk 8 autocommit: 15 sec, softcommit: 2 min Under heavy indexing load with above settings, i have seen tlog growing (into GB). After the updates stopped coming in, it settles down and takes a while to recover before cloud becomes "green". With 15 second

Re: growth of tlog

2015-10-30 Thread Rallavagu
ore idea, as per the following document I am testing "Index heavy, Query heavy" situation. https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ Thanks Best, Erick On Fri, Oct 30, 2015 at 8:28 AM, Rallavagu <rallav...@gmail.com>

Solr for Pictures

2015-10-29 Thread Rallavagu
In general, is there a built-in data handler to index pictures (essentially, EXIF and other data embedded in an image)? If not, what is the best practice to do so? Thanks.

Re: Solr for Pictures

2015-10-29 Thread Rallavagu
nfluence/display/solr/Uploading+Data+with+Solr+Cell+using+Apache+Tika> On Oct 29, 2015, at 1:47 PM, Rallavagu <rallav...@gmail.com <mailto:rallav...@gmail.com>> wrote: In general, is there a built-in data handler to index pictures (essentially, EXIF and other data embedded in an image

Commit Error

2015-10-28 Thread Rallavagu
Solr 4.6.1, cloud Seeing following commit errors. [commitScheduler-19-thread-1] ERROR org.apache.solr.update.CommitTracker – auto commit error...:java.lang.IllegalStateException: this writer hit an OutOfMemoryError; cannot commit at

Re: Commit Error

2015-10-28 Thread Rallavagu
(writing to disk). On 10/28/15 3:57 PM, Shawn Heisey wrote: On 10/28/2015 2:06 PM, Rallavagu wrote: Solr 4.6.1, cloud Seeing following commit errors. [commitScheduler-19-thread-1] ERROR org.apache.solr.update.CommitTracker – auto commit error...:java.lang.IllegalStateException: this writer hit

Re: Commit Error

2015-10-28 Thread Rallavagu
Also, is this thread that went OOM and what could cause it? The heap was doing fine and server was live and running. On 10/28/15 3:57 PM, Shawn Heisey wrote: On 10/28/2015 2:06 PM, Rallavagu wrote: Solr 4.6.1, cloud Seeing following commit errors. [commitScheduler-19-thread-1] ERROR

Re: Commit Error

2015-10-28 Thread Rallavagu
On 10/28/15 5:41 PM, Shawn Heisey wrote: On 10/28/2015 5:11 PM, Rallavagu wrote: Seeing very high CPU during this time and very high warmup times. During this time, there were plenty of these errors logged. So, trying to find out possible causes for this to occur. Could it be disk I/O issues

Re: Solr hard commit

2015-10-27 Thread Rallavagu
d (closed actually) after the last commit are _not_ read at all until the next searcher is opened via another commit. Nothing is done with these new segments before the new searcher is opened which you control with your commit strategy. I see. Thanks for the insight. Best, Erick On Mon, Oct 26,

Re: Using books.json in solr

2015-10-27 Thread Rallavagu
Could you please share your query? You could use "wt=json" query parameter to receive JSON formatted results if that is what you are looking for. On 10/27/15 10:44 AM, Salonee Rege wrote: Hello, We are trying to query the books.json that we have posted to solr. But when we try to

Re: Solr hard commit

2015-10-27 Thread Rallavagu
Is it related to this config?

Solr hard commit

2015-10-26 Thread Rallavagu
All, Are memory mapped files (mmap) flushed to disk during "hard commit"? If yes, should we disable OS level (Linux for example) memory mapped flush? I am referring to following for mmap files for Lucene/Solr http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html Linux

Re: Solr hard commit

2015-10-26 Thread Rallavagu
, but these have nothing to do with MMapDirectory. So the question is really moot ;) Best, Erick On Mon, Oct 26, 2015 at 5:47 PM, Rallavagu <rallav...@gmail.com> wrote: All, Are memory mapped files (mmap) flushed to disk during "hard commit"? If yes, should we disable OS level (L

Re: locks and high CPU

2015-10-22 Thread Rallavagu Kon
help if > you're not using > them. > > Best, > Erick > >> On Thu, Oct 22, 2015 at 9:43 AM, Rallavagu <rallav...@gmail.com> wrote: >> Solr 4.6.1 cloud >> >> Looking into thread dump 4-5 threads causing cpu to go very high and causing >> issues. These

Re: locks and high CPU

2015-10-22 Thread Rallavagu
there is for 4.6. Best, Erick On Thu, Oct 22, 2015 at 10:48 AM, Rallavagu Kon <rallav...@gmail.com> wrote: Erick, Indexing happening via Solr cloud server. This thread was from the leader. Some followers show symptom of high cpu during this time. You think this is from locking

locks and high CPU

2015-10-22 Thread Rallavagu
Solr 4.6.1 cloud Looking into thread dump 4-5 threads causing cpu to go very high and causing issues. These are tomcat's http threads and are locking. Can anybody help me understand what is going on here? I see that incoming connections coming in for updates and they are being passed on to

coreZkRegister thread

2015-10-20 Thread Rallavagu
Solr 4.6.1, 4 node cloud with 3 zk I see the following thread as blocked. Could somebody please help me understand what is going on here and how will it impact solr cloud? All four of these threads blocked. Thanks. "coreZkRegister-1-thread-1" id=74 idx=0x108 tid=32162 prio=5 alive, parked,

Re: Help me read Thread

2015-10-16 Thread Rallavagu
est, Erick On Tue, Oct 13, 2015 at 9:08 AM, Rallavagu <rallav...@gmail.com> wrote: Also, we have increased number of connections per host from default (20) to 100 for http thread pool to communicate with other nodes. Could this have caused the issues as it can now spin many threads to send upd

Help me read Thread

2015-10-13 Thread Rallavagu
Please help me understand what is going on with this thread. Solr 4.6.1, single shard, 4 node cluster, 3 node zk. Running on tomcat with 500 threads. There are 47 threads overall and designated leader becomes unresponsive though shows "green" from cloud perspective. This is causing issues.

Re: Help me read Thread

2015-10-13 Thread Rallavagu
exing load? There were some inefficiencies that caused followers to work a lot harder than the leader, but the leader had to spin off a bunch of threads to send update to followers. That's fixed int he 5.2 release. Best, Erick On Tue, Oct 13, 2015 at 8:40 AM, Rallavagu <rallav...@gmail.com> wro

Re: Help me read Thread

2015-10-13 Thread Rallavagu
indexing load? There were some inefficiencies that caused followers to work a lot harder than the leader, but the leader had to spin off a bunch of threads to send update to followers. That's fixed int he 5.2 release. Best, Erick On Tue, Oct 13, 2015 at 8:40 AM, Rallavagu <rallav...@gmail.com>

Re: Help me read Thread

2015-10-13 Thread Rallavagu
s required that you're hitting this. Are you unable to reduce the indexing rate? Best, Erick On Tue, Oct 13, 2015 at 9:08 AM, Rallavagu <rallav...@gmail.com> wrote: Also, we have increased number of connections per host from default (20) to 100 for http thread pool to communicate with ot

solr cloud recovery and search

2015-10-13 Thread Rallavagu
It appears that when a node that is in "recovery" mode queried it would defer the query to leader instead of serving from locally. Is this the expected behavior? Thanks.

Re: solr cloud recovery and search

2015-10-13 Thread Rallavagu
Great. Thanks Erick. On 10/13/15 5:39 PM, Erick Erickson wrote: More than expected, guaranteed. As long as at least one replica in a shard is active, all queries should succeed. Maybe more slowly, but they should succeed. Best, Erick On Tue, Oct 13, 2015 at 4:25 PM, Rallavagu <ral

Re: tlog replay

2015-10-08 Thread Rallavagu
sizes if there are very long hard commit intervals, but I don't see how that interval would be different on the leader and follower. So color me puzzled. Best, Erick On Wed, Oct 7, 2015 at 8:09 PM, Rallavagu <rallav...@gmail.com> wrote: Thanks Erick. Eventually, followers caught up but t

Re: tlog replay

2015-10-08 Thread Rallavagu
g. Best, Erick On Thu, Oct 8, 2015 at 12:23 PM, Rallavagu <rallav...@gmail.com> wrote: As a follow up. Eventually the tlog file is disappeared (could not track the time it took to clear out completely). However, following messages were noticed in follower's log. 5120638 [recoveryExecu

Re: tlog replay

2015-10-07 Thread Rallavagu
t 7, 2015 at 7:39 PM, Rallavagu <rallav...@gmail.com> wrote: Solr 4.6.1, single shard, 4 node cloud, 3 node zk Like to understand the behavior better when large number of updates happen on leader and it generates huge tlog (14G sometimes in my case) on other nodes. At the same time leader's tl

tlog replay

2015-10-07 Thread Rallavagu
Solr 4.6.1, single shard, 4 node cloud, 3 node zk Like to understand the behavior better when large number of updates happen on leader and it generates huge tlog (14G sometimes in my case) on other nodes. At the same time leader's tlog is few KB. So, what is the rate at which the changes from

Re: Recovery Thread Blocked

2015-10-06 Thread Rallavagu
the pace of SolrCloud, you are dealing with something fairly ancient and so it will be harder to find help with older issues most likely. - Mark On Mon, Oct 5, 2015 at 12:46 PM Rallavagu <rallav...@gmail.com> wrote: Any takers on this? Any kinda clue would help. Thanks. On 10/4/15 10

Re: Recovery Thread Blocked

2015-10-06 Thread Rallavagu
cribe what is happening with the heap. - Mark On Tue, Oct 6, 2015 at 4:04 PM Rallavagu <rallav...@gmail.com> wrote: Mark - currently 5.3 is being evaluated for upgrade purposes and hopefully get there sooner. Meanwhile, following exception is noted from logs during updates ERROR org.apache.solr.upda

Re: Recovery Thread Blocked

2015-10-06 Thread Rallavagu
It is java thread though. Does it need increasing OS level threads? On 10/6/15 6:21 PM, Mark Miller wrote: If it's a thread and you have plenty of RAM and the heap is fine, have you checked raising OS thread limits? - Mark On Tue, Oct 6, 2015 at 4:54 PM Rallavagu <rallav...@gmail.com>

Re: Recovery Thread Blocked

2015-10-05 Thread Rallavagu
Any takers on this? Any kinda clue would help. Thanks. On 10/4/15 10:14 AM, Rallavagu wrote: As there were no responses so far, I assume that this is not a very common issue that folks come across. So, I went into source (4.6.1) to see if I can figure out what could be the cause. The thread

Re: Recovery Thread Blocked

2015-10-04 Thread Rallavagu
"recoveryStrat.join()" is where things are holding up. I wonder why/how cancelRecovery would take time so around 870 threads would be waiting on. Is it possible that ZK is not responding or something else like Operating System resources could cause this? Thanks. On 10/2/15 4:

Re: Zk and Solr Cloud

2015-10-02 Thread Rallavagu
if not what are those work entries? Thanks On 10/1/15 10:58 PM, Shawn Heisey wrote: On 10/1/2015 1:26 PM, Rallavagu wrote: Solr 4.6.1 single shard with 4 nodes. Zookeeper 3.4.5 ensemble of 3. See following errors in ZK and Solr and they are connected. When I see the following error in Zookeeper

Re: Zk and Solr Cloud

2015-10-02 Thread Rallavagu
Thanks for the insight into this Erick. Thanks. On 10/2/15 8:58 AM, Erick Erickson wrote: Rallavagu: Absent nodes going up and down or otherwise changing state, Zookeeper isn't involved in the normal operations of Solr (adding docs, querying, all that). That said, things that change the state

Recovery Thread Blocked

2015-10-02 Thread Rallavagu
Solr 4.6.1 on Tomcat 7, single shard 4 node cloud with 3 node zookeeper During updates, some nodes are going very high cpu and becomes unavailable. The thread dump shows the following thread is blocked 870 threads which explains high CPU. Any clues on where to look? "Thread-56848" id=79207

Re: Recovery Thread Blocked

2015-10-02 Thread Rallavagu
/vm/RNI.c2java(J)V(Native Method) On 10/2/15 4:12 PM, Rallavagu wrote: Solr 4.6.1 on Tomcat 7, single shard 4 node cloud with 3 node zookeeper During updates, some nodes are going very high cpu and becomes unavailable. The thread dump shows the following thread is blocked 870 threads which ex

PoolingClientConnectionManager

2015-10-01 Thread Rallavagu
Solr 4.6.1, single Shard, cloud with 4 nodes Solr is running on Tomcat configured with 200 threads for thread pool. As Solr uses "org.apache.http.impl.conn.PoolingClientConnectionManager" for replication, my question is does Solr threads use connections from tomcat thread pool or they create

Re: PoolingClientConnectionManager

2015-10-01 Thread Rallavagu
Thanks for the response Andrea. Assuming that Solr has it's own thread pool, it appears that "PoolingClientConnectionManager" has a maximum 20 threads per host as default. Is there a way to changes this increase to handle heavy update traffic? Thanks. On 10/1/15 11:05 AM, Andrea Gazzarini

Re: PoolingClientConnectionManager

2015-10-01 Thread Rallavagu
Thanks Shawn. This is good data. On 10/1/15 11:43 AM, Shawn Heisey wrote: On 10/1/2015 11:50 AM, Rallavagu wrote: Solr 4.6.1, single Shard, cloud with 4 nodes Solr is running on Tomcat configured with 200 threads for thread pool. As Solr uses

Re: PoolingClientConnectionManager

2015-10-01 Thread Rallavagu
Awesome. This is what I was looking for. Will try these. Thanks. On 10/1/15 1:31 PM, Shawn Heisey wrote: On 10/1/2015 12:39 PM, Rallavagu wrote: Thanks for the response Andrea. Assuming that Solr has it's own thread pool, it appears that "PoolingClientConnectionManager" has a

Zk and Solr Cloud

2015-10-01 Thread Rallavagu
Solr 4.6.1 single shard with 4 nodes. Zookeeper 3.4.5 ensemble of 3. See following errors in ZK and Solr and they are connected. When I see the following error in Zookeeper, unexpected error, closing socket connection and attempting reconnect java.io.IOException: Packet len11823809 is out of

Re: Solr 4.6.1 Cloud Stops Replication

2015-08-31 Thread Rallavagu
ck On Mon, Aug 24, 2015 at 2:47 AM, Rallavagu <rallav...@gmail.com> wrote: As a follow up, the default is set to "NRTCachingDirectoryFactory" for DirectoryFactory but not MMapDirectory. It is mentioned that NRTCachingDirectoryFactory "caches small files in memory for better

Re: Solr 4.6.1 Cloud Stops Replication

2015-08-24 Thread Rallavagu
linear throughput with increasing numbers of shards this way. Best, Erick On Tue, Aug 18, 2015 at 9:03 AM, Rallavagu rallav...@gmail.com wrote: Thanks Shawn. All participating cloud nodes are running Tomcat and as you suggested will review the number of threads and increase them as needed

Re: GC parameters tuning for core of 140M docs on 50G of heap memory

2015-08-24 Thread Rallavagu
One other item to check is non heap memory usage. This can be monitored from admin page. On 8/23/15 11:48 PM, Pavel Hladik wrote: Hi, we have a Solr 5.2.1 with 9 cores and one of them has 140M docs. Can you please recommend tuning of those GC parameters? The performance is not a issue,

Re: Solr 4.6.1 Cloud Stops Replication

2015-08-18 Thread Rallavagu
/17/2015 10:53 PM, Rallavagu wrote: Also, I have noticed that the memory consumption goes very high. For instance, each node is configured with 48G memory while java heap is configured with 12G. The available physical memory is consumed almost 46G and the heap size is well within the limits

Re: Solr 4.6.1 Cloud Stops Replication

2015-08-18 Thread Rallavagu
/18/2015 8:18 AM, Rallavagu wrote: Thanks for the response. Does this cache behavior influence the delay in catching up with cloud? How can we explain solr cloud replication and what are the option to monitor and take proactive action (such as initializing, pausing etc) if needed? I don't know

Re: Solr 4.6.1 Cloud Stops Replication

2015-08-18 Thread Rallavagu
packets to each leader that consist of only the docs the belong on that shard. You can getnearly linear throughput with increasing numbers of shards this way. Best, Erick On Tue, Aug 18, 2015 at 9:03 AM, Rallavagu rallav...@gmail.com wrote: Thanks Shawn. All participating cloud nodes are running

Solr 4.6.1 Cloud Stops Replication

2015-08-17 Thread Rallavagu
Hello, Have 4 nodes participating solr cloud. After indexing about 2 mil documents, only two nodes are Active (green) while other two are shown as down. How can I initialize the replication from leader so other two nodes would receive updates? Thanks

Re: Solr 4.6.1 Cloud Stops Replication

2015-08-17 Thread Rallavagu
By the time the last email was sent, other node also caught up. Makes me wonder what happened and how does this work. Thanks On 8/17/15 9:53 PM, Rallavagu wrote: response inline.. On 8/17/15 8:40 PM, Erick Erickson wrote: Is this 4 shards? Two shards each with a leader and follower? Details

Re: Solr 4.6.1 Cloud Stops Replication

2015-08-17 Thread Rallavagu
to start with. Hope the provided information would help. Thanks Best, Erick On Mon, Aug 17, 2015 at 6:19 PM, Rallavagu rallav...@gmail.com wrote: Hello, Have 4 nodes participating solr cloud. After indexing about 2 mil documents, only two nodes are Active (green) while other two are shown

Collections Design

2014-04-10 Thread Rallavagu
All, What is the best practice or guideline towards considering multiple collections particularly in the solr cloud env? Thanks Srikanth

No route to host

2014-04-09 Thread Rallavagu
All, I see the following error in the log file. The host that it is trying to find is itself. Wondering if anybody experienced this before or any other info would helpful. Thanks. 709703139 [http-bio-8080-exec-43] ERROR org.apache.solr.update.SolrCmdDistributor –

Re: No route to host

2014-04-09 Thread Rallavagu
error. Thanks, Greg On Apr 9, 2014, at 3:28 PM, Rallavagu rallav...@gmail.com wrote: All, I see the following error in the log file. The host that it is trying to find is itself. Wondering if anybody experienced this before or any other info would helpful. Thanks. 709703139 [http-bio-8080-exec

Re: No route to host

2014-04-09 Thread Rallavagu
Sorry. I should have mentioned earlier. I have removed the original host name on purpose. Thanks. On 4/9/14, 1:42 PM, Siegfried Goeschl wrote: Hi folks, the URL looks wrong (misconfigured) http://host:8080/solr/collection1 Cheers, Siegfried Goeschl On 09 Apr 2014, at 14:28, Rallavagu

Re: Indexing huge data

2014-03-08 Thread Rallavagu
and incremental indexing as well. Thanks, Kranti K. Parisa http://www.linkedin.com/in/krantiparisa On Thu, Mar 6, 2014 at 9:57 AM, Rallavagu rallav...@gmail.com wrote: Yeah. I have thought about spitting out JSON and run it against Solr using parallel Http threads separately. Thanks. On 3/5/14, 6:46

Re: Indexing huge data

2014-03-06 Thread Rallavagu
...@basetechnology.com wrote: Make sure you're not doing a commit on each individual document add. Commit every few minutes or every few hundred or few thousand documents is sufficient. You can set up auto commit in solrconfig.xml. -- Jack Krupansky -Original Message- From: Rallavagu Sent

  1   2   >