ava:61(Compiled
Code))
4XESTACKTRACEat
java/lang/Thread.run(Thread.java:804(Compiled Code))
3XMTHREADINFO3 No native callstack available for this thread
From: Shawn Heisey
To: solr-user@lucene.apache.org
Date: 01/09/2015 06:09 PM
Subject:Re: Solr cloud hangs,
On 9/1/2015 12:53 AM, Arnon Yogev wrote:
> We have a Solr cloud (4.7) consisting of 5 servers.
> At some point we noticed that one of the servers had a very high CPU and
> was not responding. A few minutes later, the other 4 servers were
> responding very slowly. A restart was required.
> Looking a
We have a Solr cloud (4.7) consisting of 5 servers.
At some point we noticed that one of the servers had a very high CPU and
was not responding. A few minutes later, the other 4 servers were
responding very slowly. A restart was required.
Looking at the Solr logs, we mainly saw symptoms, i.e. error
There are also many errors in solr log like that one:
org.apache.solr.update.StreamingSolrServers$1; error
org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for
connection from pool
at
org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClient
Hi,
Here is the whole stack trace: https://gist.github.com/anonymous/9056783
--
Pawel
On Mon, Feb 17, 2014 at 4:53 PM, Mark Miller wrote:
> Can you share the full stack trace dump?
>
> - Mark
>
> http://about.me/markrmiller
>
> On Feb 17, 2014, at 7:07 AM, Pawel Rog wrote:
>
> > Hi,
> > I have
Can you share the full stack trace dump?
- Mark
http://about.me/markrmiller
On Feb 17, 2014, at 7:07 AM, Pawel Rog wrote:
> Hi,
> I have quite annoying problem with Solr cloud. I have a cluster with 8
> shards and with 2 replicas in each. (Solr 4.6.1)
> After some time cluster doesn't respond
Hi,
I have quite annoying problem with Solr cloud. I have a cluster with 8
shards and with 2 replicas in each. (Solr 4.6.1)
After some time cluster doesn't respond to any update requests. Restarting
the cluster nodes doesn't help.
There are a lot of such stack traces (waiting for very long time):
it
fixes things.
Jessica
--
View this message in context:
http://lucene.472066.n3.nabble.com/Solr-Cloud-hangs-when-replicating-updates-tp4088083p4095061.html
Sent from the Solr - User mailing list archive at Nabble.com.
Thanks a ton Mark. I have tried SOLR-4816 and it didn't help. But I will
try Mark's patch next week, and see what happens.
-Kevin
On Thu, Sep 5, 2013 at 4:46 AM, Erick Erickson wrote:
> If you run into this again, try a jstack trace. You should see
> evidence of being stuck in SolrCmdDistributo
If you run into this again, try a jstack trace. You should see
evidence of being stuck in SolrCmdDistributor on a variable
called "semaphore"... On current 4x this is around line 420.
If you're using SolrJ, then SOLR-4816 is another thing to try.
But Mark's patch would be best of all to test, If
It would be great if you could give this patch a try:
http://pastebin.com/raw.php?i=aaRWwSGP
- Mark
On Wed, Sep 4, 2013 at 8:31 AM, Kevin Osborn wrote:
> Thanks. If there is anything I can do to help you resolve this issue, let
> me know.
>
> -Kevin
>
>
> On Wed, Sep 4, 2013 at 7:51 AM, Mark M
Ill look at fixing the root issue for 4.5. I've been putting it off for way to
long.
Mark
Sent from my iPhone
On Sep 3, 2013, at 2:15 PM, Kevin Osborn wrote:
> I was having problems updating SolrCloud with a large batch of records. The
> records are coming in bursts with lulls between updat
Thanks. If there is anything I can do to help you resolve this issue, let
me know.
-Kevin
On Wed, Sep 4, 2013 at 7:51 AM, Mark Miller wrote:
> Ill look at fixing the root issue for 4.5. I've been putting it off for
> way to long.
>
> Mark
>
> Sent from my iPhone
>
> On Sep 3, 2013, at 2:15 PM,
Kevin,
Take a look at
http://lucene.472066.n3.nabble.com/updating-docs-in-solr-cloud-hangs-td4067388.html
and https://issues.apache.org/jira/browse/SOLR-4816. I had the same issue that
you're reporting for a while then I applied the patch from SOLR-4816 to my
clients and the problems
I was having problems updating SolrCloud with a large batch of records. The
records are coming in bursts with lulls between updates.
At first, I just tried large updates of 100,000 records at a time.
Eventually, this caused Solr to hang. When hung, I can still query Solr.
But I cannot do any delet
Thanks, Erick that's exactly the clarification/confirmation I was looking for!
Greg
more of a workaround than a true fix?
>
> I'm entirely willing to apply this patch to all of the code I've got that
> talks to my solr servers and try it out but I'm reluctant to because this
> looks like a client-side fix to a server-side issue.
>
> Thanks,
> G
7;ve got that
talks to my solr servers and try it out but I'm reluctant to because this
looks like a client-side fix to a server-side issue.
Thanks,
Greg
--
View this message in context:
http://lucene.472066.n3.nabble.com/updating-docs-in-solr-cloud-hangs-tp4067388p4086160.html
Sent fro
>
>
>
>
>
> -Original Message-
> From: Rishi Easwaran
> To: solr-user
> Sent: Tue, Jun 18, 2013 10:40 am
> Subject: Re: Solr Cloud Hangs consistently .
>
>
> Mark,
>
> All I am doing are inserts, afaik search side deadlocks should not be an
> iss
, which is very small for a
server app.
One of our install script had changed.
I had to up the ulimits - -n,-u,-v and for now no other issues seen.
-Original Message-
From: Rishi Easwaran
To: solr-user
Sent: Tue, Jun 18, 2013 10:40 am
Subject: Re: Solr Cloud Hangs consistently
solr community uses (preferably with jmeter),
we are write heavy so at the moment focusing on inserts only.
Thanks,
Rishi.
-Original Message-
From: Yago Riveiro
To: solr-user
Sent: Mon, Jun 17, 2013 6:19 pm
Subject: Re: Solr Cloud Hangs consistently .
I do all the indexing
I do all the indexing through a HTTP POST, with replicationFactor=1 no problem,
if is higher deadlock problems can appear
A stack trace like this
http://lucene.472066.n3.nabble.com/updating-docs-in-solr-cloud-hangs-td4067388.html#a4067862
is that I get
--
Yago Riveiro
Sent with Sparrow (http
indication seems to be netstat showing incoming request not being read
> in.
>
> Yago,
>
> I saw your previous post
> (http://lucene.472066.n3.nabble.com/updating-docs-in-solr-cloud-hangs-td4067388.html#a4067631)
> Following it, Last week, I upgraded to SOLR 4.3, to see if
showing incoming request not being read in.
Yago,
I saw your previous post
(http://lucene.472066.n3.nabble.com/updating-docs-in-solr-cloud-hangs-td4067388.html#a4067631)
Following it, Last week, I upgraded to SOLR 4.3, to see if the issue gets
fixed, but no luck.
Looks like this is a dominant
I can confirm that the deadlock happen with only 2 replicas by shard. I need
shutdown one node that host a replica of the shard to recover the indexation
capability.
--
Yago Riveiro
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Monday, June 17, 2013 at 6:44 PM, Rishi Easwaran wrot
FYI..you can ignore http4ClientExpiryService thread in the stack dump.
Its a dummy executor service, i created to test out something, unrelated to
this issue.
-Original Message-
From: Rishi Easwaran
To: solr-user
Sent: Mon, Jun 17, 2013 2:54 pm
Subject: Re: Solr Cloud Hangs
, 2013 1:57 pm
Subject: Re: Solr Cloud Hangs consistently .
Could you give a simple stack trace dump as well?
It's likely the distributed update deadlock that has been reported a few times
now - I think usually with a replication factor greater than 2, but I can't be
sure. The deadloc
Could you give a simple stack trace dump as well?
It's likely the distributed update deadlock that has been reported a few times
now - I think usually with a replication factor greater than 2, but I can't be
sure. The deadlock involves sending docs concurrently to replicas and I
wouldn't have e
Hi All,
I am trying to benchmark SOLR Cloud and it consistently hangs.
Nothing in the logs, no stack trace, no errors, no warnings, just seems stuck.
A little bit about my set up.
I have 3 benchmark hosts, each with 96GB RAM, 24 CPU's and 1TB SSD. Each host
is configured to have 8 SOLR cloud
Hi,
My cluster hangs again running an update process, the HTTP POST request was
aborted because a timeout error. After the hang, I couldn't do more updates
without restart the cluster.
I could see this error on node's log after kill it. Is like if solr waits for
the update response forever …
Did you take a stack trace of your _server_ and see if the
fragment I posted is the place a bunch of threads are
stuck? If so, then it's what I mentioned, and the patch
I pointed to should fix it up (when it's ready)...
The fact that it hangs more frequently with replication > 1
is consistent with
Shawn:
replicationFactor higher than one yes.
--
Yago Riveiro
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Sunday, June 2, 2013 at 4:07 PM, Shawn Heisey wrote:
> On 6/2/2013 8:28 AM, Yago Riveiro wrote:
> > Erick:
> >
> > In my case, when server hangs, no exception is thrown,
On 6/2/2013 8:28 AM, Yago Riveiro wrote:
> Erick:
>
> In my case, when server hangs, no exception is thrown, the logs on both
> servers stop registering the update INFO messages. if a shutdown one node,
> immediately the log of the alive node register some update INFO messages that
> appears wa
cument from the index and indexes an
> > updated one. The underlying lucene always requires to delete the old
> > document and index the new one..
> >
> >
> > We usually dont use partial update when updating huge number of documents.
> > This is really useful for small num
en updating huge number of documents.
> This is really useful for small number of documents (mostly during push
> indexing)...
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/updating-docs-in-solr-cloud-hangs-tp4067388p4067416.html
> Sent from the Solr - User mailing list archive at Nabble.com.
updating huge number of documents.
This is really useful for small number of documents (mostly during push
indexing)...
--
View this message in context:
http://lucene.472066.n3.nabble.com/updating-docs-in-solr-cloud-hangs-tp4067388p4067416.html
Sent from the Solr - User mailing list archive at
Hi,
I'm experimenting the same issue, I'm indexing a big file with 15M in batches
of 100K.
Sometimes, the indexing operation hangs and my HTTP client return an error of
timeout.
I see that is more frequent when the collection has more replicas.
Other thing that I can see is a lot of POST up
Hi all,
We're using Solr 4.1.0 and a 15 node Solr Cloud (configured for a 2 minute
autoCommit with no searcher being built). We have a large dataset in
Cassandra and use a Hadoop cluster to read over the dataset, build
documents, and insert them (via CloudSolrServer). That part works as
expected
38 matches
Mail list logo
