Re: stucked indexing process

Wed, 12 Jul 2017 09:32:43 -0700 

thanks a lot for the "hack" and jstack suggestion Uwe i will try them.

Unfortunately we are in the NFS mount since we don't have other choices.
also might be related, in the cluster(computing farm) we are indexing parallel several size of different datasets and most them are indexed without problem the one that are stucking recently always the one we allocate them 1 cpu since they are very small and easy to index we ask less cpu to use our cluster efficiently. I will also increase these datasets cpu size to 2 and to see if it helps. 

thanks again
Tamer

On 12/07/2017 16:27, Uwe Schindler wrote:

Hi Tamer,

Actually you can skip the check with a “hack”:

You can override the check by enforcing number of threads and number of merges 
at same time by setting this on your own config when using 
ConcurrentMergeScheduler:

https://goo.gl/5QJpMh

If you override maxThreadCount and maxMergeCount in the CMS instance, the check 
is not executed. You may pass the CMS config using IndexWriterConfig.

In addition to find out the real issue, we need more information: The problem I had 
was that I cannot say where it exactly stopped for your case, because your “stack” 
trace-like output had no line numbers. It would be better to run “jstack <pid>” 
on command line when it hangs to see the line number in Lucene code. There are 2 
places that might hang: Listing the mount points on its own or listing their block 
device properties inside “/sys/block” folder.

Just to be sure: I hope you don’t place your indexes on NFS mounts? Aren’t you?

Uwe

-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
http://www.thetaphi.de
eMail: u...@thetaphi.de


-----Original Message-----
From: Tamer Gur [mailto:t...@ebi.ac.uk]
Sent: Wednesday, July 12, 2017 4:57 PM
To: java-user@lucene.apache.org; Uwe Schindler <u...@thetaphi.de>
Subject: Re: stucked indexing process

thanks Uwe for reply. we are indexing data in a cluster where there are
many mount points so it is possible that one them has issue or slowness
when this check first tried but now when i execute "mount" it is
responding all the mount points.

I was wondering is there any configuration to skip this SSD check?

Tamer

On 12/07/2017 14:15, Uwe Schindler wrote:

Hi,

to figure out if you system is using an SSD drive for the index
directory, the merge scheduler has to get the underlying mount point
of the index directory. As there is no direct lookup for that, it
needs to list all mount points in the system with a Java7 FS function.
And that seems to hang for some reason. Could it be that you have a
mount (like NFS or CIFS) that no longer responds?

Just list all with “cat /proc/mounts” or the “mount” command and check
if any of them is stuck or no longer responding.

Uwe

-----

Uwe Schindler

Achterdiek 19, D-28357 Bremen

http://www.thetaphi.de <http://www.thetaphi.de/>

eMail: u...@thetaphi.de

*From:*Tamer Gur [mailto:t...@ebi.ac.uk]
*Sent:* Wednesday, July 12, 2017 12:29 PM
*To:* java-user@lucene.apache.org
*Subject:* stucked indexing process

Hi all,

we are having an issue in our indexing pipeline time to time our
indexing process are stucked. Following text&picture is from jvisualvm
and it seems process is waiting at
sun.nio.fs.UnixFileSystem$FileStoreIterator.hasNext() method all the
time. we are using lucene 5.4.1 and java 1.8.0_65-b17.

what can be the reason of this?

Many Thanks

Tamer

text version

"


org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.<init

()","100.0","73509067","73509067","3"
"


org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.<init

()","100.0","73509067","73509067","3"
"


org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.addC
ategory()","100.0","73509067","73509067","3"

"


org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.inter
nalAddCategory()","100.0","73509067","73509067","3"

"


org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.addC
ategoryDocument()","100.0","73509067","73509067","3"

"


org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.getT
axoArrays()","100.0","73509067","73509067","3"

"


org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.initR
eaderManager()","100.0","73509067","73509067","3"

"


org.apache.lucene.index.ReaderManager.<init>()","100.0","73509067","7350
9067","3"

"


org.apache.lucene.index.DirectoryReader.open()","100.0","73509067","7350
9067","3"

"


org.apache.lucene.index.IndexWriter.getReader()","100.0","73509067","735
09067","3"

"


org.apache.lucene.index.IndexWriter.maybeMerge()","100.0","73509067","7
3509067","3"

"


org.apache.lucene.index.ConcurrentMergeScheduler.merge()","100.0","7350
9067","73509067","3"

"


org.apache.lucene.index.ConcurrentMergeScheduler.initDynamicDefaults()",
"100.0","73509067","73509067","3"

"
org.apache.lucene.util.IOUtils.spins()","100.0","73509067","73509067","3"
"
org.apache.lucene.util.IOUtils.spins()","100.0","73509067","73509067","3"
"


org.apache.lucene.util.IOUtils.spinsLinux()","100.0","73509067","73509067",
"3"

"


org.apache.lucene.util.IOUtils.getFileStore()","100.0","73509067","73509067"
,"3"

"


sun.nio.fs.UnixFileSystem$FileStoreIterator.hasNext()","100.0","73509067","
73509067","3"

image version




---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org




---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org

Reply via email to