Hi,
this looks like an issue in Solr or how you use Solr. Could it be that you
are reloading the cores all the time? Because the mentioned
"IOUtils.spins()" should only be called when the index is opened and the
IndexWriter is initialized by Solr. It is unlikely that you have any
concurrency there.
There might be one problem: If you have a stuck mount point in your system
(like another NFS mount) that hangs, it might happen that Lucene's code also
hangs, as it inspects the mount points for SSD / spinning disks on starting
up IndexWriter. So please make sure that "mount" does not hang and all the
mountpoints respond (e.g. there are o hanging NFS mounts blocking lucene
from inspecting mounts).
This is also a different issue than the one mentioned before, because you
don't use NFS, it's a local disk, right?
One workaround may be to explicitely tell ConcurrentMergeScheduler to enable
SSD or spinning disk default settings in your solrconfig.xml:
<mergeScheduler
class="org.apache.lucene.index.ConcurrentMergeScheduler">
<bool name="defaultMaxMergesAndThreads">true</bool>
</mergeScheduler>
Use "true" for spinning disks and "false" for SSDs. This prevents the
auto-detection from running.
Uwe
-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: u...@thetaphi.de
> -----Original Message-----
> From: Sachin909 <sachin...@gmail.com>
> Sent: Wednesday, October 14, 2020 9:43 AM
> To: java-user@lucene.apache.org
> Subject: RE: stucked indexing process
>
> Hi Uwe,
>
> I have observed the similer issue with my application.
>
> Application stack:
>
> "coreLoadExecutor-4-thread-1" #86 prio=5 os_prio=0 tid=0x00007fbb1c364800
> *nid=0x1616* runnable [0x00007fbaa96ef000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.fs.UnixNativeDispatcher.stat0(Native Method)
> at
sun.nio.fs.UnixNativeDispatcher.stat(UnixNativeDispatcher.java:286)
> at sun.nio.fs.UnixFileAttributes.get(UnixFileAttributes.java:70)
> at sun.nio.fs.UnixFileStore.devFor(UnixFileStore.java:55)
> at sun.nio.fs.UnixFileStore.<init>(UnixFileStore.java:70)
> at sun.nio.fs.LinuxFileStore.<init>(LinuxFileStore.java:48)
> at sun.nio.fs.LinuxFileSystem.getFileStore(LinuxFileSystem.java:112)
> at
>
sun.nio.fs.UnixFileSystem$FileStoreIterator.readNext(UnixFileSystem.java:213
)
> at
>
sun.nio.fs.UnixFileSystem$FileStoreIterator.hasNext(UnixFileSystem.java:224)
> - locked <0x00000000996864f8> (a
> sun.nio.fs.UnixFileSystem$FileStoreIterator)
> at org.apache.lucene.util.IOUtils.getFileStore(IOUtils.java:543)
> at org.apache.lucene.util.IOUtils.spinsLinux(IOUtils.java:487)
> at org.apache.lucene.util.IOUtils.spins(IOUtils.java:476)
> at org.apache.lucene.util.IOUtils.spins(IOUtils.java:451)
> at
> *org.apache.lucene.index.ConcurrentMergeScheduler.initDynamicDefaults(Con
> currentMergeScheduler.java:376)*
> - locked <0x0000000099686598> (a
> org.apache.lucene.index.ConcurrentMergeScheduler)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.merge(ConcurrentMergeS
> cheduler.java:464)
> - locked <0x0000000099686598> (a
> org.apache.lucene.index.ConcurrentMergeScheduler)
> at
> org.apache.lucene.index.IndexWriter.waitForMerges(IndexWriter.java:2444)
> at
> org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1131)
> at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1175)
> at
> org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:291)
> at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:716)
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:899)
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:816)
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:890)
> at
> org.apache.solr.core.CoreContainer.lambda$load$3(CoreContainer.java:542)
> at
> org.apache.solr.core.CoreContainer$$Lambda$34/209767675.call(Unknown
> Source)
> at
>
com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(I
> nstrumentedExecutorService.java:197)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lam
> bda$execute$0(ExecutorUtil.java:229)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$La
> mbda$35/1998024988.run(Unknown
> Source)
> at
>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1
> 149)
> at
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:
> 624)
> at java.lang.Thread.run(Thread.java:748)
>
>
> Mount:
> /dev/mapper/appvg-lv_apps /apps index location has sufficient
> (100Gb+)disk free space.
>
>
> starce command:
>
> In the strace command I have observed above thread were active and
> performing the operation till some point after that it doesn't do any
> activity,
>
> Jstack *nid=0x1616* = 5654
>
> Line 54534: 5654 18:53:53.224879 open("/proc/mounts",
> O_RDONLY|O_CLOEXEC
> <unfinished ...>
> Line 54536: 5654 18:53:53.226100 <... open resumed> ) = 2231
> <0.001205>
> Line 54538: 5654 18:53:53.226206 fstat(2231, <unfinished ...>
> Line 54540: 5654 18:53:53.226255 <... fstat resumed>
> {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 <0.000039>
>
> ....
>
> Line 55026: 5654 18:53:53.247991
> stat("/net/icgudadmna1p/ctodata",
> <unfinished ...>
>
> << thread doesn't show any activity after /net/...>>>
>
> Could you please advice what could be the possible cause.
>
> Java : Azule JDK1.8
> org.apache.lucene-lucene-core-2.4.1.jar
>
>
>
> --
> Sent from: https://lucene.472066.n3.nabble.com/Lucene-Java-Users-
> f532864.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> For additional commands, e-mail: java-user-h...@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
