RE: stucked indexing process

2021-05-10 Thread peterbasut...@gmail.com 
Hi 

i am from the same team of Tamer who initiated this thread

We are indexing documents using apache lucene using several parallel
indexing pipelines(java process) to NFS mounted directory.
All of them follows same code and workflow most of the pipelines succeeds
without any issue, but only only few indexing pipelines remains in idle and
in RUN state forever , we observed the thread dump as well , it's not moving
at all.
if anyone is facing this issue and found solution do share with me

Note: We are using LSF cluster for our parallel indexings(java process) and
we launch these jobs using dynamic resource like cpu and memory , but
indexing process of individual lucene index serve by single host

Thread dump :
2021-05-10 09:26:22
Full thread dump OpenJDK 64-Bit Server VM (11.0.4+11 mixed mode):

Threads class SMR info:
_java_thread_list=0x2b9174000df0, length=14, elements={
0x2b90b8012000, 0x2b90ba0b5000, 0x2b90ba0b9000,
0x2b90ba0cc000,
0x2b90ba0ce000, 0x2b90ba0d, 0x2b90ba0d2000,
0x2b90ba13,
0x2b90ba144000, 0x2b90ba807800, 0x2b90ba817000,
0x2b9140001000,
0x2b9168019800, 0x2b916801e800
}

"main" #1 prio=5 os_prio=0 cpu=17492.51ms elapsed=24411.44s
tid=0x2b90b8012000 nid=0x600f1 runnable  [0x2b90b423a000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.fs.UnixNativeDispatcher.stat0(java.base@11.0.4/Native Method)
at
sun.nio.fs.UnixNativeDispatcher.stat(java.base@11.0.4/UnixNativeDispatcher.java:291)
at
sun.nio.fs.UnixFileAttributes.get(java.base@11.0.4/UnixFileAttributes.java:70)
at 
sun.nio.fs.UnixFileStore.devFor(java.base@11.0.4/UnixFileStore.java:57)
at 
sun.nio.fs.UnixFileStore.(java.base@11.0.4/UnixFileStore.java:72)
at
sun.nio.fs.LinuxFileStore.(java.base@11.0.4/LinuxFileStore.java:53)
at
sun.nio.fs.LinuxFileSystem.getFileStore(java.base@11.0.4/LinuxFileSystem.java:112)
at
sun.nio.fs.UnixFileSystem$FileStoreIterator.readNext(java.base@11.0.4/UnixFileSystem.java:212)
at
sun.nio.fs.UnixFileSystem$FileStoreIterator.hasNext(java.base@11.0.4/UnixFileSystem.java:223)
- locked <0x0007e75ab7e0> (a
sun.nio.fs.UnixFileSystem$FileStoreIterator)
at org.apache.lucene.util.IOUtils.getFileStore(IOUtils.java:595)
at org.apache.lucene.util.IOUtils.spinsLinux(IOUtils.java:539)
at org.apache.lucene.util.IOUtils.spins(IOUtils.java:528)
at org.apache.lucene.util.IOUtils.spins(IOUtils.java:503)
at
org.apache.lucene.index.ConcurrentMergeScheduler.initDynamicDefaults(ConcurrentMergeScheduler.java:412)
- locked <0x0007e7146348> (a
org.apache.lucene.index.ConcurrentMergeScheduler)
at
org.apache.lucene.index.ConcurrentMergeScheduler.merge(ConcurrentMergeScheduler.java:501)
- locked <0x0007e7146348> (a
org.apache.lucene.index.ConcurrentMergeScheduler)
at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2158)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:548)
at 
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:116)
at org.apache.lucene.index.ReaderManager.(ReaderManager.java:72)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.initReaderManager(DirectoryTaxonomyWriter.java:279)
- locked <0x0007e70bbff8> (a
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.getTaxoArrays(DirectoryTaxonomyWriter.java:749)
- locked <0x0007e70bbff8> (a
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.addCategoryDocument(DirectoryTaxonomyWriter.java:508)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.internalAddCategory(DirectoryTaxonomyWriter.java:462)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.addCategory(DirectoryTaxonomyWriter.java:429)
- locked <0x0007e70bbff8> (a
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.(DirectoryTaxonomyWriter.java:209)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.(DirectoryTaxonomyWriter.java:293)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.(DirectoryTaxonomyWriter.java:309)
at
uk.ac.ebi.ebinocle.indexer.steps.substeps.IndexStepIndexing.createTaxonomyWriter(IndexStepIndexing.java:304)
at
uk.ac.ebi.ebinocle.indexer.steps.substeps.IndexStepIndexing.prepareIndexWriter(IndexStepIndexing.java:217)
- locked <0x0007ebf96b70> (a
uk.ac.ebi.ebinocle.indexer.steps.substeps.IndexStepIndexing)
at
uk.ac.ebi.ebinocle.indexer.steps.substeps.IndexStepIndexing.prepareIndexWriter(IndexStepIndexing.java:206)

RE: stucked indexing process

2021-05-10 Thread peterbasut...@gmail.com 
Hi 

i am from the same team of Tamer who initiated this thread

We are indexing documents using apache lucene using several parallel
indexing pipelines(java process) to NFS mounted directory.
All of them follows same code and workflow most of the pipelines succeeds
without any issue, but only only few indexing pipelines remains in idle and
in RUN state forever , we observed the thread dump as well , it's not moving
at all.
if anyone is facing this issue and found solution do share with me

Note: We are using LSF cluster for our parallel indexings(java process) and
we launch these jobs using dynamic resource like cpu and memory , but
indexing process of individual lucene index serve by single host

Thread dump :
2021-05-10 09:26:22
Full thread dump OpenJDK 64-Bit Server VM (11.0.4+11 mixed mode):

Threads class SMR info:
_java_thread_list=0x2b9174000df0, length=14, elements={
0x2b90b8012000, 0x2b90ba0b5000, 0x2b90ba0b9000,
0x2b90ba0cc000,
0x2b90ba0ce000, 0x2b90ba0d, 0x2b90ba0d2000,
0x2b90ba13,
0x2b90ba144000, 0x2b90ba807800, 0x2b90ba817000,
0x2b9140001000,
0x2b9168019800, 0x2b916801e800
}

"main" #1 prio=5 os_prio=0 cpu=17492.51ms elapsed=24411.44s
tid=0x2b90b8012000 nid=0x600f1 runnable  [0x2b90b423a000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.fs.UnixNativeDispatcher.stat0(java.base@11.0.4/Native Method)
at
sun.nio.fs.UnixNativeDispatcher.stat(java.base@11.0.4/UnixNativeDispatcher.java:291)
at
sun.nio.fs.UnixFileAttributes.get(java.base@11.0.4/UnixFileAttributes.java:70)
at 
sun.nio.fs.UnixFileStore.devFor(java.base@11.0.4/UnixFileStore.java:57)
at 
sun.nio.fs.UnixFileStore.(java.base@11.0.4/UnixFileStore.java:72)
at
sun.nio.fs.LinuxFileStore.(java.base@11.0.4/LinuxFileStore.java:53)
at
sun.nio.fs.LinuxFileSystem.getFileStore(java.base@11.0.4/LinuxFileSystem.java:112)
at
sun.nio.fs.UnixFileSystem$FileStoreIterator.readNext(java.base@11.0.4/UnixFileSystem.java:212)
at
sun.nio.fs.UnixFileSystem$FileStoreIterator.hasNext(java.base@11.0.4/UnixFileSystem.java:223)
- locked <0x0007e75ab7e0> (a
sun.nio.fs.UnixFileSystem$FileStoreIterator)
at org.apache.lucene.util.IOUtils.getFileStore(IOUtils.java:595)
at org.apache.lucene.util.IOUtils.spinsLinux(IOUtils.java:539)
at org.apache.lucene.util.IOUtils.spins(IOUtils.java:528)
at org.apache.lucene.util.IOUtils.spins(IOUtils.java:503)
at
org.apache.lucene.index.ConcurrentMergeScheduler.initDynamicDefaults(ConcurrentMergeScheduler.java:412)
- locked <0x0007e7146348> (a
org.apache.lucene.index.ConcurrentMergeScheduler)
at
org.apache.lucene.index.ConcurrentMergeScheduler.merge(ConcurrentMergeScheduler.java:501)
- locked <0x0007e7146348> (a
org.apache.lucene.index.ConcurrentMergeScheduler)
at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2158)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:548)
at 
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:116)
at org.apache.lucene.index.ReaderManager.(ReaderManager.java:72)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.initReaderManager(DirectoryTaxonomyWriter.java:279)
- locked <0x0007e70bbff8> (a
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.getTaxoArrays(DirectoryTaxonomyWriter.java:749)
- locked <0x0007e70bbff8> (a
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.addCategoryDocument(DirectoryTaxonomyWriter.java:508)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.internalAddCategory(DirectoryTaxonomyWriter.java:462)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.addCategory(DirectoryTaxonomyWriter.java:429)
- locked <0x0007e70bbff8> (a
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.(DirectoryTaxonomyWriter.java:209)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.(DirectoryTaxonomyWriter.java:293)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.(DirectoryTaxonomyWriter.java:309)
at
uk.ac.ebi.ebinocle.indexer.steps.substeps.IndexStepIndexing.createTaxonomyWriter(IndexStepIndexing.java:304)
at
uk.ac.ebi.ebinocle.indexer.steps.substeps.IndexStepIndexing.prepareIndexWriter(IndexStepIndexing.java:217)
- locked <0x0007ebf96b70> (a
uk.ac.ebi.ebinocle.indexer.steps.substeps.IndexStepIndexing)
at
uk.ac.ebi.ebinocle.indexer.steps.substeps.IndexStepIndexing.prepareIndexWriter(IndexStepIndexing.java:206)

lucene indexing stuck with NFS storage mount

2021-05-10 Thread peterbasut...@gmail.com 
Hi all,

We are indexing documents using apache lucene using several parallel
indexing pipelines(java process) to NFS mounted directory.
All of them follows same code and workflow most of the pipelines succeeds
without any issue, but only only few indexing pipelines remains in idle and
in RUN state forever , we observed the thread dump as well , it's not moving
at all.
if anyone is facing this issue and found solution do share with me

Note: We are using LSF cluster for our parallel indexings(java process) and
we launch these jobs using dynamic resource like cpu and memory , but
indexing process of individual lucene index serve by single host

Thread dump :
2021-05-10 09:26:22
Full thread dump OpenJDK 64-Bit Server VM (11.0.4+11 mixed mode):

Threads class SMR info:
_java_thread_list=0x2b9174000df0, length=14, elements={
0x2b90b8012000, 0x2b90ba0b5000, 0x2b90ba0b9000,
0x2b90ba0cc000,
0x2b90ba0ce000, 0x2b90ba0d, 0x2b90ba0d2000,
0x2b90ba13,
0x2b90ba144000, 0x2b90ba807800, 0x2b90ba817000,
0x2b9140001000,
0x2b9168019800, 0x2b916801e800
}

"main" #1 prio=5 os_prio=0 cpu=17492.51ms elapsed=24411.44s
tid=0x2b90b8012000 nid=0x600f1 runnable  [0x2b90b423a000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.fs.UnixNativeDispatcher.stat0(java.base@11.0.4/Native
Method)
at
sun.nio.fs.UnixNativeDispatcher.stat(java.base@11.0.4/UnixNativeDispatcher.java:291)
at
sun.nio.fs.UnixFileAttributes.get(java.base@11.0.4/UnixFileAttributes.java:70)
at
sun.nio.fs.UnixFileStore.devFor(java.base@11.0.4/UnixFileStore.java:57)
at
sun.nio.fs.UnixFileStore.(java.base@11.0.4/UnixFileStore.java:72)
at
sun.nio.fs.LinuxFileStore.(java.base@11.0.4/LinuxFileStore.java:53)
at
sun.nio.fs.LinuxFileSystem.getFileStore(java.base@11.0.4/LinuxFileSystem.java:112)
at
sun.nio.fs.UnixFileSystem$FileStoreIterator.readNext(java.base@11.0.4/UnixFileSystem.java:212)
at
sun.nio.fs.UnixFileSystem$FileStoreIterator.hasNext(java.base@11.0.4/UnixFileSystem.java:223)
- locked <0x0007e75ab7e0> (a
sun.nio.fs.UnixFileSystem$FileStoreIterator)
at org.apache.lucene.util.IOUtils.getFileStore(IOUtils.java:595)
at org.apache.lucene.util.IOUtils.spinsLinux(IOUtils.java:539)
at org.apache.lucene.util.IOUtils.spins(IOUtils.java:528)
at org.apache.lucene.util.IOUtils.spins(IOUtils.java:503)
at
org.apache.lucene.index.ConcurrentMergeScheduler.initDynamicDefaults(ConcurrentMergeScheduler.java:412)
- locked <0x0007e7146348> (a
org.apache.lucene.index.ConcurrentMergeScheduler)
at
org.apache.lucene.index.ConcurrentMergeScheduler.merge(ConcurrentMergeScheduler.java:501)
- locked <0x0007e7146348> (a
org.apache.lucene.index.ConcurrentMergeScheduler)
at
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2158)
at
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:548)
at
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:116)
at
org.apache.lucene.index.ReaderManager.(ReaderManager.java:72)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.initReaderManager(DirectoryTaxonomyWriter.java:279)
- locked <0x0007e70bbff8> (a
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.getTaxoArrays(DirectoryTaxonomyWriter.java:749)
- locked <0x0007e70bbff8> (a
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.addCategoryDocument(DirectoryTaxonomyWriter.java:508)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.internalAddCategory(DirectoryTaxonomyWriter.java:462)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.addCategory(DirectoryTaxonomyWriter.java:429)
- locked <0x0007e70bbff8> (a
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.(DirectoryTaxonomyWriter.java:209)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.(DirectoryTaxonomyWriter.java:293)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.(DirectoryTaxonomyWriter.java:309)
at
uk.ac.ebi.ebinocle.indexer.steps.substeps.IndexStepIndexing.createTaxonomyWriter(IndexStepIndexing.java:304)
at
uk.ac.ebi.ebinocle.indexer.steps.substeps.IndexStepIndexing.prepareIndexWriter(IndexStepIndexing.java:217)
- locked <0x0007ebf96b70> (a
uk.ac.ebi.ebinocle.indexer.steps.substeps.IndexStepIndexing)
at
uk.ac.ebi.ebinocle.indexer.steps.substeps.IndexStepIndexing.prepareIndexWriter(IndexStepIndexing.java:206)
- locked <0x0007ebf96b70> (a
uk.ac.ebi.ebinocle.inde