[jira] [Commented] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[ https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17581906#comment-17581906 ] Giovanna Cazelato Pires commented on NIFI-9572: --- [~joewitt] Hii! My ulimit -a : !image-2022-08-19-12-03-35-722.png! > Failed to index Provenance Events and (Too many Files) > -- > > Key: NIFI-9572 > URL: https://issues.apache.org/jira/browse/NIFI-9572 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.15.2 >Reporter: mayki >Priority: Major > Attachments: bootstrap.conf, image-2022-08-19-12-03-35-722.png, lsof > (1).txt, nifi-app.log, nifi-app.log.tar.gz, nifi.properties, > nifi_691106_pid.tar.gz > > > Hello > I have upgraded NIFI 1.15.2 since 2022/01/05 > No issue until this night 2022/01/13 > * nifi version 1.15.2 > * jdk-1.8.0_311 > And the limit is high > {code:java} > Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2 > -bash-4.2@nifi$ ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) rg > pending signals (-i) 63278 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 5 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 1 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > {code} > > We got a lot error about provenance_repository, it fill our filesystem logs .. > > {code:java} > 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2] > o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events > org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877) > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891) > at > org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468) > at > org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444) > at > org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.nio.file.FileSystemException: > /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd: > Too many open files > {code} > > > We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability. > But it is impossible to do that if we got this error. > > Thanks for you help. > > Regards > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[ https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17580868#comment-17580868 ] Joe Witt commented on NIFI-9572: [~giocapi] Here again as the examples before the lsof output provided shows only 4000 or so open files (4192 in your case). If you hit out of file limits with only 4000 or so files open yet you think the limit on your system is 999K then something isn't right. We've seen more and more confusion around what people meant to set for max open files vs what they're actually seeing. You will want to show more from your logs/output and dumps so we can get the full context of what might be happening for you. Thanks > Failed to index Provenance Events and (Too many Files) > -- > > Key: NIFI-9572 > URL: https://issues.apache.org/jira/browse/NIFI-9572 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.15.2 >Reporter: mayki >Priority: Major > Attachments: bootstrap.conf, lsof (1).txt, nifi-app.log, > nifi-app.log.tar.gz, nifi.properties, nifi_691106_pid.tar.gz > > > Hello > I have upgraded NIFI 1.15.2 since 2022/01/05 > No issue until this night 2022/01/13 > * nifi version 1.15.2 > * jdk-1.8.0_311 > And the limit is high > {code:java} > Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2 > -bash-4.2@nifi$ ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) rg > pending signals (-i) 63278 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 5 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 1 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > {code} > > We got a lot error about provenance_repository, it fill our filesystem logs .. > > {code:java} > 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2] > o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events > org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877) > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891) > at > org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468) > at > org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444) > at > org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.nio.file.FileSystemException: > /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd: > Too many open files > {code} > > > We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability. > But it is impossible to do that if we got this error. > > Thanks for you help. > > Regards > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[ https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17580785#comment-17580785 ] Giovanna Cazelato Pires commented on NIFI-9572: --- Hi everyone! I'm experience this same issue for a while now. 1. The specific version of apache nifi. - 1.15.3 2. The specific os/kernel version in use (uname -a) -- 4.14.268-139.500.amzn1.x86_64 3. The specific java runtime actually being used by nifi - 11.0.16.8.1 4. A dump from 'lsof -p ' --- attachment ( lsof(1).txt ) 5. What is your ulimit for files? (ulimit -a) -- 999k [^lsof (1).txt] > Failed to index Provenance Events and (Too many Files) > -- > > Key: NIFI-9572 > URL: https://issues.apache.org/jira/browse/NIFI-9572 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.15.2 >Reporter: mayki >Priority: Major > Attachments: bootstrap.conf, lsof (1).txt, nifi-app.log, > nifi-app.log.tar.gz, nifi.properties, nifi_691106_pid.tar.gz > > > Hello > I have upgraded NIFI 1.15.2 since 2022/01/05 > No issue until this night 2022/01/13 > * nifi version 1.15.2 > * jdk-1.8.0_311 > And the limit is high > {code:java} > Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2 > -bash-4.2@nifi$ ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) rg > pending signals (-i) 63278 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 5 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 1 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > {code} > > We got a lot error about provenance_repository, it fill our filesystem logs .. > > {code:java} > 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2] > o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events > org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877) > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891) > at > org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468) > at > org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444) > at > org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.nio.file.FileSystemException: > /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd: > Too many open files > {code} > > > We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability. > But it is impossible to do that if we got this error. > > Thanks for you help. > > Regards > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[ https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528846#comment-17528846 ] Joe Witt commented on NIFI-9572: For anyone that has issues with max open files what we need in each case is: 1. The specific version of apache nifi. 2. The specific os/kernel version in use (uname -a) 3. The specific java runtime actually being used by nifi 4. A dump from 'lsof -p ' 5. What is your ulimit for files? (ulimit -a) If the lsof dump doesn't equate to a very large number (close to your systems max) then the process your listing it for isnt the right one. Or nifi isn't the one leaking. Or the leaked resources are no longer bound to the nifi process. Running out of file handles happens. It has occurred due to bugs in nifi historically. It has occurred due to bugs in the JVM. And it has happened due to bugs in custom components people created. The most common cause has been socket leaks rather than file leaks. The good thing about that is it is almost always easy to tie it to something in code/system/etc.. (except the JVM issue which honestly was a brutal nightmare). So please in each case share those things and we can help pinpoint root cause. > Failed to index Provenance Events and (Too many Files) > -- > > Key: NIFI-9572 > URL: https://issues.apache.org/jira/browse/NIFI-9572 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.15.2 >Reporter: mayki >Priority: Major > Attachments: bootstrap.conf, nifi-app.log, nifi-app.log.tar.gz, > nifi.properties, nifi_691106_pid.tar.gz > > > Hello > I have upgraded NIFI 1.15.2 since 2022/01/05 > No issue until this night 2022/01/13 > * nifi version 1.15.2 > * jdk-1.8.0_311 > And the limit is high > {code:java} > Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2 > -bash-4.2@nifi$ ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) rg > pending signals (-i) 63278 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 5 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 1 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > {code} > > We got a lot error about provenance_repository, it fill our filesystem logs .. > > {code:java} > 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2] > o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events > org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877) > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891) > at > org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468) > at > org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444) > at > org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.nio.file.FileSystemException: > /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd: > Too many open files > {code} > > > We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability. > But it is impossible to do that if we got this error. > > Thanks for you help. > > Regards > > > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[ https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528840#comment-17528840 ] Marcus Ely commented on NIFI-9572: -- Our team has also had this issue since we upgraded from 1.13 to 1.15, and we have no way of fixing it besides restarting NiFi on the node reporting 'Too many open files'. This is always temporary, and the issue comes back maybe after 1 day. Our ulimit is set properly, and again, this was never an issue until we did that upgrade. > Failed to index Provenance Events and (Too many Files) > -- > > Key: NIFI-9572 > URL: https://issues.apache.org/jira/browse/NIFI-9572 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.15.2 >Reporter: mayki >Priority: Major > Attachments: bootstrap.conf, nifi-app.log, nifi-app.log.tar.gz, > nifi.properties, nifi_691106_pid.tar.gz > > > Hello > I have upgraded NIFI 1.15.2 since 2022/01/05 > No issue until this night 2022/01/13 > * nifi version 1.15.2 > * jdk-1.8.0_311 > And the limit is high > {code:java} > Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2 > -bash-4.2@nifi$ ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) rg > pending signals (-i) 63278 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 5 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 1 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > {code} > > We got a lot error about provenance_repository, it fill our filesystem logs .. > > {code:java} > 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2] > o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events > org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877) > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891) > at > org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468) > at > org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444) > at > org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.nio.file.FileSystemException: > /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd: > Too many open files > {code} > > > We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability. > But it is impossible to do that if we got this error. > > Thanks for you help. > > Regards > > > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[ https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17524929#comment-17524929 ] mayki commented on NIFI-9572: - Hello [~joewitt] Another person has the same issue https://community.cloudera.com/t5/Support-Questions/Failed-to-index-Provenance-Events-org-apache-lucene-store/td-p/340666 > Failed to index Provenance Events and (Too many Files) > -- > > Key: NIFI-9572 > URL: https://issues.apache.org/jira/browse/NIFI-9572 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.15.2 >Reporter: mayki >Priority: Major > Attachments: bootstrap.conf, nifi-app.log, nifi-app.log.tar.gz, > nifi.properties, nifi_691106_pid.tar.gz > > > Hello > I have upgraded NIFI 1.15.2 since 2022/01/05 > No issue until this night 2022/01/13 > * nifi version 1.15.2 > * jdk-1.8.0_311 > And the limit is high > {code:java} > Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2 > -bash-4.2@nifi$ ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) rg > pending signals (-i) 63278 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 5 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 1 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > {code} > > We got a lot error about provenance_repository, it fill our filesystem logs .. > > {code:java} > 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2] > o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events > org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877) > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891) > at > org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468) > at > org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444) > at > org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.nio.file.FileSystemException: > /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd: > Too many open files > {code} > > > We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability. > But it is impossible to do that if we got this error. > > Thanks for you help. > > Regards > > > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[ https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17498911#comment-17498911 ] mayki commented on NIFI-9572: - Hello [~joewitt] Thanks for response : Here the information about whoami and ulimit -a {code:java} -bash-4.2$ whoami nifi -bash-4.2$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 63278 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 5 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 1 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited -bash-4.2$ ps -ef|grep nifi nifi 3367182 1 0 14:38 ? 00:00:00 /bin/sh /appl/nifi/nifi-current/bin/nifi.sh start nifi 3367184 3367182 0 14:38 ? 00:00:01 /appl/pkg/jdk-1.8.0_311/bin/java -cp /appl/nifi/nifi-1.15.3/conf:/appl/nifi/nifi-1.15.3/lib/bootstrap/* -Xms12m -Xmx24m -Dorg.apache.nifi.bootstrap.config.log.dir=/logs/nifi -Dorg.apache.nifi.bootstrap.config.pid.dir=/var/run/nifi -Dorg.apache.nifi.bootstrap.config.file=/appl/nifi/nifi-1.15.3/conf/bootstrap.conf org.apache.nifi.bootstrap.RunNiFi start nifi 3367203 3367184 75 14:38 ? 00:05:23 /appl/pkg/jdk-1.8.0_311/bin/java -classpath /appl/nifi/nifi-current/conf:/appl nifi/nifi-current/lib/javax.servlet-api-3.1.0.jar:/appl/nifi/nifi-current/lib/jetty-schemas-3.1.jar:/appl/nifi/nifi-current/lib/logback-classic-1.2.10.jar:/appl/nifi/nifi-current/lib/logback-core-1.2.10.jar:/appl/nifi/nifi-current/lib/jcl-over-slf4j-1.7.32.jar:/appl/nifi/nifi-current/lib/jul-to-slf4j-1.7.32.jar:/appl/nifi/nifi-current/lib/log4j-over-slf4j-1.7.32.jar:/appl/nifi/nifi-current/lib/slf4j-api-1.7.32.jar:/appl/nifi/nifi-current/lib/nifi-api-1.15.3.jar:/appl/nifi/nifi-current/lib/nifi-framework-api-1.15.3.jar:/appl/nifi/nifi-current/lib/nifi-server-api-1.15.3.jar:/appl/nifi/nifi-current/lib/nifi-runtime-1.15.3.jar:/appl/nifi/nifi-current/lib/nifi-nar-utils-1.15.3.jar:/appl/nifi/nifi-current/lib/nifi-properties-1.15.3.jar:/appl/nifi/nifi-current/lib/nifi-property-utils-1.15.3.jar:/appl/nifi/nifi-current/lib/nifi-stateless-bootstrap-1.15.3.jar:/appl/nifi/nifi-current/lib/nifi-stateless-api-1.15.3.jar -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx8192m -Xms8192m -Djavax.security.auth.useSubjectCredsOnly=true -Djava.security.egd=file:/dev/urandom -Dzookeeper.admin.enableServer=false -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djava.protocol.handler.pkgs=sun.net.www.protocol -Dnifi.properties.file.path=/appl/nifi/nifi-current/conf/nifi.properties -Dnifi.bootstrap.listen.port=25860 -Dapp=NiFi -Dorg.apache.nifi.bootstrap.config.log.dir=/logs/nifi org.apache.nifi.NiFi -K /appl/nifi/nifi-current/conf/sensitive.key root 3369114 3361513 0 14:43 pts/2 00:00:00 su - nifi nifi 3369116 3369114 0 14:43 pts/2 00:00:00 -bash nifi 3370165 3369116 0 14:45 pts/2 00:00:00 ps -ef nifi 3370166 3369116 0 14:45 pts/2 00:00:00 grep --color=auto nifi {code} Thanks > Failed to index Provenance Events and (Too many Files) > -- > > Key: NIFI-9572 > URL: https://issues.apache.org/jira/browse/NIFI-9572 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.15.2 >Reporter: mayki >Priority: Major > Attachments: bootstrap.conf, nifi-app.log, nifi-app.log.tar.gz, > nifi.properties, nifi_691106_pid.tar.gz > > > Hello > I have upgraded NIFI 1.15.2 since 2022/01/05 > No issue until this night 2022/01/13 > * nifi version 1.15.2 > * jdk-1.8.0_311 > And the limit is high > {code:java} > Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2 > -bash-4.2@nifi$ ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) rg > pending signals (-i) 63278 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 5 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 1 > virtual memory (kbytes, -v) unlimited > file locks
[jira] [Commented] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[ https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17496967#comment-17496967 ] Joe Witt commented on NIFI-9572: Hello. I have again reviewed the data you've provided and it remains the case that all the lsof output provided shows never exceeding around 7500 open file handles at once yet your app log shows you hit max open files and your 'ulimit -a' output shows you mean for it to be at least 50,000 or 100,000. To me this again suggests that the ulimit settings for your actual 'nifi' user is not what you mean for it to be. Please run the following commands on the command line as the 'nifi' user that nifi actually runs as and show the command and results of both. whoami ulimit -a It should show 'nifi' then should show the actual ulumits for that specific user. There is no evidence in any of the lsof outputs you showed that indicate you came even close to 10,000 much less 50,000 or more. Thanks > Failed to index Provenance Events and (Too many Files) > -- > > Key: NIFI-9572 > URL: https://issues.apache.org/jira/browse/NIFI-9572 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.15.2 >Reporter: mayki >Priority: Major > Attachments: bootstrap.conf, nifi-app.log, nifi-app.log.tar.gz, > nifi.properties, nifi_691106_pid.tar.gz > > > Hello > I have upgraded NIFI 1.15.2 since 2022/01/05 > No issue until this night 2022/01/13 > * nifi version 1.15.2 > * jdk-1.8.0_311 > And the limit is high > {code:java} > Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2 > -bash-4.2@nifi$ ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) rg > pending signals (-i) 63278 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 5 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 1 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > {code} > > We got a lot error about provenance_repository, it fill our filesystem logs .. > > {code:java} > 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2] > o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events > org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877) > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891) > at > org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468) > at > org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444) > at > org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.nio.file.FileSystemException: > /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd: > Too many open files > {code} > > > We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability. > But it is impossible to do that if we got this error. > > Thanks for you help. > > Regards > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[ https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17496761#comment-17496761 ] mayki commented on NIFI-9572: - Hello [~joewitt] For information, if we downgrade the nifi to 1.15.3 to 1.11.4. The error "Too many open files" disappear. Thanks if someone can help us to found what is happenend > Failed to index Provenance Events and (Too many Files) > -- > > Key: NIFI-9572 > URL: https://issues.apache.org/jira/browse/NIFI-9572 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.15.2 >Reporter: mayki >Priority: Major > Attachments: bootstrap.conf, nifi-app.log, nifi-app.log.tar.gz, > nifi.properties, nifi_691106_pid.tar.gz > > > Hello > I have upgraded NIFI 1.15.2 since 2022/01/05 > No issue until this night 2022/01/13 > * nifi version 1.15.2 > * jdk-1.8.0_311 > And the limit is high > {code:java} > Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2 > -bash-4.2@nifi$ ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) rg > pending signals (-i) 63278 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 5 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 1 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > {code} > > We got a lot error about provenance_repository, it fill our filesystem logs .. > > {code:java} > 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2] > o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events > org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877) > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891) > at > org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468) > at > org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444) > at > org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.nio.file.FileSystemException: > /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd: > Too many open files > {code} > > > We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability. > But it is impossible to do that if we got this error. > > Thanks for you help. > > Regards > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[ https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17494609#comment-17494609 ] mayki commented on NIFI-9572: - [^bootstrap.conf] > Failed to index Provenance Events and (Too many Files) > -- > > Key: NIFI-9572 > URL: https://issues.apache.org/jira/browse/NIFI-9572 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.15.2 >Reporter: mayki >Priority: Major > Attachments: bootstrap.conf, nifi-app.log, nifi-app.log.tar.gz, > nifi.properties, nifi_691106_pid.tar.gz > > > Hello > I have upgraded NIFI 1.15.2 since 2022/01/05 > No issue until this night 2022/01/13 > * nifi version 1.15.2 > * jdk-1.8.0_311 > And the limit is high > {code:java} > Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2 > -bash-4.2@nifi$ ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) rg > pending signals (-i) 63278 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 5 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 1 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > {code} > > We got a lot error about provenance_repository, it fill our filesystem logs .. > > {code:java} > 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2] > o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events > org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877) > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891) > at > org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468) > at > org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444) > at > org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.nio.file.FileSystemException: > /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd: > Too many open files > {code} > > > We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability. > But it is impossible to do that if we got this error. > > Thanks for you help. > > Regards > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[ https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17494606#comment-17494606 ] mayki commented on NIFI-9572: - Hello [~joewitt] I pushed the information in the files: * NIFI version 1.15.3 * JAVA /appl/pkg/jdk-1.8.0_311/bin/java * nifi_691106_pid.tar.gz ==> resultat of losf -p pid * nifi-app.log.tar.gz ==> nifi-app.log {code:java} -bash-4.2$ whoami nifi -bash-4.2$ ps -ef|grep nifi nifi 691085 1 0 Feb11 ? 00:00:00 /bin/sh /appl/nifi/nifi-current/bin/nifi.sh start nifi 691087 691085 0 Feb11 ? 00:07:44 /appl/pkg/jdk-1.8.0_311/bin/java -cp /appl/nifi/nifi-1.15.3/conf:/appl/nifi/nifi-1.15.3/lib/bootstrap/* -Xms12m -Xmx24m -Dorg.apache.nifi.bootstrap.config.log.dir=/logs/nifi -Dorg.apache.nifi.bootstrap.config.pid.dir=/var/run/nifi -Dorg.apache.nifi.bootstrap.config.file=/appl/nifi/nifi-1.15.3/conf/bootstrap.conf org.apache.nifi.bootstrap.RunNiFi start nifi 691106 691087 36 Feb11 ? 2-13:36:35 /appl/pkg/jdk-1.8.0_311/bin/java -classpath /appl/nifi/nifi-current/conf:/appl/nifi/nifi-current/lib/javax.servlet-api-3.1.0.jar:/appl/nifi/nifi-current/lib/jetty-schemas-3.1.jar:/appl/nifi/nifi-current/lib/logback-classic-1.2.10.jar:/appl/nifi/nifi-current/lib/logback-core-1.2.10.jar:/appl/nifi/nifi-current/lib/jcl-over-slf4j-1.7.32.jar:/appl/nifi/nifi-current/lib/jul-to-slf4j-1.7.32.jar:/appl/nifi/nifi-current/lib/log4j-over-slf4j-1.7.32.jar:/appl/nifi/nifi-current/lib/slf4j-api-1.7.32.jar:/appl/nifi/nifi-current/lib/nifi-api-1.15.3.jar:/appl/nifi/nifi-current/lib/nifi-framework-api-1.15.3.jar:/appl/nifi/nifi-current/lib/nifi-server-api-1.15.3.jar:/appl/nifi/nifi-current/lib/nifi-runtime-1.15.3.jar:/appl/nifi/nifi-current/lib/nifi-nar-utils-1.15.3.jar:/appl/nifi/nifi-current/lib/nifi-properties-1.15.3.jar:/appl/nifi/nifi-current/lib/nifi-property-utils-1.15.3.jar:/appl/nifi/nifi-current/lib/nifi-stateless-bootstrap-1.15.3.jar:/appl/nifi/nifi-current/lib/nifi-stateless-api-1.15.3.jar -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx8192m -Xms8192m -Djavax.security.auth.useSubjectCredsOnly=true -Djava.security.egd=file:/dev/urandom -Dzookeeper.admin.enableServer=false -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djava.protocol.handler.pkgs=sun.net.www.protocol -Dnifi.properties.file.path=/appl/nifi/nifi-current/conf/nifi.properties -Dnifi.bootstrap.listen.port=27307 -Dapp=NiFi -Dorg.apache.nifi.bootstrap.config.log.dir=/logs/nifi org.apache.nifi.NiFi -K /appl/nifi/nifi-current/conf/sensitive.key-bash-4.2$ /appl/pkg/jdk-1.8.0_311/bin/java -version java version "1.8.0_311" Java(TM) SE Runtime Environment (build 1.8.0_311-b11) Java HotSpot(TM) 64-Bit Server VM (build 25.311-b11, mixed mode) -bash-4.2$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 63278 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 5 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 1 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited {code} [^nifi_691106_pid.tar.gz] [^nifi-app.log.tar.gz] Thanks > Failed to index Provenance Events and (Too many Files) > -- > > Key: NIFI-9572 > URL: https://issues.apache.org/jira/browse/NIFI-9572 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.15.2 >Reporter: mayki >Priority: Major > Attachments: nifi-app.log, nifi-app.log.tar.gz, nifi_691106_pid.tar.gz > > > Hello > I have upgraded NIFI 1.15.2 since 2022/01/05 > No issue until this night 2022/01/13 > * nifi version 1.15.2 > * jdk-1.8.0_311 > And the limit is high > {code:java} > Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2 > -bash-4.2@nifi$ ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) rg > pending signals (-i) 63278 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 5 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u)
[jira] [Commented] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[ https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17493820#comment-17493820 ] mayki commented on NIFI-9572: - Hello Ok, sorry for confusing... I will send you all thing you ask in the order. Thanks > Failed to index Provenance Events and (Too many Files) > -- > > Key: NIFI-9572 > URL: https://issues.apache.org/jira/browse/NIFI-9572 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.15.2 >Reporter: mayki >Priority: Major > Attachments: nifi-app.log > > > Hello > I have upgraded NIFI 1.15.2 since 2022/01/05 > No issue until this night 2022/01/13 > * nifi version 1.15.2 > * jdk-1.8.0_311 > And the limit is high > {code:java} > Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2 > -bash-4.2@nifi$ ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) rg > pending signals (-i) 63278 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 5 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 1 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > {code} > > We got a lot error about provenance_repository, it fill our filesystem logs .. > > {code:java} > 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2] > o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events > org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877) > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891) > at > org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468) > at > org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444) > at > org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.nio.file.FileSystemException: > /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd: > Too many open files > {code} > > > We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability. > But it is impossible to do that if we got this error. > > Thanks for you help. > > Regards > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[ https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17492641#comment-17492641 ] Joe Witt commented on NIFI-9572: Hello [~mayki] In requesting the 'java -version' output you instead appear to have sent the output of other bash commands. Further the link implies you're on JDK 1.8 update 261. However, the beginning of the JIRA you suggest you're on update 311. In requesting the lsof output periodically as the number of open files grow you are instead showing that now nifi fails to start with an open file handle limit hit during the startup process. I again suspect you do not have the proper file limit set for the user you're actually running nifi as (which the above info implies might be the user 'nifi'). I dont know that the ulimit results you've shown are for the user nifi or whether they're for the root user or some other user. It is very difficult to be helpful when the various details don't line up because we're not making progress. I appreciate there is a chance that there is some bug in NIFi causing this but at this stage I dont suspect that - I suspect environment/config setup not matching what you're intending. We really need to see full commands/full output at this point to help progress. I think for me to be helpful via a JIRA here I still need to see: 1. 'whoami' 2. 'java -version' including command entry and results 3. 'ulimit -a' including command entry and results 4. 'lsof -p ' run every 30 minutes while nifi is running Thanks Joe > Failed to index Provenance Events and (Too many Files) > -- > > Key: NIFI-9572 > URL: https://issues.apache.org/jira/browse/NIFI-9572 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.15.2 >Reporter: mayki >Priority: Major > Attachments: nifi-app.log > > > Hello > I have upgraded NIFI 1.15.2 since 2022/01/05 > No issue until this night 2022/01/13 > * nifi version 1.15.2 > * jdk-1.8.0_311 > And the limit is high > {code:java} > Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2 > -bash-4.2@nifi$ ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) rg > pending signals (-i) 63278 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 5 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 1 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > {code} > > We got a lot error about provenance_repository, it fill our filesystem logs .. > > {code:java} > 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2] > o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events > org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877) > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891) > at > org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468) > at > org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444) > at > org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.nio.file.FileSystemException: > /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd: > Too many open files > {code} > > > We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability. > But it is impossible to do that if we got this error. > > Thanks for you help. > > Regards > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[ https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17492538#comment-17492538 ] mayki commented on NIFI-9572: - Hello [~joewitt] here the information, this time, NIFI failed to start {code:java} # grep java /appl/nifi/nifi-current/bin/nifi-env.sh # The java implementation to use. export JAVA_HOME=$(readlink -f /appl/pkg/jdk-current/bin/java | sed "s:/bin/java::") # ls -rtl /appl/pkg |grep jdk-current lrwxrwxrwx 1 root root 13 Feb 15 10:47 jdk-current -> jdk-1.8.0.261 # grep run.as bootstrap.conf run.as=nifi {code} I send the nifi-app.log about restart failed[^nifi-app.log] > Failed to index Provenance Events and (Too many Files) > -- > > Key: NIFI-9572 > URL: https://issues.apache.org/jira/browse/NIFI-9572 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.15.2 >Reporter: mayki >Priority: Major > Attachments: nifi-app.log > > > Hello > I have upgraded NIFI 1.15.2 since 2022/01/05 > No issue until this night 2022/01/13 > * nifi version 1.15.2 > * jdk-1.8.0_311 > And the limit is high > {code:java} > Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2 > -bash-4.2@nifi$ ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) rg > pending signals (-i) 63278 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 5 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 1 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > {code} > > We got a lot error about provenance_repository, it fill our filesystem logs .. > > {code:java} > 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2] > o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events > org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877) > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891) > at > org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468) > at > org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444) > at > org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.nio.file.FileSystemException: > /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd: > Too many open files > {code} > > > We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability. > But it is impossible to do that if we got this error. > > Thanks for you help. > > Regards > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[ https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17492077#comment-17492077 ] mayki commented on NIFI-9572: - I will rerun again the next time Also I have never this issue on nifi 1.11.4 > Failed to index Provenance Events and (Too many Files) > -- > > Key: NIFI-9572 > URL: https://issues.apache.org/jira/browse/NIFI-9572 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.15.2 >Reporter: mayki >Priority: Major > > Hello > I have upgraded NIFI 1.15.2 since 2022/01/05 > No issue until this night 2022/01/13 > * nifi version 1.15.2 > * jdk-1.8.0_311 > And the limit is high > {code:java} > Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2 > -bash-4.2@nifi$ ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) rg > pending signals (-i) 63278 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 5 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 1 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > {code} > > We got a lot error about provenance_repository, it fill our filesystem logs .. > > {code:java} > 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2] > o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events > org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877) > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891) > at > org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468) > at > org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444) > at > org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.nio.file.FileSystemException: > /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd: > Too many open files > {code} > > > We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability. > But it is impossible to do that if we got this error. > > Thanks for you help. > > Regards > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[ https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17492082#comment-17492082 ] Joe Witt commented on NIFI-9572: [~mayki] The results I see for lsof are a 'count of results'. I'm saying we need to see the results of the 'lsof -p ' command. Not that command then summarized with ' | wc -l'. The counts you show are nowhere near 50,000 or the newly assigned 100,000 so I am not sure whether we're looking at the right data. Again users often think they have the ulimit set correctly only to discover for the actual user nifi is run as that it is not. Might be a factor here - might not. Lastly, not seeing this in nifi 1.11.4 might simply suggest we kept certain handles closed more aggressively, or never opened them, and now we exceed some default. We will see But lets look at a result where the lsof output reveals we're exceeding 10,000 or 20,000 or more entries climbing toward the out of file handle condition the results suggest. If we get the lsof output then it will often be highly helpful. Lastly, what is the JVM/JRE in use? Please show 'java -version' output > Failed to index Provenance Events and (Too many Files) > -- > > Key: NIFI-9572 > URL: https://issues.apache.org/jira/browse/NIFI-9572 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.15.2 >Reporter: mayki >Priority: Major > > Hello > I have upgraded NIFI 1.15.2 since 2022/01/05 > No issue until this night 2022/01/13 > * nifi version 1.15.2 > * jdk-1.8.0_311 > And the limit is high > {code:java} > Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2 > -bash-4.2@nifi$ ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) rg > pending signals (-i) 63278 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 5 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 1 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > {code} > > We got a lot error about provenance_repository, it fill our filesystem logs .. > > {code:java} > 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2] > o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events > org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877) > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891) > at > org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468) > at > org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444) > at > org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.nio.file.FileSystemException: > /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd: > Too many open files > {code} > > > We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability. > But it is impossible to do that if we got this error. > > Thanks for you help. > > Regards > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[ https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17492074#comment-17492074 ] mayki commented on NIFI-9572: - Thanks [~joewitt] for response, I have already sent the result of "lsof" {code:java} core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 63278 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 10 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 1 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited {code} > Failed to index Provenance Events and (Too many Files) > -- > > Key: NIFI-9572 > URL: https://issues.apache.org/jira/browse/NIFI-9572 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.15.2 >Reporter: mayki >Priority: Major > > Hello > I have upgraded NIFI 1.15.2 since 2022/01/05 > No issue until this night 2022/01/13 > * nifi version 1.15.2 > * jdk-1.8.0_311 > And the limit is high > {code:java} > Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2 > -bash-4.2@nifi$ ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) rg > pending signals (-i) 63278 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 5 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 1 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > {code} > > We got a lot error about provenance_repository, it fill our filesystem logs .. > > {code:java} > 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2] > o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events > org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877) > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891) > at > org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468) > at > org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444) > at > org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.nio.file.FileSystemException: > /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd: > Too many open files > {code} > > > We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability. > But it is impossible to do that if we got this error. > > Thanks for you help. > > Regards > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[ https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17492073#comment-17492073 ] Joe Witt commented on NIFI-9572: Actually yeah you're ulimit data shows you intent to allow as many as 50,000 open files at once. Yet the lsof runs you've done with 'wc -l' show less than 10,000. So either you dont have the right limit for the user nifi actually runs as OR you are somehow growing file handles and then hitting 50,000. In either case the raw lsof output will help us see what is in the list for the 20,30,40thousand entries.Also if this is slowing happening over time are you monitoring file handle growth happening over time? How long from restart to out of file handles? Lets see lsof output/counts every 30 minutes > Failed to index Provenance Events and (Too many Files) > -- > > Key: NIFI-9572 > URL: https://issues.apache.org/jira/browse/NIFI-9572 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.15.2 >Reporter: mayki >Priority: Major > > Hello > I have upgraded NIFI 1.15.2 since 2022/01/05 > No issue until this night 2022/01/13 > * nifi version 1.15.2 > * jdk-1.8.0_311 > And the limit is high > {code:java} > Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2 > -bash-4.2@nifi$ ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) rg > pending signals (-i) 63278 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 5 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 1 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > {code} > > We got a lot error about provenance_repository, it fill our filesystem logs .. > > {code:java} > 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2] > o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events > org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877) > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891) > at > org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468) > at > org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444) > at > org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.nio.file.FileSystemException: > /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd: > Too many open files > {code} > > > We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability. > But it is impossible to do that if we got this error. > > Thanks for you help. > > Regards > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[ https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17492071#comment-17492071 ] Joe Witt commented on NIFI-9572: [~mayki] Please get the process ID of nifi. Then run 'lsof -p ' and capture the entire output. Please share that attached on here. It will almost always make clear what resource is the issue. I'll add that very often users think they've set the ulimit for max files correctly when they have not. You may well have it correct just a heads up. Lets see lsof. Thanks > Failed to index Provenance Events and (Too many Files) > -- > > Key: NIFI-9572 > URL: https://issues.apache.org/jira/browse/NIFI-9572 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.15.2 >Reporter: mayki >Priority: Major > > Hello > I have upgraded NIFI 1.15.2 since 2022/01/05 > No issue until this night 2022/01/13 > * nifi version 1.15.2 > * jdk-1.8.0_311 > And the limit is high > {code:java} > Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2 > -bash-4.2@nifi$ ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) rg > pending signals (-i) 63278 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 5 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 1 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > {code} > > We got a lot error about provenance_repository, it fill our filesystem logs .. > > {code:java} > 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2] > o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events > org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877) > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891) > at > org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468) > at > org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444) > at > org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.nio.file.FileSystemException: > /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd: > Too many open files > {code} > > > We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability. > But it is impossible to do that if we got this error. > > Thanks for you help. > > Regards > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[ https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17492062#comment-17492062 ] mayki commented on NIFI-9572: - Hello Anybody can help me ? All my instances NIFI with version 1.15.2 and 1.15.3 failed with "Too many open files" is seem a generic error message. Thanks for your helps > Failed to index Provenance Events and (Too many Files) > -- > > Key: NIFI-9572 > URL: https://issues.apache.org/jira/browse/NIFI-9572 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.15.2 >Reporter: mayki >Priority: Major > > Hello > I have upgraded NIFI 1.15.2 since 2022/01/05 > No issue until this night 2022/01/13 > * nifi version 1.15.2 > * jdk-1.8.0_311 > And the limit is high > {code:java} > Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2 > -bash-4.2@nifi$ ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) rg > pending signals (-i) 63278 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 5 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 1 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > {code} > > We got a lot error about provenance_repository, it fill our filesystem logs .. > > {code:java} > 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2] > o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events > org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877) > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891) > at > org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468) > at > org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444) > at > org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.nio.file.FileSystemException: > /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd: > Too many open files > {code} > > > We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability. > But it is impossible to do that if we got this error. > > Thanks for you help. > > Regards > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[ https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17480907#comment-17480907 ] mayki commented on NIFI-9572: - Hello Have you got any news? I have upgraded 10 NIFI instances in standalone or cluster node. All have the same issue {code:java} Caused by: java.nio.file.FileSystemException: ./provenance_repository/lucene-8-index-1637634601351/_14g8_Lucene80_0.dvd: Too many open files {code} And only workaround is remove all files in the repository, remove old logs, an restart the NIFI. Please help us > Failed to index Provenance Events and (Too many Files) > -- > > Key: NIFI-9572 > URL: https://issues.apache.org/jira/browse/NIFI-9572 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.15.2 >Reporter: mayki >Priority: Major > > Hello > I have upgraded NIFI 1.15.2 since 2022/01/05 > No issue until this night 2022/01/13 > * nifi version 1.15.2 > * jdk-1.8.0_311 > And the limit is high > {code:java} > Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2 > -bash-4.2@nifi$ ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) rg > pending signals (-i) 63278 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 5 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 1 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > {code} > > We got a lot error about provenance_repository, it fill our filesystem logs .. > > {code:java} > 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2] > o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events > org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877) > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891) > at > org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468) > at > org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444) > at > org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.nio.file.FileSystemException: > /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd: > Too many open files > {code} > > > We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability. > But it is impossible to do that if we got this error. > > Thanks for you help. > > Regards > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[ https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17477184#comment-17477184 ] mayki commented on NIFI-9572: - After restart We got this thing: {code:java} HTTP ERROR 500 javax.servlet.ServletException: org.apache.jasper.JasperException: java.lang.ClassNotFoundException: org.apache.jsp.WEB_002dINF.pages.canvas_jspURI:/nifi/STATUS:500MESSAGE:javax.servlet.ServletException: org.apache.jasper.JasperException: java.lang.ClassNotFoundException: org.apache.jsp.WEB_002dINF.pages.canvas_jspSERVLET:defaultCAUSED BY:javax.servlet.ServletException: org.apache.jasper.JasperException: java.lang.ClassNotFoundException: org.apache.jsp.WEB_002dINF.pages.canvas_jspCAUSED BY:org.apache.jasper.JasperException: java.lang.ClassNotFoundException: org.apache.jsp.WEB_002dINF.pages.canvas_jspCAUSED BY:java.lang.ClassNotFoundException: org.apache.jsp.WEB_002dINF.pages.canvas_jspCaused by:javax.servlet.ServletException: org.apache.jasper.JasperException: java.lang.ClassNotFoundException: org.apache.jsp.WEB_002dINF.pages.canvas_jsp at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191) at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:516) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.jasper.JasperException: java.lang.ClassNotFoundException: org.apache.jsp.WEB_002dINF.pages.canvas_jsp at org.apache.jasper.servlet.JspServletWrapper.getServlet(JspServletWrapper.java:196) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:412) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:383) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:331) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1459) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1631) at org.apache.nifi.web.filter.SanitizeContextPathFilter.doFilter(SanitizeContextPathFilter.java:58) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:487) at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:336) at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:301) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at
[jira] [Commented] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[ https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17477179#comment-17477179 ] mayki commented on NIFI-9572: - I also remove all files in the repository {code:java} rm -fr content_repository/* rm -fr database_repository/* rm -fr flowfile_repository/* rm -fr provenance_repository/* {code} Now it failed to start {code:java} 2022-01-17 13:36:01,040 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down. java.io.IOException: Too many open files at sun.nio.ch.IOUtil.makePipe(Native Method) at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:65) at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36) at java.nio.channels.Selector.open(Selector.java:227) at org.eclipse.jetty.io.SelectorManager.newSelector(SelectorManager.java:383) at org.eclipse.jetty.io.ManagedSelector.doStart(ManagedSelector.java:115) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.io.SelectorManager.doStart(SelectorManager.java:262) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110) at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:321) at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81) at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.eclipse.jetty.server.Server.doStart(Server.java:401) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1142) at org.apache.nifi.NiFi.(NiFi.java:170) at org.apache.nifi.NiFi.(NiFi.java:82) at org.apache.nifi.NiFi.main(NiFi.java:330) 2022-01-17 13:36:01,043 INFO [Thread-2] org.apache.nifi.NiFi Application Server shutdown started 2022-01-17 13:36:01,047 INFO [Thread-2] o.eclipse.jetty.server.AbstractConnector Stopped ServerConnector@4387d2f4{SSL, (ssl, http/1.1)}{s3575tos.mc2.renault.fr:9091} 2022-01-17 13:36:01,047 INFO [Thread-2] org.eclipse.jetty.server.session node0 Stopped scavenging {code} {code:java} # lsof -p 775865 | wc -l 8052 # ls -rtl /proc/775865/fd | wc -l 4097 {code} > Failed to index Provenance Events and (Too many Files) > -- > > Key: NIFI-9572 > URL: https://issues.apache.org/jira/browse/NIFI-9572 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.15.2 >Reporter: mayki >Priority: Major > > Hello > I have upgraded NIFI 1.15.2 since 2022/01/05 > No issue until this night 2022/01/13 > * nifi version 1.15.2 > * jdk-1.8.0_311 > And the limit is high > {code:java} > Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2 > -bash-4.2@nifi$ ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) rg > pending signals (-i) 63278 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 5 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 1 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > {code} > > We got a lot error about provenance_repository, it fill our filesystem logs .. > > {code:java} > 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2] > o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events > org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877) > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891) > at > org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468) > at >
[jira] [Commented] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[ https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17477120#comment-17477120 ] mayki commented on NIFI-9572: - [~joewitt] thanks for your comment but I'm already check these things: {code:java} -bash-4.2$ id uid=1001(nifi) gid=1001(nifi) groups=1001(nifi) -bash-4.2$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 63278 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 5 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 1 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited -bash-4.2$ ps -ef|grep nifi nifi 2317374 1 0 11:39 ? 00:00:00 /bin/sh /appl/nifi/nifi-current/bin/nifi.sh start nifi 2317376 2317374 0 11:39 ? 00:00:03 /appl/pkg/jdk-1.8.0_311/bin/java -cp /appl/nifi/nifi-1.15.2/conf:/appl/nifi/nifi-1.15.2/lib/bootstrap/* -Xms12m -Xmx24m -Dorg.apache.nifi.bootstrap.config.log.dir=/logs/nifi -Dorg.apache.nifi.bootstrap.config.pid.dir=/var/run/nifi -Dorg.apache.nifi.bootstrap.config.file=/appl/nifi/nifi-1.15.2/conf/bootstrap.conf org.apache.nifi.bootstrap.RunNiFi start nifi 2317425 2317376 57 11:39 ? 00:05:02 /appl/pkg/jdk-1.8.0_311/bin/java -classpath /appl/nifi/nifi-current/conf:/appl nifi/nifi-current/lib/javax.servlet-api-3.1.0.jar:/appl/nifi/nifi-current/lib/jetty-schemas-3.1.jar:/appl/nifi/nifi-current/lib/logback-classic-1.2.9.jar:/appl/nifi/nifi-current/lib/logback-core-1.2.9.jar:/appl/nifi/nifi-current/lib/jcl-over-slf4j-1.7.32.jar:/appl/nifi/nifi-current/lib/jul-to-slf4j-1.7.32.jar:/appl/nifi/nifi-current/lib/log4j-over-slf4j-1.7.32.jar:/appl/nifi/nifi-current/lib/slf4j-api-1.7.32.jar:/appl/nifi/nifi-current/lib/nifi-api-1.15.2.jar:/appl/nifi/nifi-current/lib/nifi-framework-api-1.15.2.jar:/appl/nifi/nifi-current/lib/nifi-server-api-1.15.2.jar:/appl/nifi/nifi-current/lib/nifi-runtime-1.15.2.jar:/appl/nifi/nifi-current/lib/nifi-nar-utils-1.15.2.jar:/appl/nifi/nifi-current/lib/nifi-properties-1.15.2.jar:/appl/nifi/nifi-current/lib/nifi-property-utils-1.15.2.jar:/appl/nifi/nifi-current/lib/nifi-stateless-bootstrap-1.15.2.jar:/appl/nifi/nifi-current/lib/nifi-stateless-api-1.15.2.jar -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx8192m -Xms8192m -Djavax.security.auth.useSubjectCredsOnly=true -Djava.security.egd=file:/dev/urandom -Dzookeeper.admin.enableServer=false -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djava.protocol.handler.pkgs=sun.net.www.protocol -Dnifi.properties.file.path=/appl/nifi/nifi-current/conf/nifi.properties -Dnifi.bootstrap.listen.port=20995 -Dapp=NiFi -Dorg.apache.nifi.bootstrap.config.log.dir=/logs/nifi org.apache.nifi.NiFi -K /appl/nifi/nifi-current/conf/sensitive.key {code} During running : {code:java} -bash-4.2$ lsof -p 2317425 | wc -l 7139 -bash-4.2$ ls -l /proc/2317425/fd | wc -l 3618 {code} > Failed to index Provenance Events and (Too many Files) > -- > > Key: NIFI-9572 > URL: https://issues.apache.org/jira/browse/NIFI-9572 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.15.2 >Reporter: mayki >Priority: Major > > Hello > I have upgraded NIFI 1.15.2 since 2022/01/05 > No issue until this night 2022/01/13 > * nifi version 1.15.2 > * jdk-1.8.0_311 > And the limit is high > {code:java} > Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2 > -bash-4.2@nifi$ ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) rg > pending signals (-i) 63278 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 5 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 1 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > {code} > > We got a lot error about provenance_repository, it fill our filesystem logs .. > > {code:java} > 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2] > o.a.n.p.index.lucene.EventIndexTask Failed
[jira] [Commented] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[ https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17476249#comment-17476249 ] Joe Witt commented on NIFI-9572: There are no known issues involving bugs in provenance leading to too many open files right now so probably this is solvable/configuration or the underlying issue is likely present in older versions too. First, we often have users report they are certain max open file settings are correct by them running ulimit -a but only then discover they ran it as a different user than what their nifi really launches as. Please double check this. Second, what output we see complain of open file exhaustion is often just the victim that hit the limit. There could be a totally different underlying root cause (socket exhaustion for instance). Run 'lsof -p ' to get a full listing of all the file handles nifi has. Run this right after a restart/nifi initially comes up. Then run it periodically over time. If there is indeed a leak it will become pretty obvious pretty fast in terms of the likely culprit. > Failed to index Provenance Events and (Too many Files) > -- > > Key: NIFI-9572 > URL: https://issues.apache.org/jira/browse/NIFI-9572 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.15.2 >Reporter: mayki >Priority: Major > > Hello > I have upgraded NIFI 1.15.2 since 2022/01/05 > No issue until this night 2022/01/13 > * nifi version 1.15.2 > * jdk-1.8.0_311 > And the limit is high > {code:java} > Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2 > -bash-4.2@nifi$ ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) rg > pending signals (-i) 63278 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 5 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 1 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > {code} > > We got a lot error about provenance_repository, it fill our filesystem logs .. > > {code:java} > 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2] > o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events > org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877) > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891) > at > org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468) > at > org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444) > at > org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202) > at > org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.nio.file.FileSystemException: > /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd: > Too many open files > {code} > > > We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability. > But it is impossible to do that if we got this error. > > Thanks for you help. > > Regards > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)