[jira] [Updated] (NIFI-9711) Support Flow JSON in set-sensitive-properties-key Command
[
https://issues.apache.org/jira/browse/NIFI-9711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-9711:
---
Affects Version/s: 1.16.0
Status: Patch Available (was: Open)
> Support Flow JSON in set-sensitive-properties-key Command
> -
>
> Key: NIFI-9711
> URL: https://issues.apache.org/jira/browse/NIFI-9711
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The {{SetSensitivePropertiesKey}} command class supports changing the
> {{nifi.sensitive.props.key}} property by decrypting and encrypting sensitive
> values in the serialized NiFi Flow configuration.
> Now that NiFi supports both XML and JSON for storing the flow configuration,
> the command should be updated to check for the existence of both files and
> update the JSON version as well as the XML version when found.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [nifi] exceptionfactory opened a new pull request #5783: NIFI-9711 Add support for flow.json.gz in SetSensitivePropertiesKey
exceptionfactory opened a new pull request #5783:
URL: https://github.com/apache/nifi/pull/5783
Description of PR
NIFI-9711 Adds support for processing the flow configuration stored in
`flow.json.gz` when running the `set-sensitive-properties-key` command.
Changes for [NIFI-9069](https://issues.apache.org/jira/browse/NIFI-9069) in
PR #5514 introduced support for persisting the NiFi flow configuration in both
XML and JSON.
Updates to the `set-sensitive-properties-key` command support processing
both the XML and JSON configuration files when found, providing compatibility
with earlier versions.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [X] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [X] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [X] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [X] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NIFI-9711) Support Flow JSON in set-sensitive-properties-key Command
David Handermann created NIFI-9711:
--
Summary: Support Flow JSON in set-sensitive-properties-key Command
Key: NIFI-9711
URL: https://issues.apache.org/jira/browse/NIFI-9711
Project: Apache NiFi
Issue Type: Improvement
Reporter: David Handermann
Assignee: David Handermann
The {{SetSensitivePropertiesKey}} command class supports changing the
{{nifi.sensitive.props.key}} property by decrypting and encrypting sensitive
values in the serialized NiFi Flow configuration.
Now that NiFi supports both XML and JSON for storing the flow configuration,
the command should be updated to check for the existence of both files and
update the JSON version as well as the XML version when found.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [nifi] exceptionfactory commented on a change in pull request #5732: NIFI-9647 Added ExtractDocumentText which is a processor based on Tim…
exceptionfactory commented on a change in pull request #5732:
URL: https://github.com/apache/nifi/pull/5732#discussion_r810323812
##
File path:
nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/document/ExtractDocumentText.java
##
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.document;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.tika.Tika;
+import org.apache.tika.exception.TikaException;
+
+import java.io.BufferedInputStream;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+
+@Tags({"extract, text, pdf, word, excel, powerpoint, office"})
+@CapabilityDescription("Run Apache Tika text extraction to extra the text from
supported binary file formats such as PDF " +
+"and Microsoft Office files.")
+public class ExtractDocumentText extends AbstractProcessor {
+private static final String TEXT_PLAIN = "text/plain";
+
+public static final String FIELD_MAX_TEXT_LENGTH = "max-text-length";
+public static final String FIELD_SUCCESS = "success";
+public static final String FIELD_FAILURE = "failure";
+
+public static final PropertyDescriptor MAX_TEXT_LENGTH = new
PropertyDescriptor.Builder()
+.name(FIELD_MAX_TEXT_LENGTH)
+.displayName("Max Output Text Length")
+.description("The maximum length of text to retrieve. This is used
to limit memory usage for " +
+"dealing with large files. Specify -1 for unlimited
length.")
+
.required(false).defaultValue("-1").addValidator(StandardValidators.INTEGER_VALIDATOR)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES).build();
+
+public static final Relationship REL_SUCCESS = new
Relationship.Builder().name(FIELD_SUCCESS)
+.description("Content extraction successful").build();
+
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name(FIELD_FAILURE)
+.description("Content extraction failed").build();
+
+private List descriptors =
Collections.unmodifiableList(Arrays.asList(MAX_TEXT_LENGTH));
+private Set relationships = Collections.unmodifiableSet(new
HashSet<>(Arrays.asList(REL_SUCCESS, REL_FAILURE)));
+
+@Override
+public Set getRelationships() {
+return this.relationships;
+}
+
+@Override
+public final List getSupportedPropertyDescriptors() {
+return descriptors;
+}
+
+@Override
+public void onTrigger(final ProcessContext context, final ProcessSession
session) throws ProcessException {
+FlowFile flowFile = session.get();
+if (flowFile == null) {
+return;
+}
+
+final int maxTextLength =
context.getProperty(MAX_TEXT_LENGTH).evaluateAttributeExpressions(flowFile).asInteger();
+final String filename = flowFile.getAttribute("filename");
+
+try {
+final AtomicReference type = new AtomicReference<>();
+final AtomicReference exceptionThrown = new
AtomicReference<>(false);
+
+flowFile = session.write(flowFile, (inputStream, outputStream) -> {
+BufferedInputStream buffStream = new
BufferedInputStream(inputStream);
+Tika tika = new Tika();
+String text = "";
+try {
+type.set(tika.detect(buffStream, filename));
+
[jira] [Resolved] (NIFI-7328) Improve OIDC Identity Provider
[ https://issues.apache.org/jira/browse/NIFI-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-7328. Fix Version/s: 1.16.0 Assignee: Nathan Gough (was: David Handermann) Resolution: Fixed > Improve OIDC Identity Provider > -- > > Key: NIFI-7328 > URL: https://issues.apache.org/jira/browse/NIFI-7328 > Project: Apache NiFi > Issue Type: Epic > Components: Core Framework, Extensions >Affects Versions: 1.11.4 >Reporter: Andy LoPresto >Assignee: Nathan Gough >Priority: Major > Labels: authentication, identity, keystore, logging, oidc, > security, tls > Fix For: 1.16.0 > > > A number of issues with the OIDC identity provider have been discovered > recently. > * The logging is insufficient to debug issues with the IdP > * It does not use the NiFi keystore & truststore but rather the JVM default > * There are grammatical and syntactic errors in log and error messages > * It may not process all claims in the IdP response correctly -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (NIFI-7333) OIDC provider should use NiFi keystore & truststore
[ https://issues.apache.org/jira/browse/NIFI-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-7333. Fix Version/s: 1.16.0 Resolution: Fixed > OIDC provider should use NiFi keystore & truststore > --- > > Key: NIFI-7333 > URL: https://issues.apache.org/jira/browse/NIFI-7333 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Security >Affects Versions: 1.11.4 >Reporter: Andy LoPresto >Assignee: Nathan Gough >Priority: Major > Labels: keystore, oidc, security, tls > Fix For: 1.16.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > The OIDC provider uses generic HTTPS requests to the OIDC IdP, but does not > configure these requests to use the NiFi keystore or truststore. Rather, it > uses the default JVM keystore and truststore, which leads to difficulty > debugging PKIX and other TLS negotiation errors. It should be switched to use > the NiFi keystore and truststore as other NiFi framework services do. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-7333) OIDC provider should use NiFi keystore & truststore
[ https://issues.apache.org/jira/browse/NIFI-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17494813#comment-17494813 ] ASF subversion and git services commented on NIFI-7333: --- Commit 7ef2fd2986461b632e398b7ddff47db153c3c0e5 in nifi's branch refs/heads/main from Nathan Gough [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=7ef2fd2 ] NIFI-7333 Added OIDC trust store strategy property This closes #5753 Signed-off-by: David Handermann > OIDC provider should use NiFi keystore & truststore > --- > > Key: NIFI-7333 > URL: https://issues.apache.org/jira/browse/NIFI-7333 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Security >Affects Versions: 1.11.4 >Reporter: Andy LoPresto >Assignee: Nathan Gough >Priority: Major > Labels: keystore, oidc, security, tls > Time Spent: 20m > Remaining Estimate: 0h > > The OIDC provider uses generic HTTPS requests to the OIDC IdP, but does not > configure these requests to use the NiFi keystore or truststore. Rather, it > uses the default JVM keystore and truststore, which leads to difficulty > debugging PKIX and other TLS negotiation errors. It should be switched to use > the NiFi keystore and truststore as other NiFi framework services do. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi] asfgit closed pull request #5753: NIFI-7333 - Added a new property to allow specifying whether to use N…
asfgit closed pull request #5753: URL: https://github.com/apache/nifi/pull/5753 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-7192) nifi.sh install not working on systemd servers
[
https://issues.apache.org/jira/browse/NIFI-7192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17494791#comment-17494791
]
ASF subversion and git services commented on NIFI-7192:
---
Commit a8fd5993eb164d2b8b1d531ecf9b95e79232404f in nifi's branch
refs/heads/main from Roberto Marturano
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=a8fd599 ]
NIFI-7192 Added systemd reload to nifi.sh install on systemd servers
This closes #4079
Signed-off-by: Joey Frazee
> nifi.sh install not working on systemd servers
> --
>
> Key: NIFI-7192
> URL: https://issues.apache.org/jira/browse/NIFI-7192
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Roberto Marturano
>Assignee: Roberto Marturano
>Priority: Major
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Systemd is still honouring init.d services, but we need to trigger the
> {{systemd-sysv-generator}} in order to create a .service wrapper.
> Right now works like this:
>
> {code:java}
> $ ./bin/nifi.sh install nifi
> Service nifi installed
> $ service nifi start
> Failed to start nifi.service: Unit nifi.service not found.
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [nifi] jfrazee commented on pull request #4079: NIFI-7192 - nifi.sh install not working on systemd servers
jfrazee commented on pull request #4079: URL: https://github.com/apache/nifi/pull/4079#issuecomment-1045049299 @MikeThomsen Was just trying to snag some time to test on something besides Ubuntu. I validated it on Ubuntu 20.04 and OpenSuSE 15.3. This fix works as expected for both. There was a wrinkle with OpenSuSE -- they removed the rc*.d dirs entirely now so need to open another issue for that. This doesn't have anything to do with this PR though so merged. Thanks @rmarturano -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] jfrazee closed pull request #4079: NIFI-7192 - nifi.sh install not working on systemd servers
jfrazee closed pull request #4079: URL: https://github.com/apache/nifi/pull/4079 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[
https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17494609#comment-17494609
]
mayki edited comment on NIFI-9572 at 2/18/22, 1:31 PM:
---
[^bootstrap.conf]
[^bootstrap.conf][^bootstrap.conf][^bootstrap.conf]
[^nifi.properties]
was (Author: mayki):
[^bootstrap.conf]
> Failed to index Provenance Events and (Too many Files)
> --
>
> Key: NIFI-9572
> URL: https://issues.apache.org/jira/browse/NIFI-9572
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
>Affects Versions: 1.15.2
>Reporter: mayki
>Priority: Major
> Attachments: bootstrap.conf, nifi-app.log, nifi-app.log.tar.gz,
> nifi.properties, nifi_691106_pid.tar.gz
>
>
> Hello
> I have upgraded NIFI 1.15.2 since 2022/01/05
> No issue until this night 2022/01/13
> * nifi version 1.15.2
> * jdk-1.8.0_311
> And the limit is high
> {code:java}
> Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2
> -bash-4.2@nifi$ ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) rg
> pending signals (-i) 63278
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 5
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 1
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
> {code}
>
> We got a lot error about provenance_repository, it fill our filesystem logs ..
>
> {code:java}
> 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2]
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877)
> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891)
> at
> org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468)
> at
> org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444)
> at
> org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70)
> at
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202)
> at
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.nio.file.FileSystemException:
> /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd:
> Too many open files
> {code}
>
>
> We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability.
> But it is impossible to do that if we got this error.
>
> Thanks for you help.
>
> Regards
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[
https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17494609#comment-17494609
]
mayki commented on NIFI-9572:
-
[^bootstrap.conf]
> Failed to index Provenance Events and (Too many Files)
> --
>
> Key: NIFI-9572
> URL: https://issues.apache.org/jira/browse/NIFI-9572
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
>Affects Versions: 1.15.2
>Reporter: mayki
>Priority: Major
> Attachments: bootstrap.conf, nifi-app.log, nifi-app.log.tar.gz,
> nifi.properties, nifi_691106_pid.tar.gz
>
>
> Hello
> I have upgraded NIFI 1.15.2 since 2022/01/05
> No issue until this night 2022/01/13
> * nifi version 1.15.2
> * jdk-1.8.0_311
> And the limit is high
> {code:java}
> Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2
> -bash-4.2@nifi$ ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) rg
> pending signals (-i) 63278
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 5
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 1
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
> {code}
>
> We got a lot error about provenance_repository, it fill our filesystem logs ..
>
> {code:java}
> 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2]
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877)
> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891)
> at
> org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468)
> at
> org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444)
> at
> org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70)
> at
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202)
> at
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.nio.file.FileSystemException:
> /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd:
> Too many open files
> {code}
>
>
> We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability.
> But it is impossible to do that if we got this error.
>
> Thanks for you help.
>
> Regards
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Comment Edited] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[
https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17494609#comment-17494609
]
mayki edited comment on NIFI-9572 at 2/18/22, 1:31 PM:
---
[^bootstrap.conf]
[^nifi.properties]
was (Author: mayki):
[^bootstrap.conf]
[^bootstrap.conf][^bootstrap.conf][^bootstrap.conf]
[^nifi.properties]
> Failed to index Provenance Events and (Too many Files)
> --
>
> Key: NIFI-9572
> URL: https://issues.apache.org/jira/browse/NIFI-9572
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
>Affects Versions: 1.15.2
>Reporter: mayki
>Priority: Major
> Attachments: bootstrap.conf, nifi-app.log, nifi-app.log.tar.gz,
> nifi.properties, nifi_691106_pid.tar.gz
>
>
> Hello
> I have upgraded NIFI 1.15.2 since 2022/01/05
> No issue until this night 2022/01/13
> * nifi version 1.15.2
> * jdk-1.8.0_311
> And the limit is high
> {code:java}
> Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2
> -bash-4.2@nifi$ ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) rg
> pending signals (-i) 63278
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 5
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 1
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
> {code}
>
> We got a lot error about provenance_repository, it fill our filesystem logs ..
>
> {code:java}
> 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2]
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877)
> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891)
> at
> org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468)
> at
> org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444)
> at
> org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70)
> at
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202)
> at
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.nio.file.FileSystemException:
> /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd:
> Too many open files
> {code}
>
>
> We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability.
> But it is impossible to do that if we got this error.
>
> Thanks for you help.
>
> Regards
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[
https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mayki updated NIFI-9572:
Attachment: bootstrap.conf
nifi.properties
> Failed to index Provenance Events and (Too many Files)
> --
>
> Key: NIFI-9572
> URL: https://issues.apache.org/jira/browse/NIFI-9572
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
>Affects Versions: 1.15.2
>Reporter: mayki
>Priority: Major
> Attachments: bootstrap.conf, nifi-app.log, nifi-app.log.tar.gz,
> nifi.properties, nifi_691106_pid.tar.gz
>
>
> Hello
> I have upgraded NIFI 1.15.2 since 2022/01/05
> No issue until this night 2022/01/13
> * nifi version 1.15.2
> * jdk-1.8.0_311
> And the limit is high
> {code:java}
> Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2
> -bash-4.2@nifi$ ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) rg
> pending signals (-i) 63278
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 5
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 1
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
> {code}
>
> We got a lot error about provenance_repository, it fill our filesystem logs ..
>
> {code:java}
> 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2]
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877)
> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891)
> at
> org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468)
> at
> org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444)
> at
> org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70)
> at
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202)
> at
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.nio.file.FileSystemException:
> /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd:
> Too many open files
> {code}
>
>
> We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability.
> But it is impossible to do that if we got this error.
>
> Thanks for you help.
>
> Regards
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[
https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17494606#comment-17494606
]
mayki commented on NIFI-9572:
-
Hello [~joewitt]
I pushed the information in the files:
* NIFI version 1.15.3
* JAVA /appl/pkg/jdk-1.8.0_311/bin/java
* nifi_691106_pid.tar.gz ==> resultat of losf -p pid
* nifi-app.log.tar.gz ==> nifi-app.log
{code:java}
-bash-4.2$ whoami
nifi
-bash-4.2$ ps -ef|grep nifi
nifi 691085 1 0 Feb11 ? 00:00:00 /bin/sh
/appl/nifi/nifi-current/bin/nifi.sh start
nifi 691087 691085 0 Feb11 ? 00:07:44
/appl/pkg/jdk-1.8.0_311/bin/java -cp
/appl/nifi/nifi-1.15.3/conf:/appl/nifi/nifi-1.15.3/lib/bootstrap/* -Xms12m
-Xmx24m -Dorg.apache.nifi.bootstrap.config.log.dir=/logs/nifi
-Dorg.apache.nifi.bootstrap.config.pid.dir=/var/run/nifi
-Dorg.apache.nifi.bootstrap.config.file=/appl/nifi/nifi-1.15.3/conf/bootstrap.conf
org.apache.nifi.bootstrap.RunNiFi start
nifi 691106 691087 36 Feb11 ? 2-13:36:35
/appl/pkg/jdk-1.8.0_311/bin/java -classpath
/appl/nifi/nifi-current/conf:/appl/nifi/nifi-current/lib/javax.servlet-api-3.1.0.jar:/appl/nifi/nifi-current/lib/jetty-schemas-3.1.jar:/appl/nifi/nifi-current/lib/logback-classic-1.2.10.jar:/appl/nifi/nifi-current/lib/logback-core-1.2.10.jar:/appl/nifi/nifi-current/lib/jcl-over-slf4j-1.7.32.jar:/appl/nifi/nifi-current/lib/jul-to-slf4j-1.7.32.jar:/appl/nifi/nifi-current/lib/log4j-over-slf4j-1.7.32.jar:/appl/nifi/nifi-current/lib/slf4j-api-1.7.32.jar:/appl/nifi/nifi-current/lib/nifi-api-1.15.3.jar:/appl/nifi/nifi-current/lib/nifi-framework-api-1.15.3.jar:/appl/nifi/nifi-current/lib/nifi-server-api-1.15.3.jar:/appl/nifi/nifi-current/lib/nifi-runtime-1.15.3.jar:/appl/nifi/nifi-current/lib/nifi-nar-utils-1.15.3.jar:/appl/nifi/nifi-current/lib/nifi-properties-1.15.3.jar:/appl/nifi/nifi-current/lib/nifi-property-utils-1.15.3.jar:/appl/nifi/nifi-current/lib/nifi-stateless-bootstrap-1.15.3.jar:/appl/nifi/nifi-current/lib/nifi-stateless-api-1.15.3.jar
-Dorg.apache.jasper.compiler.disablejsr199=true -Xmx8192m -Xms8192m
-Djavax.security.auth.useSubjectCredsOnly=true
-Djava.security.egd=file:/dev/urandom -Dzookeeper.admin.enableServer=false
-Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true
-Djava.awt.headless=true -Djava.protocol.handler.pkgs=sun.net.www.protocol
-Dnifi.properties.file.path=/appl/nifi/nifi-current/conf/nifi.properties
-Dnifi.bootstrap.listen.port=27307 -Dapp=NiFi
-Dorg.apache.nifi.bootstrap.config.log.dir=/logs/nifi org.apache.nifi.NiFi -K
/appl/nifi/nifi-current/conf/sensitive.key-bash-4.2$
/appl/pkg/jdk-1.8.0_311/bin/java -version
java version "1.8.0_311"
Java(TM) SE Runtime Environment (build 1.8.0_311-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.311-b11, mixed mode)
-bash-4.2$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 63278
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 5
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 1
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited {code}
[^nifi_691106_pid.tar.gz]
[^nifi-app.log.tar.gz]
Thanks
> Failed to index Provenance Events and (Too many Files)
> --
>
> Key: NIFI-9572
> URL: https://issues.apache.org/jira/browse/NIFI-9572
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
>Affects Versions: 1.15.2
>Reporter: mayki
>Priority: Major
> Attachments: nifi-app.log, nifi-app.log.tar.gz, nifi_691106_pid.tar.gz
>
>
> Hello
> I have upgraded NIFI 1.15.2 since 2022/01/05
> No issue until this night 2022/01/13
> * nifi version 1.15.2
> * jdk-1.8.0_311
> And the limit is high
> {code:java}
> Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2
> -bash-4.2@nifi$ ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) rg
> pending signals (-i) 63278
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 5
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes
[jira] [Updated] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[
https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mayki updated NIFI-9572:
Attachment: nifi_691106_pid.tar.gz
nifi-app.log.tar.gz
> Failed to index Provenance Events and (Too many Files)
> --
>
> Key: NIFI-9572
> URL: https://issues.apache.org/jira/browse/NIFI-9572
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
>Affects Versions: 1.15.2
>Reporter: mayki
>Priority: Major
> Attachments: nifi-app.log, nifi-app.log.tar.gz, nifi_691106_pid.tar.gz
>
>
> Hello
> I have upgraded NIFI 1.15.2 since 2022/01/05
> No issue until this night 2022/01/13
> * nifi version 1.15.2
> * jdk-1.8.0_311
> And the limit is high
> {code:java}
> Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2
> -bash-4.2@nifi$ ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) rg
> pending signals (-i) 63278
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 5
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 1
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
> {code}
>
> We got a lot error about provenance_repository, it fill our filesystem logs ..
>
> {code:java}
> 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2]
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877)
> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891)
> at
> org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468)
> at
> org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444)
> at
> org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70)
> at
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202)
> at
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.nio.file.FileSystemException:
> /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd:
> Too many open files
> {code}
>
>
> We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability.
> But it is impossible to do that if we got this error.
>
> Thanks for you help.
>
> Regards
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [nifi] MikeThomsen commented on pull request #4079: NIFI-7192 - nifi.sh install not working on systemd servers
MikeThomsen commented on pull request #4079: URL: https://github.com/apache/nifi/pull/4079#issuecomment-1044482037 @jfrazee why did you hold off on merge after approving? Was more work required? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] gresockj commented on a change in pull request #5665: NIFI-9581 add PutElasticsearchRecord relationship for output of successful Records sent to Elasticsearch
gresockj commented on a change in pull request #5665:
URL: https://github.com/apache/nifi/pull/5665#discussion_r809891315
##
File path:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchJson.java
##
@@ -95,27 +101,36 @@
.name("put-es-json-error-documents")
.displayName("Output Error Documents")
.description("If this configuration property is true, the response
from Elasticsearch will be examined for failed documents " +
-"and the failed documents will be sent to the \"errors\"
relationship.")
+"and the failed documents will be sent to the \"" +
REL_FAILED_DOCUMENTS.getName() + "\" relationship.")
.allowableValues("true", "false")
.defaultValue("false")
.expressionLanguageSupported(ExpressionLanguageScope.NONE)
.required(true)
.build();
-static final Relationship REL_FAILED_DOCUMENTS = new Relationship.Builder()
-.name("errors").description("If \"" +
OUTPUT_ERROR_DOCUMENTS.getDisplayName() + "\" is set, " +
-"any FlowFile that failed to process the way it was
configured will be sent to this relationship " +
-"as part of a failed document set.")
-.autoTerminateDefault(true).build();
+static final PropertyDescriptor NOT_FOUND_IS_SUCCESSFUL = new
PropertyDescriptor.Builder()
+.name("put-es-not_found-is-error")
+.displayName("Treat \"Not Found\" as Error")
+.description("If \"" + OUTPUT_ERROR_DOCUMENTS.getName() + "\" is true,
\"not_found\" Elasticsearch Documents " +
Review comment:
Since this depends on OUTPUT_ERROR_DOCUMENTS, I don't think you need to
add the conditional language here, since any time this property is exposed, you
have OUTPUT_ERROR_DOCUMENTS set to true. Instead, I think this could be more
clear by saying: "If true, 'not_found' Elasticsearch Documents will be routed
to the 'success' relationship; otherwise they are routed to 'error'."
##
File path:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchRecord.java
##
@@ -86,6 +96,16 @@
resource = SystemResource.MEMORY,
description = "The Batch of Records will be stored in memory until the
bulk operation is performed.")
public class PutElasticsearchRecord extends AbstractPutElasticsearch {
+static final Relationship REL_FAILED_RECORDS = new Relationship.Builder()
+.name("errors").description("If a \"Result Record Writer\" is set,
any record that failed to process the way it was " +
+"configured will be sent to this relationship as part of a
failed record set.")
+.autoTerminateDefault(true).build();
+
+static final Relationship REL_SUCCESSFUL_RECORDS = new
Relationship.Builder()
+.name("successful").description("If a \"Result Record Writer\" is
set, any record that successfully processed the way it was " +
Review comment:
Could use the same language as suggested above
##
File path:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchRecord.java
##
@@ -86,6 +96,16 @@
resource = SystemResource.MEMORY,
description = "The Batch of Records will be stored in memory until the
bulk operation is performed.")
public class PutElasticsearchRecord extends AbstractPutElasticsearch {
+static final Relationship REL_FAILED_RECORDS = new Relationship.Builder()
+.name("errors").description("If a \"Result Record Writer\" is set,
any record that failed to process the way it was " +
Review comment:
I find this "the way it was configured" language confusing. Since I
think this relationship is about capturing Record Writer errors, we might take
the opportunity to update this to:
"If a 'Result Record Writer' is set, any record that fails during Record
Writing will be sent to this relationship as part of a failed record set."
##
File path:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchRecord.java
##
@@ -182,17 +202,31 @@
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
.build();
-static final PropertyDescriptor ERROR_RECORD_WRITER = new
PropertyDescriptor.Builder()
+static final PropertyDescriptor RESULT_RECORD_WRITER = new
PropertyDescriptor.Builder()
.name("put-es-record-error-writer")
-.displayName("Error Record Writer")
+.displayName("Result Record Writer")
.description("If this configuration property is set, the response from
Elasticsearch w
[GitHub] [nifi] lawye1973 commented on pull request #5778: NIFI-9696 : DeleteS3Object don't have provenance
lawye1973 commented on pull request #5778: URL: https://github.com/apache/nifi/pull/5778#issuecomment-1044131491 > d...@nifi.apache.org just sent again. > Sent: Friday, February 18, 2022 4:22 PM > To: d...@nifi.apache.org > Subject: Re: Request for "Jira contributor access" > > resending > > Sent: Wednesday, February 16, 2022 11:18 PM > To: d...@nifi.apache.org > Subject: Request for "Jira contributor access" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
