[jira] [Comment Edited] (NIFI-9574) Failed to decrypt data from Peer
[
https://issues.apache.org/jira/browse/NIFI-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17476325#comment-17476325
]
Mahieddine Cherif edited comment on NIFI-9574 at 1/14/22, 10:06 PM:
Yeah we have a quiet constraint env :/ but the thing also is that we didn't
have this behaviour before and we have quiet light flows only light api calls
mainly.
Here are some system diagnostics captures, thanks a lot !
!Screenshot 2022-01-14 at 19.21.11.png|width=478,height=316!
!Screenshot 2022-01-14 at 19.21.16.png|width=447,height=147!
!Screenshot 2022-01-14 at 19.21.46.png|width=491,height=288! !Screenshot
2022-01-14 at 19.21.36.png|width=521,height=339!
was (Author: mahieddine):
Yeah we have a quiet constraint env :/ but the thing also is that we didn't
have this behaviour before and we have quiet light flows only light api calls
mainly.
Here are some system diagnostics captures, thanks a lot !
!Screenshot 2022-01-14 at 19.21.11.png|width=478,height=316! !Screenshot
2022-01-14 at 19.21.46.png|width=491,height=288! !Screenshot 2022-01-14 at
19.21.36.png|width=521,height=339! !Screenshot 2022-01-14 at
19.21.16.png|width=447,height=147!
> Failed to decrypt data from Peer
> -
>
> Key: NIFI-9574
> URL: https://issues.apache.org/jira/browse/NIFI-9574
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Mahieddine Cherif
>Priority: Major
> Attachments: Screenshot 2022-01-14 at 19.21.11.png, Screenshot
> 2022-01-14 at 19.21.16.png, Screenshot 2022-01-14 at 19.21.36.png, Screenshot
> 2022-01-14 at 19.21.46.png
>
>
> After a migration to 1.15.2 it seems like we have almost systematically this
> error on our cluster all the time
> {code:java}
> Failed to communicate with Peer
> nifi-0.nifi-headless.apache-nifi.svc.cluster.local:8443 when load balancing
> data for Connection with ID b76e7297-e8a0-3b2b-ba30-d338db411301 due to
> java.io.IOException: Failed to decrypt data from Peer
> nifi-0.nifi-headless.apache-nifi.svc.cluster.local:8443 because Peer
> unexpectedly closed connection
> {code}
> Files get stuck in the queue mentioned, it's not always the same, these are
> simple round robin queues with or without compression.
> When we recreate the cluster it goes for like some time and then it occurs
> again and again
> Is there a particular reason ?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [nifi] Dye357 opened a new pull request #5660: NIFI-9568 - Jolt .war file cleanup.
Dye357 opened a new pull request #5660: URL: https://github.com/apache/nifi/pull/5660 Thank you for submitting a contribution to Apache NiFi. Please provide a short description of the PR here: Updating pom.xml to only include *.css and *.js assets in the output .war file. fixes bug NIFI-9568. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [Y] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [Y] Does your PR title start with **NIFI-** where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [Y] Has your PR been rebased against the latest commit within the target branch (typically `main`)? - [Y] Is your initial contribution a single, squashed commit? _Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not `squash` or use `--force` when pushing to allow for clean monitoring of changes._ ### For code changes: - [Y] Have you ensured that the full suite of tests is executed via `mvn -Pcontrib-check clean install` at the root `nifi` folder? - [N/A] Have you written or updated unit tests to verify your changes? - [Y] Have you verified that the full build is successful on JDK 8? - [Y] Have you verified that the full build is successful on JDK 11? - [N/A ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [N/A ] If applicable, have you updated the `LICENSE` file, including the main `LICENSE` file under `nifi-assembly`? - [N/A] If applicable, have you updated the `NOTICE` file, including the main `NOTICE` file found under `nifi-assembly`? - [N/A] If adding new Properties, have you added `.displayName` in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ N/A] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a change in pull request #5324: NIFI-9072: improvements to ValidateXML including validate XML in attr…
exceptionfactory commented on a change in pull request #5324:
URL: https://github.com/apache/nifi/pull/5324#discussion_r785171728
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ValidateXml.java
##
@@ -64,26 +72,40 @@
@WritesAttribute(attribute = "validatexml.invalid.error", description =
"If the flow file is routed to the invalid relationship "
+ "the attribute will contain the error message resulting from the
validation failure.")
})
-@CapabilityDescription("Validates the contents of FlowFiles against a
user-specified XML Schema file")
+@CapabilityDescription("Validates XML contained in a FlowFile. By default, the
XML is contained in the FlowFile content. If the 'XML Source Attribute'
property is set, the XML to be validated "
++ "is contained in the specified attribute. It is not recommended to
use attributes to hold large XML documents; doing so could adversely affect
system performance. "
++ "Full schema validation is performed if the processor is configured
with the XSD schema details. Otherwise, the only validation performed is "
++ "to ensure the XML syntax is correct and well-formed, e.g. all
opening tags are properly closed.")
+@SystemResourceConsideration(resource = SystemResource.MEMORY, description =
"While this processor supports processing XML within attributes, it is strongly
discouraged to hold "
++ "large amounts of data in attributes. In general, attribute values
should be as small as possible and hold no more than a couple hundred
characters.")
public class ValidateXml extends AbstractProcessor {
public static final String ERROR_ATTRIBUTE_KEY =
"validatexml.invalid.error";
public static final PropertyDescriptor SCHEMA_FILE = new
PropertyDescriptor.Builder()
.name("Schema File")
-.description("The path to the Schema file that is to be used for
validation")
-.required(true)
+.displayName("Schema File")
+.description("The file path or URL to the XSD Schema file that is
to be used for validation. If this property is blank, only XML syntax/structure
will be validated.")
+.required(false)
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
.identifiesExternalResource(ResourceCardinality.SINGLE,
ResourceType.FILE, ResourceType.URL)
.build();
+public static final PropertyDescriptor XML_SOURCE_ATTRIBUTE = new
PropertyDescriptor.Builder()
+.name("XML Source Attribute")
Review comment:
Rather than making this property an attribute name, what do you think
about changing it to `Schema` and supporting expression language using FlowFile
attributes? That would allow direct configuration of the Schema as a processor
property, while also giving the option for FlowFile attribute based validation.
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ValidateXml.java
##
@@ -134,32 +157,73 @@ public void onTrigger(final ProcessContext context, final
ProcessSession session
}
final Schema schema = schemaRef.get();
-final Validator validator = schema.newValidator();
+final Validator validator = schema == null ? null :
schema.newValidator();
final ComponentLog logger = getLogger();
+final boolean attributeContainsXML =
context.getProperty(XML_SOURCE_ATTRIBUTE).isSet();
for (FlowFile flowFile : flowFiles) {
final AtomicBoolean valid = new AtomicBoolean(true);
-final AtomicReference exception = new
AtomicReference(null);
-
-session.read(flowFile, new InputStreamCallback() {
-@Override
-public void process(final InputStream in) throws IOException {
-try {
-validator.validate(new StreamSource(in));
-} catch (final IllegalArgumentException | SAXException e) {
-valid.set(false);
-exception.set(e);
+final AtomicReference exception = new
AtomicReference<>(null);
+SafeXMLConfiguration safeXMLConfiguration = new
SafeXMLConfiguration();
+safeXMLConfiguration.setValidating(false);
+
+try {
+DocumentBuilder docBuilder =
safeXMLConfiguration.createDocumentBuilder();
+
+if (attributeContainsXML) {
+// If XML source attribute is set, validate attribute value
+String xml =
flowFile.getAttribute(context.getProperty(XML_SOURCE_ATTRIBUTE).evaluateAttributeExpressions().getValue());
+ByteArrayInputStream bais = new
ByteArrayInputStream(xml.getBytes(StandardCharsets.UTF_8));
+
+if (validator != n
[GitHub] [nifi] ChrisSamo632 commented on a change in pull request #5659: NIFI-9570 Updating AWS SDK version for NiFi Registry AWS extension mo…
ChrisSamo632 commented on a change in pull request #5659: URL: https://github.com/apache/nifi/pull/5659#discussion_r785136847 ## File path: nifi-registry/nifi-registry-extensions/nifi-registry-aws/pom.xml ## @@ -30,7 +30,7 @@ -2.5.9 +2.17.106 Review comment: If this needs to match another version value but the two can't use the same maven property (because they're not sharing a parent), is it worth adding a comment here and in the other `pom.xml` so there's more chance of people realising that updating one means they need to update the other? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (NIFI-9574) Failed to decrypt data from Peer
[
https://issues.apache.org/jira/browse/NIFI-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17476325#comment-17476325
]
Mahieddine Cherif edited comment on NIFI-9574 at 1/14/22, 6:25 PM:
---
Yeah we have a quiet constraint env :/ but the thing also is that we didn't
have this behaviour before and we have quiet light flows only light api calls
mainly.
Here are some system diagnostics captures, thanks a lot !
!Screenshot 2022-01-14 at 19.21.11.png|width=478,height=316! !Screenshot
2022-01-14 at 19.21.46.png|width=491,height=288! !Screenshot 2022-01-14 at
19.21.36.png|width=521,height=339! !Screenshot 2022-01-14 at
19.21.16.png|width=447,height=147!
was (Author: mahieddine):
Yeah we have a quiet constraint env :/ but the thing also is that we didn't
have this behaviour before and we have quiet light flows only light api calls
mainly.
Here are some system diagnostics captures, thanks a lot !
!Screenshot 2022-01-14 at 19.21.11.png! !Screenshot 2022-01-14 at
19.21.46.png! !Screenshot 2022-01-14 at 19.21.36.png! !Screenshot 2022-01-14
at 19.21.16.png!
> Failed to decrypt data from Peer
> -
>
> Key: NIFI-9574
> URL: https://issues.apache.org/jira/browse/NIFI-9574
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Mahieddine Cherif
>Priority: Major
> Attachments: Screenshot 2022-01-14 at 19.21.11.png, Screenshot
> 2022-01-14 at 19.21.16.png, Screenshot 2022-01-14 at 19.21.36.png, Screenshot
> 2022-01-14 at 19.21.46.png
>
>
> After a migration to 1.15.2 it seems like we have almost systematically this
> error on our cluster all the time
> {code:java}
> Failed to communicate with Peer
> nifi-0.nifi-headless.apache-nifi.svc.cluster.local:8443 when load balancing
> data for Connection with ID b76e7297-e8a0-3b2b-ba30-d338db411301 due to
> java.io.IOException: Failed to decrypt data from Peer
> nifi-0.nifi-headless.apache-nifi.svc.cluster.local:8443 because Peer
> unexpectedly closed connection
> {code}
> Files get stuck in the queue mentioned, it's not always the same, these are
> simple round robin queues with or without compression.
> When we recreate the cluster it goes for like some time and then it occurs
> again and again
> Is there a particular reason ?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (NIFI-9574) Failed to decrypt data from Peer
[
https://issues.apache.org/jira/browse/NIFI-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17476325#comment-17476325
]
Mahieddine Cherif commented on NIFI-9574:
-
Yeah we have a quiet constraint env :/ but the thing also is that we didn't
have this behaviour before and we have quiet light flows only light api calls
mainly.
Here are some system diagnostics captures, thanks a lot !
!Screenshot 2022-01-14 at 19.21.11.png! !Screenshot 2022-01-14 at
19.21.46.png! !Screenshot 2022-01-14 at 19.21.36.png! !Screenshot 2022-01-14
at 19.21.16.png!
> Failed to decrypt data from Peer
> -
>
> Key: NIFI-9574
> URL: https://issues.apache.org/jira/browse/NIFI-9574
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Mahieddine Cherif
>Priority: Major
> Attachments: Screenshot 2022-01-14 at 19.21.11.png, Screenshot
> 2022-01-14 at 19.21.16.png, Screenshot 2022-01-14 at 19.21.36.png, Screenshot
> 2022-01-14 at 19.21.46.png
>
>
> After a migration to 1.15.2 it seems like we have almost systematically this
> error on our cluster all the time
> {code:java}
> Failed to communicate with Peer
> nifi-0.nifi-headless.apache-nifi.svc.cluster.local:8443 when load balancing
> data for Connection with ID b76e7297-e8a0-3b2b-ba30-d338db411301 due to
> java.io.IOException: Failed to decrypt data from Peer
> nifi-0.nifi-headless.apache-nifi.svc.cluster.local:8443 because Peer
> unexpectedly closed connection
> {code}
> Files get stuck in the queue mentioned, it's not always the same, these are
> simple round robin queues with or without compression.
> When we recreate the cluster it goes for like some time and then it occurs
> again and again
> Is there a particular reason ?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (NIFI-9574) Failed to decrypt data from Peer
[
https://issues.apache.org/jira/browse/NIFI-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahieddine Cherif updated NIFI-9574:
Attachment: Screenshot 2022-01-14 at 19.21.11.png
Screenshot 2022-01-14 at 19.21.46.png
Screenshot 2022-01-14 at 19.21.36.png
Screenshot 2022-01-14 at 19.21.16.png
> Failed to decrypt data from Peer
> -
>
> Key: NIFI-9574
> URL: https://issues.apache.org/jira/browse/NIFI-9574
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Mahieddine Cherif
>Priority: Major
> Attachments: Screenshot 2022-01-14 at 19.21.11.png, Screenshot
> 2022-01-14 at 19.21.16.png, Screenshot 2022-01-14 at 19.21.36.png, Screenshot
> 2022-01-14 at 19.21.46.png
>
>
> After a migration to 1.15.2 it seems like we have almost systematically this
> error on our cluster all the time
> {code:java}
> Failed to communicate with Peer
> nifi-0.nifi-headless.apache-nifi.svc.cluster.local:8443 when load balancing
> data for Connection with ID b76e7297-e8a0-3b2b-ba30-d338db411301 due to
> java.io.IOException: Failed to decrypt data from Peer
> nifi-0.nifi-headless.apache-nifi.svc.cluster.local:8443 because Peer
> unexpectedly closed connection
> {code}
> Files get stuck in the queue mentioned, it's not always the same, these are
> simple round robin queues with or without compression.
> When we recreate the cluster it goes for like some time and then it occurs
> again and again
> Is there a particular reason ?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[
https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17476249#comment-17476249
]
Joe Witt commented on NIFI-9572:
There are no known issues involving bugs in provenance leading to too many open
files right now so probably this is solvable/configuration or the underlying
issue is likely present in older versions too.
First, we often have users report they are certain max open file settings are
correct by them running ulimit -a but only then discover they ran it as a
different user than what their nifi really launches as. Please double check
this.
Second, what output we see complain of open file exhaustion is often just the
victim that hit the limit. There could be a totally different underlying root
cause (socket exhaustion for instance). Run 'lsof -p ' to get a full
listing of all the file handles nifi has. Run this right after a restart/nifi
initially comes up. Then run it periodically over time. If there is indeed a
leak it will become pretty obvious pretty fast in terms of the likely culprit.
> Failed to index Provenance Events and (Too many Files)
> --
>
> Key: NIFI-9572
> URL: https://issues.apache.org/jira/browse/NIFI-9572
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
>Affects Versions: 1.15.2
>Reporter: mayki
>Priority: Major
>
> Hello
> I have upgraded NIFI 1.15.2 since 2022/01/05
> No issue until this night 2022/01/13
> * nifi version 1.15.2
> * jdk-1.8.0_311
> And the limit is high
> {code:java}
> Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2
> -bash-4.2@nifi$ ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) rg
> pending signals (-i) 63278
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 5
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 1
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
> {code}
>
> We got a lot error about provenance_repository, it fill our filesystem logs ..
>
> {code:java}
> 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2]
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877)
> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891)
> at
> org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468)
> at
> org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444)
> at
> org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70)
> at
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202)
> at
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.nio.file.FileSystemException:
> /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd:
> Too many open files
> {code}
>
>
> We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability.
> But it is impossible to do that if we got this error.
>
> Thanks for you help.
>
> Regards
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Comment Edited] (NIFI-9574) Failed to decrypt data from Peer
[
https://issues.apache.org/jira/browse/NIFI-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17476167#comment-17476167
]
Mark Payne edited comment on NIFI-9574 at 1/14/22, 3:11 PM:
Sounds like the "sending node" timed out when sending the data. How many nodes
are in the cluster? What values do you have set in {{nifi.properties}} for
{{nifi.cluster.load.balance.connections.per.node,}}
{{{}nifi.cluster.load.balance.max.thread.count and
nifi.cluster.load.balance.comms.timeout{}}}? You said you migrated to 1.15.2.
What version were you running before that?
was (Author: markap14):
Sounds like the "sending node" timed out when sending the data. How many nodes
are in the cluster? What values do you have set in {{nifi.properties}} for
{{nifi.cluster.load.balance.connections.per.node,}}
{{{}nifi.cluster.load.balance.max.thread.count and
{{nifi.cluster.load.balance.comms.timeout}}? You said you migrated to 1.15.2.
What version were you running before that?
> Failed to decrypt data from Peer
> -
>
> Key: NIFI-9574
> URL: https://issues.apache.org/jira/browse/NIFI-9574
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Mahieddine Cherif
>Priority: Major
>
> After a migration to 1.15.2 it seems like we have almost systematically this
> error on our cluster all the time
> {code:java}
> Failed to communicate with Peer
> nifi-0.nifi-headless.apache-nifi.svc.cluster.local:8443 when load balancing
> data for Connection with ID b76e7297-e8a0-3b2b-ba30-d338db411301 due to
> java.io.IOException: Failed to decrypt data from Peer
> nifi-0.nifi-headless.apache-nifi.svc.cluster.local:8443 because Peer
> unexpectedly closed connection
> {code}
> Files get stuck in the queue mentioned, it's not always the same, these are
> simple round robin queues with or without compression.
> When we recreate the cluster it goes for like some time and then it occurs
> again and again
> Is there a particular reason ?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (NIFI-9574) Failed to decrypt data from Peer
[
https://issues.apache.org/jira/browse/NIFI-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17476199#comment-17476199
]
Mark Payne commented on NIFI-9574:
--
[~mahieddine] I wouldn't recommend running nifi on a 2-core VM. At the absolute
minimum I'd use 4, preferably 8+ cores.
I'd also look for other errors in the logs. How healthy is the heap? How large
is it? How much garbage collection is occurring? Given that it sounds like it's
timing out, I'd be concerned with what else is happening that's causing the
nodes not to communicate well.
> Failed to decrypt data from Peer
> -
>
> Key: NIFI-9574
> URL: https://issues.apache.org/jira/browse/NIFI-9574
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Mahieddine Cherif
>Priority: Major
>
> After a migration to 1.15.2 it seems like we have almost systematically this
> error on our cluster all the time
> {code:java}
> Failed to communicate with Peer
> nifi-0.nifi-headless.apache-nifi.svc.cluster.local:8443 when load balancing
> data for Connection with ID b76e7297-e8a0-3b2b-ba30-d338db411301 due to
> java.io.IOException: Failed to decrypt data from Peer
> nifi-0.nifi-headless.apache-nifi.svc.cluster.local:8443 because Peer
> unexpectedly closed connection
> {code}
> Files get stuck in the queue mentioned, it's not always the same, these are
> simple round robin queues with or without compression.
> When we recreate the cluster it goes for like some time and then it occurs
> again and again
> Is there a particular reason ?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Comment Edited] (NIFI-9574) Failed to decrypt data from Peer
[
https://issues.apache.org/jira/browse/NIFI-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17476167#comment-17476167
]
Mark Payne edited comment on NIFI-9574 at 1/14/22, 3:10 PM:
Sounds like the "sending node" timed out when sending the data. How many nodes
are in the cluster? What values do you have set in {{nifi.properties}} for
{{nifi.cluster.load.balance.connections.per.node,}}
{{{}nifi.cluster.load.balance.max.thread.count and
{{nifi.cluster.load.balance.comms.timeout}}? You said you migrated to 1.15.2.
What version were you running before that?
was (Author: markap14):
Sounds like the "sending node" timed out when sending the data. How many nodes
are in the cluster? What values do you have set in {{nifi.properties}} for
{{nifi.cluster.load.balance.connections.per.node,}}
{{{}nifi.cluster.load.balance.max.thread.count and
{{nifi.cluster.load.balance.comms.timeout}}{}}}? You said you migrated to
1.15.2. What version were you running before that?
> Failed to decrypt data from Peer
> -
>
> Key: NIFI-9574
> URL: https://issues.apache.org/jira/browse/NIFI-9574
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Mahieddine Cherif
>Priority: Major
>
> After a migration to 1.15.2 it seems like we have almost systematically this
> error on our cluster all the time
> {code:java}
> Failed to communicate with Peer
> nifi-0.nifi-headless.apache-nifi.svc.cluster.local:8443 when load balancing
> data for Connection with ID b76e7297-e8a0-3b2b-ba30-d338db411301 due to
> java.io.IOException: Failed to decrypt data from Peer
> nifi-0.nifi-headless.apache-nifi.svc.cluster.local:8443 because Peer
> unexpectedly closed connection
> {code}
> Files get stuck in the queue mentioned, it's not always the same, these are
> simple round robin queues with or without compression.
> When we recreate the cluster it goes for like some time and then it occurs
> again and again
> Is there a particular reason ?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Comment Edited] (NIFI-9574) Failed to decrypt data from Peer
[
https://issues.apache.org/jira/browse/NIFI-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17476169#comment-17476169
]
Mahieddine Cherif edited comment on NIFI-9574 at 1/14/22, 2:26 PM:
---
We have 3 nodes x 2 core (8GB each), regarding the configuration aspects all
the nifi.cluster.load.balance.* properties are not set so we are using normally
the default values (i suppose)
We we using the 1.15.1 and 1.14.0 before
was (Author: mahieddine):
We have 3 nodes, regarding the configuration aspects all the
nifi.cluster.load.balance.* properties are not set so we are using normally the
default values (i suppose)
We we using the 1.15.1 and 1.14.0 before
> Failed to decrypt data from Peer
> -
>
> Key: NIFI-9574
> URL: https://issues.apache.org/jira/browse/NIFI-9574
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Mahieddine Cherif
>Priority: Major
>
> After a migration to 1.15.2 it seems like we have almost systematically this
> error on our cluster all the time
> {code:java}
> Failed to communicate with Peer
> nifi-0.nifi-headless.apache-nifi.svc.cluster.local:8443 when load balancing
> data for Connection with ID b76e7297-e8a0-3b2b-ba30-d338db411301 due to
> java.io.IOException: Failed to decrypt data from Peer
> nifi-0.nifi-headless.apache-nifi.svc.cluster.local:8443 because Peer
> unexpectedly closed connection
> {code}
> Files get stuck in the queue mentioned, it's not always the same, these are
> simple round robin queues with or without compression.
> When we recreate the cluster it goes for like some time and then it occurs
> again and again
> Is there a particular reason ?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (NIFI-9574) Failed to decrypt data from Peer
[
https://issues.apache.org/jira/browse/NIFI-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17476169#comment-17476169
]
Mahieddine Cherif commented on NIFI-9574:
-
We have 3 nodes, regarding the configuration aspects all the
nifi.cluster.load.balance.* properties are not set so we are using normally the
default values (i suppose)
We we using the 1.15.1 and 1.14.0 before
> Failed to decrypt data from Peer
> -
>
> Key: NIFI-9574
> URL: https://issues.apache.org/jira/browse/NIFI-9574
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Mahieddine Cherif
>Priority: Major
>
> After a migration to 1.15.2 it seems like we have almost systematically this
> error on our cluster all the time
> {code:java}
> Failed to communicate with Peer
> nifi-0.nifi-headless.apache-nifi.svc.cluster.local:8443 when load balancing
> data for Connection with ID b76e7297-e8a0-3b2b-ba30-d338db411301 due to
> java.io.IOException: Failed to decrypt data from Peer
> nifi-0.nifi-headless.apache-nifi.svc.cluster.local:8443 because Peer
> unexpectedly closed connection
> {code}
> Files get stuck in the queue mentioned, it's not always the same, these are
> simple round robin queues with or without compression.
> When we recreate the cluster it goes for like some time and then it occurs
> again and again
> Is there a particular reason ?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (NIFI-9574) Failed to decrypt data from Peer
[
https://issues.apache.org/jira/browse/NIFI-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17476167#comment-17476167
]
Mark Payne commented on NIFI-9574:
--
Sounds like the "sending node" timed out when sending the data. How many nodes
are in the cluster? What values do you have set in {{nifi.properties}} for
{{nifi.cluster.load.balance.connections.per.node,}}
{{{}nifi.cluster.load.balance.max.thread.count and
{{nifi.cluster.load.balance.comms.timeout}}{}}}? You said you migrated to
1.15.2. What version were you running before that?
> Failed to decrypt data from Peer
> -
>
> Key: NIFI-9574
> URL: https://issues.apache.org/jira/browse/NIFI-9574
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Mahieddine Cherif
>Priority: Major
>
> After a migration to 1.15.2 it seems like we have almost systematically this
> error on our cluster all the time
> {code:java}
> Failed to communicate with Peer
> nifi-0.nifi-headless.apache-nifi.svc.cluster.local:8443 when load balancing
> data for Connection with ID b76e7297-e8a0-3b2b-ba30-d338db411301 due to
> java.io.IOException: Failed to decrypt data from Peer
> nifi-0.nifi-headless.apache-nifi.svc.cluster.local:8443 because Peer
> unexpectedly closed connection
> {code}
> Files get stuck in the queue mentioned, it's not always the same, these are
> simple round robin queues with or without compression.
> When we recreate the cluster it goes for like some time and then it occurs
> again and again
> Is there a particular reason ?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (NIFI-9574) Failed to decrypt data from Peer
[
https://issues.apache.org/jira/browse/NIFI-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahieddine Cherif updated NIFI-9574:
Description:
After a migration to 1.15.2 it seems like we have almost systematically this
error on our cluster all the time
{code:java}
Failed to communicate with Peer
nifi-0.nifi-headless.apache-nifi.svc.cluster.local:8443 when load balancing
data for Connection with ID b76e7297-e8a0-3b2b-ba30-d338db411301 due to
java.io.IOException: Failed to decrypt data from Peer
nifi-0.nifi-headless.apache-nifi.svc.cluster.local:8443 because Peer
unexpectedly closed connection
{code}
Files get stuck in the queue mentioned, it's not always the same, these are
simple round robin queues with or without compression.
When we recreate the cluster it goes for like some time and then it occurs
again and again
Is there a particular reason ?
was:
After a migration to 1.15.2 it seems like we have almost systematically this
error on our cluster all the time
{code:java}
Failed to communicate with Peer
nifi-0.nifi-headless.apache-nifi.svc.cluster.local:8443 when load balancing
data for Connection with ID b76e7297-e8a0-3b2b-ba30-d338db411301 due to
java.io.IOException: Failed to decrypt data from Peer
nifi-0.nifi-headless.apache-nifi.svc.cluster.local:8443 because Peer
unexpectedly closed connection
{code}
When we recreate the cluster it goes for like some time and then it occurs
again and again
Is there a particular reason ?
> Failed to decrypt data from Peer
> -
>
> Key: NIFI-9574
> URL: https://issues.apache.org/jira/browse/NIFI-9574
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Mahieddine Cherif
>Priority: Major
>
> After a migration to 1.15.2 it seems like we have almost systematically this
> error on our cluster all the time
> {code:java}
> Failed to communicate with Peer
> nifi-0.nifi-headless.apache-nifi.svc.cluster.local:8443 when load balancing
> data for Connection with ID b76e7297-e8a0-3b2b-ba30-d338db411301 due to
> java.io.IOException: Failed to decrypt data from Peer
> nifi-0.nifi-headless.apache-nifi.svc.cluster.local:8443 because Peer
> unexpectedly closed connection
> {code}
> Files get stuck in the queue mentioned, it's not always the same, these are
> simple round robin queues with or without compression.
> When we recreate the cluster it goes for like some time and then it occurs
> again and again
> Is there a particular reason ?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (NIFI-9574) Failed to decrypt data from Peer
Mahieddine Cherif created NIFI-9574:
---
Summary: Failed to decrypt data from Peer
Key: NIFI-9574
URL: https://issues.apache.org/jira/browse/NIFI-9574
Project: Apache NiFi
Issue Type: Bug
Reporter: Mahieddine Cherif
After a migration to 1.15.2 it seems like we have almost systematically this
error on our cluster all the time
{code:java}
Failed to communicate with Peer
nifi-0.nifi-headless.apache-nifi.svc.cluster.local:8443 when load balancing
data for Connection with ID b76e7297-e8a0-3b2b-ba30-d338db411301 due to
java.io.IOException: Failed to decrypt data from Peer
nifi-0.nifi-headless.apache-nifi.svc.cluster.local:8443 because Peer
unexpectedly closed connection
{code}
When we recreate the cluster it goes for like some time and then it occurs
again and again
Is there a particular reason ?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (NIFI-9573) Registry docker image does not allow setting property "Remote Clone Repository" from environment variables
[ https://issues.apache.org/jira/browse/NIFI-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikko Koppanen resolved NIFI-9573. -- Resolution: Invalid > Registry docker image does not allow setting property "Remote Clone > Repository" from environment variables > -- > > Key: NIFI-9573 > URL: https://issues.apache.org/jira/browse/NIFI-9573 > Project: Apache NiFi > Issue Type: Improvement > Environment: docker >Reporter: Mikko Koppanen >Priority: Minor > > Currently the update_flow_provider.sh is not allowing to set the "Remote > Clone Repository" property from environment variables. > Reference: > https://github.com/apache/nifi/blob/main/nifi-registry/nifi-registry-docker-maven/dockermaven/sh/update_flow_provider.sh#L40 > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-9573) Registry docker image does not allow setting property "Remote Clone Repository" from environment variables
[ https://issues.apache.org/jira/browse/NIFI-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17476146#comment-17476146 ] Mikko Koppanen commented on NIFI-9573: -- Apologies, found the env variable NIFI_REGISTRY_GIT_REPO > Registry docker image does not allow setting property "Remote Clone > Repository" from environment variables > -- > > Key: NIFI-9573 > URL: https://issues.apache.org/jira/browse/NIFI-9573 > Project: Apache NiFi > Issue Type: Improvement > Environment: docker >Reporter: Mikko Koppanen >Priority: Minor > > Currently the update_flow_provider.sh is not allowing to set the "Remote > Clone Repository" property from environment variables. > Reference: > https://github.com/apache/nifi/blob/main/nifi-registry/nifi-registry-docker-maven/dockermaven/sh/update_flow_provider.sh#L40 > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (NIFI-9573) Registry docker image does not allow setting property "Remote Clone Repository" from environment variables
Mikko Koppanen created NIFI-9573: Summary: Registry docker image does not allow setting property "Remote Clone Repository" from environment variables Key: NIFI-9573 URL: https://issues.apache.org/jira/browse/NIFI-9573 Project: Apache NiFi Issue Type: Improvement Environment: docker Reporter: Mikko Koppanen Currently the update_flow_provider.sh is not allowing to set the "Remote Clone Repository" property from environment variables. Reference: https://github.com/apache/nifi/blob/main/nifi-registry/nifi-registry-docker-maven/dockermaven/sh/update_flow_provider.sh#L40 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [nifi] sedadgn commented on pull request #5458: NIFI-7865 amqp$header is splitted in the wrong way for "," and "}"
sedadgn commented on pull request #5458: URL: https://github.com/apache/nifi/pull/5458#issuecomment-1012994358 > I'm not convinced that adding guava is necessary just for the split/join use case (I was ok with the previous logic, although perhaps it could be refactored out into a dedicated joinMap/splitMap helper method), but if you do want to stick with guava, it looks to me like most other NARs that bundle it call that out in the NOTICE file in the `nifi-*-bundle/nifi-*-nar/src/main/resources/META-INF/` directory, so that would probably be required in this case as well. > > Everything else looks good to me aside from one minor comment about code organization in StandardValidators. Hello @kevdoran , Thank you very much for your review. I removed guava and added 2 methods for splitMap/joinMap. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] sedadgn commented on a change in pull request #5458: NIFI-7865 amqp$header is splitted in the wrong way for "," and "}"
sedadgn commented on a change in pull request #5458:
URL: https://github.com/apache/nifi/pull/5458#discussion_r784726662
##
File path:
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/processor/util/StandardValidators.java
##
@@ -980,4 +980,25 @@ public ValidationResult validate(final String subject,
final String value, final
return new
ValidationResult.Builder().subject(subject).input(value).explanation(reason).valid(reason
== null).build();
}
}
+
+public static final Validator SINGLE_CHAR_VALIDATOR = (subject, input,
context) -> {
Review comment:
I moved the method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
mayki created NIFI-9572:
---
Summary: Failed to index Provenance Events and (Too many Files)
Key: NIFI-9572
URL: https://issues.apache.org/jira/browse/NIFI-9572
Project: Apache NiFi
Issue Type: Bug
Components: Core UI
Affects Versions: 1.15.2
Reporter: mayki
Hello
I have upgraded NIFI 1.15.2 since 2022/01/05
No issue until this night 2022/01/13
* nifi version 1.15.2
* jdk-1.8.0_311
And the limit is high
{code:java}
Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2
-bash-4.2@nifi$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) rg
pending signals (-i) 63278
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 5
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 1
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
{code}
We got a lot error about provenance_repository, it fill our filesystem logs ..
{code:java}
2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2]
o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877)
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891)
at
org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468)
at
org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444)
at
org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70)
at
org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202)
at
org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.file.FileSystemException:
/data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd:
Too many open files
{code}
We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability.
But it is impossible to do that if we got this error.
Thanks for you help.
Regards
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
