[jira] [Created] (NIFI-10369) Apache NiFi 1.16.x windows server closes immediately
Nikolay Akimov created NIFI-10369: - Summary: Apache NiFi 1.16.x windows server closes immediately Key: NIFI-10369 URL: https://issues.apache.org/jira/browse/NIFI-10369 Project: Apache NiFi Issue Type: Bug Components: Examples Affects Versions: 1.16.3 Environment: Windows, Linux Reporter: Nikolay Akimov Attachments: Output_Port.xml When I try to restart the server it's throwing with the following error message. *org.apache.nifi.controller.serialization.FlowSynchronizationException: java.lang.IllegalStateException: Cannot add Connection to Process Group because source is an Output Port that does not belong to a child Process Group* I have simplified the scenario to such an extent that it is reproducible. I attach the file. After importing this file and if you place the template on the canvas, WiFi does not start after the restart. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] rliszli commented on a diff in pull request #6281: NIFI-10312 - Fix MiNiFi C2 integration
rliszli commented on code in PR #6281:
URL: https://github.com/apache/nifi/pull/6281#discussion_r948705878
##
c2/c2-client-bundle/c2-client-service/src/main/java/org/apache/nifi/c2/client/service/C2ClientService.java:
##
@@ -41,8 +41,12 @@ public C2ClientService(C2Client client, C2HeartbeatFactory
c2HeartbeatFactory, C
}
public void sendHeartbeat(RuntimeInfoWrapper runtimeInfoWrapper) {
-C2Heartbeat c2Heartbeat =
c2HeartbeatFactory.create(runtimeInfoWrapper);
-client.publishHeartbeat(c2Heartbeat).ifPresent(this::processResponse);
+try {
+C2Heartbeat c2Heartbeat =
c2HeartbeatFactory.create(runtimeInfoWrapper);
+
client.publishHeartbeat(c2Heartbeat).ifPresent(this::processResponse);
+} catch (Exception e) {
Review Comment:
This was added as a safety net. Without it, if an unhandled exception
thrown, the heartbeating stops. During the local development I run into this 2
times. One from the deserialization in C2HttpClient.sendHeartbeat() and one
from the UpdateConfigurationOperationHandler. The later got fixed locally as
you suggested. The 1st happened due to wrong input which, I think, can happen.
If you still think it should be removed, let me know and I'll remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] rliszli commented on a diff in pull request #6281: NIFI-10312 - Fix MiNiFi C2 integration
rliszli commented on code in PR #6281: URL: https://github.com/apache/nifi/pull/6281#discussion_r948704835 ## c2/c2-protocol/c2-protocol-api/pom.xml: ## @@ -33,5 +33,9 @@ limitations under the License. c2-protocol-component-api 1.18.0-SNAPSHOT + +com.fasterxml.jackson.core Review Comment: The reason behind this change is that different clients sends the orderType values in different ways(upper or lower case). Before adding the @JsonCreatorit annotation, I checked and it is used several places in nifi-api modules. I can have the same result by writing a Custom Deserializer and add it to an ObjectMapper in C2JacksonSerializer. I think it's a more complicated solution and I don't think that there is any problem with adding the jackson dependency to an api module. Please let me know, if you still insist to change it, I'll do that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] rliszli commented on a diff in pull request #6281: NIFI-10312 - Fix MiNiFi C2 integration
rliszli commented on code in PR #6281:
URL: https://github.com/apache/nifi/pull/6281#discussion_r948703762
##
minifi/minifi-integration-tests/src/test/java/org/apache/nifi/minifi/integration/c2/C2ProtocolIntegrationTest.java:
##
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.minifi.integration.c2;
+
+import com.palantir.docker.compose.DockerComposeExtension;
+import com.palantir.docker.compose.connection.waiting.HealthChecks;
+import org.apache.nifi.minifi.integration.util.LogUtil;
+import org.apache.nifi.security.util.KeystoreType;
+import org.apache.nifi.security.util.SslContextFactory;
+import org.apache.nifi.security.util.StandardTlsConfiguration;
+import org.apache.nifi.security.util.TlsConfiguration;
+import org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone;
+import org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandaloneCommandLine;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLSocketFactory;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+@Timeout(60)
+public class C2ProtocolIntegrationTest {
+private static final String AGENT_1 = "minifi-edge1";
+private static final String AGENT_2 = "minifi-edge2";
+private static final String AGENT_CLASS = "raspi3";
+private static final String SERVICE = "c2-authoritative";
+private static final String CONFIG_YAML = "config.text.yml.v2";
+private static Path certificatesDirectory;
+private static SSLContext trustSslContext;
+private static SSLSocketFactory healthCheckSocketFactory;
+public static DockerComposeExtension docker =
DockerComposeExtension.builder()
+.file("target/test-classes/docker-compose-c2-protocol.yml")
+.waitingForService(AGENT_1,
HealthChecks.toRespond2xxOverHttp(8000, dockerPort -> "http://"; +
dockerPort.getIp() + ":" + dockerPort.getExternalPort()))
+.waitingForService(AGENT_2,
HealthChecks.toRespond2xxOverHttp(8000, dockerPort -> "http://"; +
dockerPort.getIp() + ":" + dockerPort.getExternalPort()))
+.build();
+
+private static Path resourceDirectory;
+private static Path authoritativeFiles;
+private static Path minifiEdge1Version2;
+private static Path minifiEdge2Version2;
+
+/**
+ * Generates certificates with the tls-toolkit and then starts up the
docker compose file
+ */
+@BeforeAll
+public static void init() throws Exception {
+resourceDirectory =
Paths.get(C2ProtocolIntegrationTest.class.getClassLoader()
+
.getResource("docker-compose-c2-protocol.yml").getFile()).getParent();
+certificatesDirectory =
resourceDirectory.toAbsolutePath().resolve("certificates-c2-protocol");
+authoritativeFiles =
resourceDirectory.resolve("c2").resolve("protocol").resolve(SERVICE).resolve("files");
+minifiEdge1Version2 =
authoritativeFiles.resolve("edge1").resolve(AGENT_CLASS).resolve(CONFIG_YAML);
+minifiEdge2Version2 =
authoritativeFiles.resolve("edge2").resolve(AGENT_CLASS).resolve(CONFIG_YAML);
+
+if (Files.exists(minifiEdge1Version2)) {
+Files.delete(minifiEdge1Version2);
+}
+if (Files.exists(minifiEdge2Version2)) {
+Files.delete(minifiEdge2Version2);
+}
+
+List toolkitCommandLine = new ArrayList<>(Arrays.asList("-O",
"-o", certificatesDirectory.toFile().getAbsolutePath(), "-S",
"badKeystorePass", "-P", "badTrustPass"));
+for (String serverHostname : Arrays.asList(SERVICE, AGENT_1, AGENT_2))
{
+toolkitCommandLine.add("-n");
+toolkitCommandLine.add(serverHostname);
+}
+Files.createDirectories(certificatesDirectory);
+TlsToolkitStandaloneCommandLine tlsToolkitStandaloneCommandLine = new
TlsToolkitStandaloneCommandLine();
+tlsToolkitStandaloneCommandLine.parse(toolkitCommandLine.toArray(new
String[toolkitCommandLine.size
[GitHub] [nifi] rliszli commented on a diff in pull request #6281: NIFI-10312 - Fix MiNiFi C2 integration
rliszli commented on code in PR #6281: URL: https://github.com/apache/nifi/pull/6281#discussion_r948702984 ## minifi/minifi-integration-tests/src/test/resources/logback.xml: ## @@ -55,7 +55,7 @@ - + Review Comment: Thanks, reverted. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] rliszli commented on a diff in pull request #6281: NIFI-10312 - Fix MiNiFi C2 integration
rliszli commented on code in PR #6281:
URL: https://github.com/apache/nifi/pull/6281#discussion_r948702181
##
c2/c2-client-bundle/c2-client-service/src/test/java/org/apache/nifi/c2/client/service/operation/UpdateConfigurationOperationHandlerTest.java:
##
@@ -62,12 +61,34 @@ void testUpdateConfigurationOperationHandlerCreateSuccess()
{
}
@Test
-void testHandleThrowsExceptionForIncorrectArg() {
+void testHandleIncorrectArg() {
UpdateConfigurationOperationHandler handler = new
UpdateConfigurationOperationHandler(null, null, null);
C2Operation operation = new C2Operation();
operation.setArgs(INCORRECT_LOCATION_MAP);
-IllegalStateException exception =
assertThrows(IllegalStateException.class, () -> handler.handle(operation));
+C2OperationAck response = handler.handle(operation);
+
+assertEquals(C2OperationState.OperationState.NOT_APPLIED,
response.getOperationState().getState());
+}
+
+@Test
+void testHandleFlowIdInArg() {
+Function successUpdate = x -> true;
+when(flowIdHolder.getFlowId()).thenReturn(FLOW_ID);
+
when(client.retrieveUpdateContent(any())).thenReturn(Optional.of("content".getBytes()));
+UpdateConfigurationOperationHandler handler = new
UpdateConfigurationOperationHandler(client, flowIdHolder, successUpdate);
+C2Operation operation = new C2Operation();
+operation.setIdentifier(OPERATION_ID);
+
+Map args = new HashMap<>();
+args.putAll(INCORRECT_LOCATION_MAP);
+args.put("flowId", "argsFlowId");
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] rliszli commented on a diff in pull request #6281: NIFI-10312 - Fix MiNiFi C2 integration
rliszli commented on code in PR #6281: URL: https://github.com/apache/nifi/pull/6281#discussion_r948701968 ## c2/c2-client-bundle/c2-client-service/src/test/java/org/apache/nifi/c2/client/service/operation/UpdateConfigurationOperationHandlerTest.java: ## @@ -17,13 +17,12 @@ package org.apache.nifi.c2.client.service.operation; import static org.apache.commons.lang3.StringUtils.EMPTY; -import static org.apache.nifi.c2.client.service.operation.UpdateConfigurationOperationHandler.LOCATION; Review Comment: Sure, done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6265: NIFI-10234 Implement PutIoTDB
exceptionfactory commented on code in PR #6265:
URL: https://github.com/apache/nifi/pull/6265#discussion_r946762657
##
nifi-nar-bundles/nifi-iotdb-bundle/nifi-iotdb-processors/src/main/java/org/apache/nifi/processors/PutIoTDB.java:
##
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors;
+
+import java.io.InputStream;
+import java.sql.Timestamp;
+import java.time.LocalDateTime;
+import java.time.format.DateTimeFormatter;
+import java.util.Set;
+import java.util.List;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.nifi.processors.model.IoTDBSchema;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.write.record.Tablet;
+import org.apache.iotdb.tsfile.write.schema.MeasurementSchema;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.model.ValidationResult;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.record.Record;
+
+@Tags({"iotdb", "insert", "tablet"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription(
+"This is a record aware processor that reads the content of the
incoming FlowFile as individual records using the "
++ "configured 'Record Reader' and writes them to Apache IoTDB
using native interface.")
+public class PutIoTDB extends AbstractIoTDB {
+
+static final PropertyDescriptor RECORD_READER_FACTORY =
+new PropertyDescriptor.Builder()
+.name("Record Reader")
+.description(
+"Specifies the type of Record Reader controller
service to use for parsing the incoming data "
++ "and determining the schema")
+.identifiesControllerService(RecordReaderFactory.class)
+.required(true)
+.build();
+
+static final PropertyDescriptor SCHEMA =
+new PropertyDescriptor.Builder()
+.name("Schema")
+.description(
+"The schema that IoTDB needs doesn't support good
by NiFi.\n"
++ "Therefore, you can define the schema
here.\n"
++ "Besides, you can set encoding type and
compression type by this method.\n"
++ "If you don't set this property, the
inferred schema will be used.\n")
Review Comment:
The wording of this description is not quite clear, and does not need to
state what NiFi does not support. Recommend adjusting the wording to include
more details about the expected format, along the following lines:
The Apache IoTDB Schema Template defined using JSON. The Processor will
infer the IoTDB Schema when this property is not configured.
The [IoTDB Schema
Template](https://iotdb.apache.org/UserGuide/Master/Data-Concept/Schema-Template.html)
documentation provides some details, but is there an official format
specification? It would be helpful to link to that documentation if available,
otherwise it will be unclear how to define this property.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries
[GitHub] [nifi] xuanronaldo commented on pull request #6265: NIFI-10234 Implement PutIoTDB
xuanronaldo commented on PR #6265: URL: https://github.com/apache/nifi/pull/6265#issuecomment-1218890319 > Okay, I got it. I will update the code asap. Thanks for your suggestions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] (NIFI-9989) sftp processors Received SSH_MSG_UNIMPLEMENTED
[ https://issues.apache.org/jira/browse/NIFI-9989 ] Hadi deleted comment on NIFI-9989: was (Author: hadi): I've been following this problem. Get data from many SFTP, so i see the error eveyday, like: !image-2022-08-16-09-44-10-682.png! > sftp processors Received SSH_MSG_UNIMPLEMENTED > -- > > Key: NIFI-9989 > URL: https://issues.apache.org/jira/browse/NIFI-9989 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.16.1 > Environment: Ubuntu 20.04, openjdk version "1.8.0_312" > OpenJDK Runtime Environment (build 1.8.0_312-8u312-b07-0ubuntu1~20.04-b07) > OpenJDK 64-Bit Server VM (build 25.312-b07, mixed mode) >Reporter: Jens M Kofoed >Assignee: David Handermann >Priority: Major > Attachments: image-2022-08-16-09-44-10-682.png > > > After upgrading from single instance version 1.13.2 to a 3 node cluster > version 1.16.1 All sftp processors occasionally report errors: > 2022-05-05 07:26:08,318 INFO [Timer-Driven Process Thread-7] > net.schmizz.sshj.transport.random.JCERandom Creating new SecureRandom. > 2022-05-05 07:26:08,318 DEBUG [Timer-Driven Process Thread-7] > net.schmizz.sshj.transport.random.JCERandom Random creation took 0 ms > 2022-05-05 07:26:08,318 DEBUG [Timer-Driven Process Thread-7] > net.schmizz.sshj.DefaultConfig Available cipher factories: > [chacha20-poly1...@openssh.com, aes128-cbc, aes128-ctr, aes192-cbc, > aes192-ctr, aes256-cbc, aes256-ctr, aes128-...@openssh.com, > aes256-...@openssh.com, blowfish-cbc, blowfish-ctr, cast128-cbc, cast128-ctr, > idea-cbc, idea-ctr, serpent128-cbc, serpent128-ctr, serpent192-cbc, > serpent192-ctr, serpent256-cbc, serpent256-ctr, 3des-cbc, 3des-ctr, > twofish128-cbc, twofish128-ctr, twofish192-cbc, twofish192-ctr, > twofish256-cbc, twofish256-ctr, twofish-cbc, arcfour, arcfour128, arcfour256] > 2022-05-05 07:26:08,319 INFO [Timer-Driven Process Thread-7] > net.schmizz.sshj.transport.TransportImpl Client identity string: > SSH-2.0-SSHJ_0.33.0 > 2022-05-05 07:26:08,334 INFO [Timer-Driven Process Thread-7] > net.schmizz.sshj.transport.TransportImpl Server identity string: > SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.4 > 2022-05-05 07:26:08,339 DEBUG [sshj-Reader-hostname/192.168.1.3:22] > net.schmizz.sshj.transport.KeyExchanger Received SSH_MSG_KEXINIT > 2022-05-05 07:26:08,340 DEBUG [sshj-Reader-hostname/192.168.1.3:22] > net.schmizz.sshj.transport.KeyExchanger Sending SSH_MSG_KEXINIT > 2022-05-05 07:26:08,340 DEBUG [sshj-Reader-hostname/192.168.1.3:22] > net.schmizz.sshj.transport.KeyExchanger Negotiated algorithms: [ > kex=curve25519-sha256; sig=ssh-ed25519; > c2sCipher=chacha20-poly1...@openssh.com; > s2cCipher=chacha20-poly1...@openssh.com; c2sMAC=hmac-sha1; s2cMAC=hmac-sha1; > c2sComp=none; s2cComp=none; ] > 2022-05-05 07:26:08,340 DEBUG [sshj-Reader-hostname/192.168.1.3:22] > net.schmizz.sshj.transport.kex.Curve25519SHA256 Sending SSH_MSG_KEXDH_INIT > 2022-05-05 07:26:08,349 DEBUG [sshj-Reader-hostname/192.168.1.3:22] > net.schmizz.sshj.transport.KeyExchanger Received kex followup data > 2022-05-05 07:26:08,349 DEBUG [sshj-Reader-hostname/192.168.1.3:22] > net.schmizz.sshj.transport.kex.Curve25519SHA256 Received SSH_MSG_KEXDH_REPLY > 2022-05-05 07:26:08,349 DEBUG [sshj-Reader-hostname/192.168.1.3:22] > net.schmizz.sshj.common.KeyType Key algo: ssh-ed25519, Key curve: 25519, Key > Len: 32p: [52, 76, 38, 101, -74, -101, -94, -14, 109, -121, -87, -122, -50, > -103, -88, -87, -50, -64, -55, 41, -109, -93, 60, 84, -75, -85, 59, -48, -34, > 25, -77, -53] > 2022-05-05 07:26:08,351 DEBUG [sshj-Reader-hostname/192.168.1.3:22] > net.schmizz.sshj.transport.KeyExchanger Trying to verify host key with > net.schmizz.sshj.transport.verification.PromiscuousVerifier@45c3e734 > 2022-05-05 07:26:08,351 DEBUG [sshj-Reader-hostname/192.168.1.3:22] > net.schmizz.sshj.transport.KeyExchanger Sending SSH_MSG_NEWKEYS > 2022-05-05 07:26:08,351 DEBUG [sshj-Reader-hostname/192.168.1.3:22] > net.schmizz.sshj.transport.KeyExchanger Received SSH_MSG_NEWKEYS > 2022-05-05 07:26:08,351 DEBUG [sshj-Reader-hostname/192.168.1.3:22] > net.schmizz.sshj.transport.TransportImpl Received SSH_MSG_EXT_INFO > 2022-05-05 07:26:08,354 DEBUG [Timer-Driven Process Thread-7] > net.schmizz.sshj.transport.KeyExchanger Sending SSH_MSG_KEXINIT > 2022-05-05 07:26:08,355 DEBUG [sshj-Reader-hostname/192.168.1.3:22] > net.schmizz.sshj.transport.TransportImpl Received SSH_MSG_UNIMPLEMENTED #3 > 2022-05-05 07:26:08,355 ERROR [sshj-Reader-hostname/192.168.1.3:22] > net.schmizz.sshj.transport.TransportImpl Dying because - Received > SSH_MSG_UNIMPLEMENTED while exchanging keys > net.schmizz.sshj.transport.TransportException: Received SSH_MSG_UNIMPLEMENTED > while exchanging keys > at > net.schmizz.sshj
[jira] [Commented] (NIFI-9989) sftp processors Received SSH_MSG_UNIMPLEMENTED
[ https://issues.apache.org/jira/browse/NIFI-9989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17581095#comment-17581095 ] Hadi commented on NIFI-9989: I've been following this problem. Get data from many SFTP, so i see the error eveyday, like: !image-2022-08-16-09-44-10-682.png! > sftp processors Received SSH_MSG_UNIMPLEMENTED > -- > > Key: NIFI-9989 > URL: https://issues.apache.org/jira/browse/NIFI-9989 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.16.1 > Environment: Ubuntu 20.04, openjdk version "1.8.0_312" > OpenJDK Runtime Environment (build 1.8.0_312-8u312-b07-0ubuntu1~20.04-b07) > OpenJDK 64-Bit Server VM (build 25.312-b07, mixed mode) >Reporter: Jens M Kofoed >Assignee: David Handermann >Priority: Major > Attachments: image-2022-08-16-09-44-10-682.png > > > After upgrading from single instance version 1.13.2 to a 3 node cluster > version 1.16.1 All sftp processors occasionally report errors: > 2022-05-05 07:26:08,318 INFO [Timer-Driven Process Thread-7] > net.schmizz.sshj.transport.random.JCERandom Creating new SecureRandom. > 2022-05-05 07:26:08,318 DEBUG [Timer-Driven Process Thread-7] > net.schmizz.sshj.transport.random.JCERandom Random creation took 0 ms > 2022-05-05 07:26:08,318 DEBUG [Timer-Driven Process Thread-7] > net.schmizz.sshj.DefaultConfig Available cipher factories: > [chacha20-poly1...@openssh.com, aes128-cbc, aes128-ctr, aes192-cbc, > aes192-ctr, aes256-cbc, aes256-ctr, aes128-...@openssh.com, > aes256-...@openssh.com, blowfish-cbc, blowfish-ctr, cast128-cbc, cast128-ctr, > idea-cbc, idea-ctr, serpent128-cbc, serpent128-ctr, serpent192-cbc, > serpent192-ctr, serpent256-cbc, serpent256-ctr, 3des-cbc, 3des-ctr, > twofish128-cbc, twofish128-ctr, twofish192-cbc, twofish192-ctr, > twofish256-cbc, twofish256-ctr, twofish-cbc, arcfour, arcfour128, arcfour256] > 2022-05-05 07:26:08,319 INFO [Timer-Driven Process Thread-7] > net.schmizz.sshj.transport.TransportImpl Client identity string: > SSH-2.0-SSHJ_0.33.0 > 2022-05-05 07:26:08,334 INFO [Timer-Driven Process Thread-7] > net.schmizz.sshj.transport.TransportImpl Server identity string: > SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.4 > 2022-05-05 07:26:08,339 DEBUG [sshj-Reader-hostname/192.168.1.3:22] > net.schmizz.sshj.transport.KeyExchanger Received SSH_MSG_KEXINIT > 2022-05-05 07:26:08,340 DEBUG [sshj-Reader-hostname/192.168.1.3:22] > net.schmizz.sshj.transport.KeyExchanger Sending SSH_MSG_KEXINIT > 2022-05-05 07:26:08,340 DEBUG [sshj-Reader-hostname/192.168.1.3:22] > net.schmizz.sshj.transport.KeyExchanger Negotiated algorithms: [ > kex=curve25519-sha256; sig=ssh-ed25519; > c2sCipher=chacha20-poly1...@openssh.com; > s2cCipher=chacha20-poly1...@openssh.com; c2sMAC=hmac-sha1; s2cMAC=hmac-sha1; > c2sComp=none; s2cComp=none; ] > 2022-05-05 07:26:08,340 DEBUG [sshj-Reader-hostname/192.168.1.3:22] > net.schmizz.sshj.transport.kex.Curve25519SHA256 Sending SSH_MSG_KEXDH_INIT > 2022-05-05 07:26:08,349 DEBUG [sshj-Reader-hostname/192.168.1.3:22] > net.schmizz.sshj.transport.KeyExchanger Received kex followup data > 2022-05-05 07:26:08,349 DEBUG [sshj-Reader-hostname/192.168.1.3:22] > net.schmizz.sshj.transport.kex.Curve25519SHA256 Received SSH_MSG_KEXDH_REPLY > 2022-05-05 07:26:08,349 DEBUG [sshj-Reader-hostname/192.168.1.3:22] > net.schmizz.sshj.common.KeyType Key algo: ssh-ed25519, Key curve: 25519, Key > Len: 32p: [52, 76, 38, 101, -74, -101, -94, -14, 109, -121, -87, -122, -50, > -103, -88, -87, -50, -64, -55, 41, -109, -93, 60, 84, -75, -85, 59, -48, -34, > 25, -77, -53] > 2022-05-05 07:26:08,351 DEBUG [sshj-Reader-hostname/192.168.1.3:22] > net.schmizz.sshj.transport.KeyExchanger Trying to verify host key with > net.schmizz.sshj.transport.verification.PromiscuousVerifier@45c3e734 > 2022-05-05 07:26:08,351 DEBUG [sshj-Reader-hostname/192.168.1.3:22] > net.schmizz.sshj.transport.KeyExchanger Sending SSH_MSG_NEWKEYS > 2022-05-05 07:26:08,351 DEBUG [sshj-Reader-hostname/192.168.1.3:22] > net.schmizz.sshj.transport.KeyExchanger Received SSH_MSG_NEWKEYS > 2022-05-05 07:26:08,351 DEBUG [sshj-Reader-hostname/192.168.1.3:22] > net.schmizz.sshj.transport.TransportImpl Received SSH_MSG_EXT_INFO > 2022-05-05 07:26:08,354 DEBUG [Timer-Driven Process Thread-7] > net.schmizz.sshj.transport.KeyExchanger Sending SSH_MSG_KEXINIT > 2022-05-05 07:26:08,355 DEBUG [sshj-Reader-hostname/192.168.1.3:22] > net.schmizz.sshj.transport.TransportImpl Received SSH_MSG_UNIMPLEMENTED #3 > 2022-05-05 07:26:08,355 ERROR [sshj-Reader-hostname/192.168.1.3:22] > net.schmizz.sshj.transport.TransportImpl Dying because - Received > SSH_MSG_UNIMPLEMENTED while exchanging keys > net.schmizz.sshj.transport.TransportException: Received SSH_MS
[GitHub] [nifi-minifi-cpp] szaszm commented on a diff in pull request #1370: MINIFICPP-1866 Secure connection for ListenSyslog, ListenTCP
szaszm commented on code in PR #1370:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1370#discussion_r948553599
##
libminifi/include/utils/net/SessionHandlingServer.h:
##
@@ -0,0 +1,67 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#pragma once
+
+#include
+#include
+
+#include "Server.h"
+#include "asio/ssl.hpp"
+
+namespace org::apache::nifi::minifi::utils::net {
+
+template
+class SessionHandlingServer : public Server {
+ public:
+ SessionHandlingServer(std::optional max_queue_size, uint16_t port,
std::shared_ptr logger)
+ : Server(max_queue_size, std::move(logger)),
+acceptor_(io_context_, asio::ip::tcp::endpoint(asio::ip::tcp::v4(),
port)) {
+ }
+
+ void run() override {
+startAccept();
+Server::run();
+ }
+
+ protected:
+ void startAccept() {
+auto new_session = createSession();
+acceptor_.async_accept(new_session->getSocket(),
+ [this, new_session](const auto& error_code) -> void
{
+ handleAccept(new_session, error_code);
+ });
Review Comment:
Did you query the socket in a separate statement? Because the evaluation
order of function arguments are unspecified, so if you don't extract the socket
into a separate variable, you can easily move from new_session (as part of the
lambda expression) before calling getSocket on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] mr1716 commented on pull request #6311: NIFI-10367 update easy-rules-mvel to version 4.1.0
mr1716 commented on PR #6311: URL: https://github.com/apache/nifi/pull/6311#issuecomment-1218693613 @exceptionfactory So instead of updating, so I will remove the dependency completely. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] emiliosetiadarma opened a new pull request, #6313: NIFI-9451 add input character set property for PutEmail and additiona…
emiliosetiadarma opened a new pull request, #6313: URL: https://github.com/apache/nifi/pull/6313 …l tests # Summary [NIFI-9451](https://issues.apache.org/jira/browse/NIFI-9451) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [x] Pull Request based on current revision of the `main` branch - [x] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [x] Build completed using `mvn clean install -P contrib-check` - [x] JDK 8 - [x] JDK 11 - [x] JDK 17 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-10368) Upgrade jQuery UI to 1.13.2
[
https://issues.apache.org/jira/browse/NIFI-10368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-10368:
Status: Patch Available (was: Open)
> Upgrade jQuery UI to 1.13.2
> ---
>
> Key: NIFI-10368
> URL: https://issues.apache.org/jira/browse/NIFI-10368
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core UI
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Labels: dependency-upgrade
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The jQuery UI library before version 1.13.2 has an XSS vulnerability
> described in [CVE-2022-31160|https://nvd.nist.gov/vuln/detail/CVE-2022-31160].
> Although Apache NiFi does not make direct use of the vulnerable
> {{checkboxradio()}} function, upgrading jQuery UI to 1.13.2 mitigates
> potential issues and moves away from version 1.12.1, which is no longer
> supported.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [nifi] exceptionfactory opened a new pull request, #6312: NIFI-10368 Upgrade jQuery UI from 1.12.1 to 1.13.2
exceptionfactory opened a new pull request, #6312: URL: https://github.com/apache/nifi/pull/6312 # Summary [NIFI-10368](https://issues.apache.org/jira/browse/NIFI-10368) Upgrades jQuery UI from 1.12.1 to 1.13.2 to resolve CVE-2022-31160, related to the `checkboxradio()` function, which Apache NiFi does not use directly. The 1.12.1 version of jQuery UI is no longer supported, and version [1.13.2](https://jqueryui.com/changelog/1.13.2/) provides a compatible upgrade with a number of fixes added in [1.13.0](https://jqueryui.com/changelog/1.13.0/). # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X] Pull Request based on current revision of the `main` branch - [X] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [X] Build completed using `mvn clean install -P contrib-check` - [X] JDK 8 - [ ] JDK 11 - [ ] JDK 17 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-10368) Upgrade jQuery UI to 1.13.2
David Handermann created NIFI-10368:
---
Summary: Upgrade jQuery UI to 1.13.2
Key: NIFI-10368
URL: https://issues.apache.org/jira/browse/NIFI-10368
Project: Apache NiFi
Issue Type: Improvement
Components: Core UI
Reporter: David Handermann
Assignee: David Handermann
The jQuery UI library before version 1.13.2 has an XSS vulnerability described
in [CVE-2022-31160|https://nvd.nist.gov/vuln/detail/CVE-2022-31160].
Although Apache NiFi does not make direct use of the vulnerable
{{checkboxradio()}} function, upgrading jQuery UI to 1.13.2 mitigates potential
issues and moves away from version 1.12.1, which is no longer supported.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6310: NIFI-10366: Make Default Run Duration configurable
exceptionfactory commented on code in PR #6310:
URL: https://github.com/apache/nifi/pull/6310#discussion_r948394015
##
nifi-api/src/main/java/org/apache/nifi/annotation/behavior/RunDuration.java:
##
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.annotation.behavior;
+
+public enum RunDuration {
+ZERO_MILLIS(0),
+TWENTY_FIVE_MILLIS(25),
+FIFTY_MILLIS(50),
+ONE_HUNDRED_MILLIS(100),
+TWO_HUNDRED_FIFTY_MILLIS(250),
+FIVE_HUNDRED_MILLIS(500),
+ONE_SECONDS(1000),
+TWO_SECONDS(2000);
+
+private final long millis;
+
+RunDuration(final long millis) {
+this.millis = millis;
+}
+
+public long getMillis() {
+return millis;
+}
Review Comment:
@Lehel44 and @markap14, what do you think about returning a
[java.time.Duration](https://docs.oracle.com/javase/8/docs/api/java/time/Duration.html)
object? That would provide greater flexibility on the consuming side, and also
preserve the intent of the time being a duration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] markap14 commented on a diff in pull request #6310: NIFI-10366: Make Default Run Duration configurable
markap14 commented on code in PR #6310:
URL: https://github.com/apache/nifi/pull/6310#discussion_r948385921
##
nifi-api/src/main/java/org/apache/nifi/annotation/behavior/SupportsBatching.java:
##
@@ -47,5 +47,5 @@
@Retention(RetentionPolicy.RUNTIME)
@Inherited
public @interface SupportsBatching {
-
+RunDuration duration() default RunDuration.ZERO_MILLIS;
Review Comment:
We should probably call this `defaultDuration` or `defaultRunDuration`,
rather than just `duration`. Can we also add some JavaDocs here to explain
exactly what happens when this value is set?
##
nifi-api/src/main/java/org/apache/nifi/annotation/behavior/RunDuration.java:
##
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.annotation.behavior;
+
+public enum RunDuration {
+ZERO_MILLIS(0),
Review Comment:
Perhaps it makes sense to call this `NO_BATCHING`? Will leave it up to you,
if you think `ZERO_MILLIS` works better than `NO_BATCHING` that's fine, just
wanted to make the suggestion
##
nifi-api/src/main/java/org/apache/nifi/annotation/behavior/RunDuration.java:
##
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.annotation.behavior;
+
+public enum RunDuration {
+ZERO_MILLIS(0),
+TWENTY_FIVE_MILLIS(25),
+FIFTY_MILLIS(50),
+ONE_HUNDRED_MILLIS(100),
+TWO_HUNDRED_FIFTY_MILLIS(250),
+FIVE_HUNDRED_MILLIS(500),
+ONE_SECONDS(1000),
Review Comment:
Should probably use `ONE_SECOND` rather than `ONE_SECONDS`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6300: NIFI-9374 Add Standard Deprecation Logger
exceptionfactory commented on code in PR #6300:
URL: https://github.com/apache/nifi/pull/6300#discussion_r948372681
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/logback.xml:
##
@@ -88,6 +88,17 @@
+
+
${org.apache.nifi.bootstrap.config.log.dir}/nifi-deprecation.log
+
+
${org.apache.nifi.bootstrap.config.log.dir}/nifi-deprecation_%d.log
+30
+
+
+%date %level [%thread] %logger %msg%n
Review Comment:
The logger name is not truncated in this specific scenario because it is
important to know the exact class name containing deprecated references.
Under optimal circumstances, the nifi-deprecation.log will be empty, but
when a deployment includes deprecated components or features, the logger name
and associated stack trace will provide a reference to component usage that
needs to be changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] Lehel44 commented on a diff in pull request #6300: NIFI-9374 Add Standard Deprecation Logger
Lehel44 commented on code in PR #6300:
URL: https://github.com/apache/nifi/pull/6300#discussion_r948370365
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/logback.xml:
##
@@ -88,6 +88,17 @@
+
+
${org.apache.nifi.bootstrap.config.log.dir}/nifi-deprecation.log
+
+
${org.apache.nifi.bootstrap.config.log.dir}/nifi-deprecation_%d.log
+30
+
+
+%date %level [%thread] %logger %msg%n
Review Comment:
Do you think the logger name could be shortened in the layout pattern with
an upper limit as in other examples?
The abbreviation algorithm with examples can be found
[here](https://logback.qos.ch/manual/layouts.html#conversionWord).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (NIFI-1130) InvokeHttp option to reuse connections
[
https://issues.apache.org/jira/browse/NIFI-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann resolved NIFI-1130.
Resolution: Fixed
The {{Max Idle Connections}} property implemented in NIFI-7393 enabled
configurable connection pooling with OkHttp.
> InvokeHttp option to reuse connections
> --
>
> Key: NIFI-1130
> URL: https://issues.apache.org/jira/browse/NIFI-1130
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joe Percivall
>Priority: Minor
>
> The original ticket for this was a collections of improvements to InvokeHttp
> but it was out the scope of the ticket. The original wording is:
> Configuration option for the number of files pulled from the queue at a time
> What this is asking for is a way to reuse connections. This is a bit
> different when we use OkHttp vs HttpUrlConnection and was much more important
> to HttpUrlConnection because it was slower. One challenge facing this is that
> some of the processor properties relating to the connection rely on FlowFile
> attributes so the connection could change in the middle of processing a
> queue.
> If we did this the user would need to be aware that if I problem came up in
> the middle of processing a batch the non-idempotent actions that occured
> outside of NiFi can't be rolled back.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-8029) InvokeHTTP throwing illegalArgumentException when dynamic property has accent
[ https://issues.apache.org/jira/browse/NIFI-8029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-8029: --- Assignee: David Handermann Resolution: Information Provided Status: Resolved (was: Patch Available) As described in comments on the associated pull request, [RFC 2616 Section 2.2|https://www.rfc-editor.org/rfc/rfc2616#section-2.2] limits allowed characters in HTTP Headers. The OkHttp library supporting InvokeHTTP enforces these constraints. Although it is possible to encode other characters, an encoding strategy is not a universal solution, and would require the receive HTTP server to understand and decode the headers. For this reason, it would be necessary to encode header names and values prior to sending them through InvokeHTTP. > InvokeHTTP throwing illegalArgumentException when dynamic property has accent > - > > Key: NIFI-8029 > URL: https://issues.apache.org/jira/browse/NIFI-8029 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework > Environment: Linux/Windows >Reporter: André Luiz Fernandes do Prado >Assignee: David Handermann >Priority: Trivial > Attachments: Seleção_042.png, Seleção_043.png, Seleção_044.png, > Seleção_046.png, Seleção_047.png > > Time Spent: 1h > Remaining Estimate: 0h > > !Seleção_047.png|width=932,height=671! > !Seleção_046.png|width=933,height=345! > > When i set a dynamic property with accent invokehttp is throwing an > java.lang.illegalArgumentException and when it is without accent this is > working normally. > > !Seleção_042.png|width=920,height=370! > !Seleção_043.png|width=921,height=651! > !Seleção_044.png|width=920,height=609! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-7084) Upgrade from 1.10 to 1.11 fails
[ https://issues.apache.org/jira/browse/NIFI-7084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-7084. Resolution: Cannot Reproduce This appears to be an issue with missing NAR files in the referenced NiFi lib directory, and does not seem to be a problem in more recent versions. This issue could be reopened if it impacts more recent versions. > Upgrade from 1.10 to 1.11 fails > --- > > Key: NIFI-7084 > URL: https://issues.apache.org/jira/browse/NIFI-7084 > Project: Apache NiFi > Issue Type: Bug > Components: Docker >Affects Versions: 1.10.0, 1.11.0 > Environment: Docker version 19.03.2 > NiFi 1.10.0 and NiFi 1.11.0 >Reporter: James Sevener >Priority: Major > Attachments: nifi.1.11.stacktrace > > > Spun up a fresh NiFi 1.10 instance and created an InvokeHTTP processor, then > SSL context service. Then upgraded the instance from 1.10.0 to 1.11.0 and get > the following error. > Caused by: java.lang.IllegalStateException: > org.apache.nifi.ssl.StandardRestrictedSSLContextService from > org.apache.nifi:nifi-ssl-context-service-nar:1.10.0 is not known to this NiFi > instance. > > Attached is the full stack trace from the error to the instance shutdown. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (MINIFICPP-1910) Rename main directory to avoid name clashes with main branch
Ádám Markovics created MINIFICPP-1910: - Summary: Rename main directory to avoid name clashes with main branch Key: MINIFICPP-1910 URL: https://issues.apache.org/jira/browse/MINIFICPP-1910 Project: Apache NiFi MiNiFi C++ Issue Type: Improvement Reporter: Ádám Markovics Assignee: Ádám Markovics Git commands always have to be told whether I meant main as a filename or a revision name. By renaming the directory ambiguity would be removed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-3125) GetHTTP username property should support EL statements
[
https://issues.apache.org/jira/browse/NIFI-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann resolved NIFI-3125.
Resolution: Won't Fix
GetHTTP is deprecated in favor of InvokeHTTP. Component properties can be
abstracted using Parameter Contexts as of NiFi 1.10.0.
> GetHTTP username property should support EL statements
> --
>
> Key: NIFI-3125
> URL: https://issues.apache.org/jira/browse/NIFI-3125
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.1.0
> Environment: $ sw_vers
> ProductName: Mac OS X
> ProductVersion: 10.12.1
> BuildVersion: 16B2555
> $ java -version
> java version "1.8.0_112"
> Java(TM) SE Runtime Environment (build 1.8.0_112-b16)
> Java HotSpot(TM) 64-Bit Server VM (build 25.112-b16, mixed mode)
>Reporter: Alec Wenzowski
>Priority: Minor
>
> Given a variable registry property file has been configured
> {code:title=conf/nifi.properties|borderStyle=solid}
> nifi.variable.registry.properties=./conf/service.properties
> {code}
> And it contains one valid key
> {code:title=conf/service.properties|borderStyle=solid}
> nifi.service.api_key=pony_foo
> {code}
> Upon booting nifi and inspecting I see
> {code:title=logs/nifi-app.log|borderStyle=solid}
> 745:2016-11-29 19:12:47,020 INFO [main]
> o.a.nifi.util.FileBasedVariableRegistry Loaded 1 properties from
> './conf/service.properties'
> {code}
> Yet when I configure a {{GetHTTP}} processor with its Username property set
> to the Value $\{nifi.service.api_key\} it issues HTTP requests without
> authentication–the configured HTTP Basic Authentication username of
> `pony_foo` is not used–and behaves as if the $\{nifi.service.api_key\} value
> has not been set.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (NIFI-3280) PostHTTP Option to write response to attribute or flowfile content
[ https://issues.apache.org/jira/browse/NIFI-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-3280. Resolution: Won't Fix PostHTTP is deprecated, and InvokeHTTP implements the capabilities described. > PostHTTP Option to write response to attribute or flowfile content > -- > > Key: NIFI-3280 > URL: https://issues.apache.org/jira/browse/NIFI-3280 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Nicholas Carenza >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > Currently there doesn't seem to be a way to access anything about the > response of an HTTP request from the PostHTTP processor other than whether it > failed or succeeded. > I would like to be able to: > - store the response body in an attribute or replace the flowfile content > - store the status code in an attribute -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-6057) For InvokeHTTP processor, allow request body to come from an EL expression
[ https://issues.apache.org/jira/browse/NIFI-6057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-6057. Resolution: Duplicate Closing as duplicate in favor of NIFI-9894. > For InvokeHTTP processor, allow request body to come from an EL expression > -- > > Key: NIFI-6057 > URL: https://issues.apache.org/jira/browse/NIFI-6057 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Alex Savitsky >Assignee: Alex Savitsky >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > Currently, there's either no body, or the body is taken from the flowfile. > This Jira adds the ability to specify an EL expression for the request body > instead. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-6995) InvokeHTTP processor Auth user name didn't support registry variable
[ https://issues.apache.org/jira/browse/NIFI-6995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-6995. Resolution: Workaround As mentioned in comments, integrating Parameter Contexts provides the recommend solution for reusable component configurations. > InvokeHTTP processor Auth user name didn't support registry variable > > > Key: NIFI-6995 > URL: https://issues.apache.org/jira/browse/NIFI-6995 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.9.2 >Reporter: tony wang >Priority: Minor > > InvokeHTTP processor property (Basic Authentication Username) didn't support > registry variable and any changes to this value will impact the nifi registry > version management. It's hard to managed it cross multiple environment in > case different user id used. > It will be better ether mark this property as secure or support registry > variable -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-4831) InvokeHTTP Processor overwrites filename attribute when doing a POST
[ https://issues.apache.org/jira/browse/NIFI-4831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-4831. Resolution: Cannot Reproduce > InvokeHTTP Processor overwrites filename attribute when doing a POST > > > Key: NIFI-4831 > URL: https://issues.apache.org/jira/browse/NIFI-4831 > Project: Apache NiFi > Issue Type: Bug >Reporter: Tyler Bryant >Priority: Minor > > The 'filename' attribute gets overwritten when doing a POST in an InvokeHTTP > Processor. I can seem to get around it by putting a 'fileName' attribute in > before the InvokeHTTP Processor, and then it correctly keeps the 'filename'. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-7596) InvokeHTTP - Get Request Body From Attribute
[ https://issues.apache.org/jira/browse/NIFI-7596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-7596. Resolution: Duplicate Closing as duplicate in favor of NIFI-9894, which describes the same feature request with some additional details. > InvokeHTTP - Get Request Body From Attribute > > > Key: NIFI-7596 > URL: https://issues.apache.org/jira/browse/NIFI-7596 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Mermillod >Priority: Minor > > *Use case :* the flow file contains 1 big data (ex : PDF or pictures), but > you want to lookup metadata with InvokeHTTP using a POST (or PUT). POST is > often used in http API for "search" with lot of params or list of params. > Other exemple is the 3-steps batch-upload of products like Nuxeo or > Salesforce Bulk API : you need to "open" a batch with a POTS invokeHTTP. > > *Proposition* : In the InvokeHTTP, add a "Get Request Body From Attribute" > (like Put Response Body In Attribute). If this field not empty, the chosen > attribute will be read as the request, insted of the flow file. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10052) Avoid obtaining any locks when creating/sending heartbeats
[
https://issues.apache.org/jira/browse/NIFI-10052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-10052:
--
Fix Version/s: 1.18.0
> Avoid obtaining any locks when creating/sending heartbeats
> --
>
> Key: NIFI-10052
> URL: https://issues.apache.org/jira/browse/NIFI-10052
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Hsin-Ying Lee
>Priority: Major
> Labels: cluster, heartbeat, stability
> Fix For: 1.18.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When NiFi creates a heartbeat to send to the coordinator, it must obtain a
> few locks in order to generate that heartbeat. We should avoid obtaining any
> read locks, write locks, or synchronized monitors, especially those that may
> be held for a while. Doing so can result in NiFi getting disconnected from
> the cluster if a write lock is held for a long time.
> Specifically, the following locks are obtained, at minimum:
> * FlowController readLock in the createHeartbeatMessage() method. Due to
> refactoring, this read lock is not necessary at all.
> * revisionManager.getRevisionUpdateCount() is synchronized. However, the
> synchronization here is not needed, as it just returns an AtomicLong.get().
> This is perhaps the most important lock to avoid because any update to a
> component or group of components happens within
> revisionManager.updateRevision, which also is synchronized. So a large
> request like deleting thousands of components will block heartbeats from
> being created until this completes.
> * FlowController.getTotalFlowFileCount - this may be the most challenging to
> eliminate. It calls ProcessGroup.getConnections() and
> ProcessGroup.getProcessGroups(), which means that it must obtain the read
> lock of the Process Group twice - for every Process Group in the flow. We may
> be able to change StandardProcessGroup's connections and processGroups maps
> to ConcurrentHashMap's and just introduce a getQueueSize() method on
> ProcessGroup that can avoid having to lock so much
> * This createHeartbeatMessage() method also appears to reference
> FlowController's {{connectionStatus}} member variable without any locks,
> although it is not volatile and documentation indicates that it's guarded by
> read/write lock. So that needs to be addressed in order to ensure that the
> connectionStatus is always accurately reported.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-10052) Avoid obtaining any locks when creating/sending heartbeats
[
https://issues.apache.org/jira/browse/NIFI-10052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-10052:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Avoid obtaining any locks when creating/sending heartbeats
> --
>
> Key: NIFI-10052
> URL: https://issues.apache.org/jira/browse/NIFI-10052
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Hsin-Ying Lee
>Priority: Major
> Labels: cluster, heartbeat, stability
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When NiFi creates a heartbeat to send to the coordinator, it must obtain a
> few locks in order to generate that heartbeat. We should avoid obtaining any
> read locks, write locks, or synchronized monitors, especially those that may
> be held for a while. Doing so can result in NiFi getting disconnected from
> the cluster if a write lock is held for a long time.
> Specifically, the following locks are obtained, at minimum:
> * FlowController readLock in the createHeartbeatMessage() method. Due to
> refactoring, this read lock is not necessary at all.
> * revisionManager.getRevisionUpdateCount() is synchronized. However, the
> synchronization here is not needed, as it just returns an AtomicLong.get().
> This is perhaps the most important lock to avoid because any update to a
> component or group of components happens within
> revisionManager.updateRevision, which also is synchronized. So a large
> request like deleting thousands of components will block heartbeats from
> being created until this completes.
> * FlowController.getTotalFlowFileCount - this may be the most challenging to
> eliminate. It calls ProcessGroup.getConnections() and
> ProcessGroup.getProcessGroups(), which means that it must obtain the read
> lock of the Process Group twice - for every Process Group in the flow. We may
> be able to change StandardProcessGroup's connections and processGroups maps
> to ConcurrentHashMap's and just introduce a getQueueSize() method on
> ProcessGroup that can avoid having to lock so much
> * This createHeartbeatMessage() method also appears to reference
> FlowController's {{connectionStatus}} member variable without any locks,
> although it is not volatile and documentation indicates that it's guarded by
> read/write lock. So that needs to be addressed in order to ensure that the
> connectionStatus is always accurately reported.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-10052) Avoid obtaining any locks when creating/sending heartbeats
[
https://issues.apache.org/jira/browse/NIFI-10052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17580923#comment-17580923
]
ASF subversion and git services commented on NIFI-10052:
Commit 2685856c629aa1bda20019981ed932ccecf9415a in nifi's branch
refs/heads/main from Hsin-Ying Lee
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=2685856c62 ]
NIFI-10052 Avoid obtaining any locks when creating/sending heartbeats (#6298)
> Avoid obtaining any locks when creating/sending heartbeats
> --
>
> Key: NIFI-10052
> URL: https://issues.apache.org/jira/browse/NIFI-10052
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Hsin-Ying Lee
>Priority: Major
> Labels: cluster, heartbeat, stability
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When NiFi creates a heartbeat to send to the coordinator, it must obtain a
> few locks in order to generate that heartbeat. We should avoid obtaining any
> read locks, write locks, or synchronized monitors, especially those that may
> be held for a while. Doing so can result in NiFi getting disconnected from
> the cluster if a write lock is held for a long time.
> Specifically, the following locks are obtained, at minimum:
> * FlowController readLock in the createHeartbeatMessage() method. Due to
> refactoring, this read lock is not necessary at all.
> * revisionManager.getRevisionUpdateCount() is synchronized. However, the
> synchronization here is not needed, as it just returns an AtomicLong.get().
> This is perhaps the most important lock to avoid because any update to a
> component or group of components happens within
> revisionManager.updateRevision, which also is synchronized. So a large
> request like deleting thousands of components will block heartbeats from
> being created until this completes.
> * FlowController.getTotalFlowFileCount - this may be the most challenging to
> eliminate. It calls ProcessGroup.getConnections() and
> ProcessGroup.getProcessGroups(), which means that it must obtain the read
> lock of the Process Group twice - for every Process Group in the flow. We may
> be able to change StandardProcessGroup's connections and processGroups maps
> to ConcurrentHashMap's and just introduce a getQueueSize() method on
> ProcessGroup that can avoid having to lock so much
> * This createHeartbeatMessage() method also appears to reference
> FlowController's {{connectionStatus}} member variable without any locks,
> although it is not volatile and documentation indicates that it's guarded by
> read/write lock. So that needs to be addressed in order to ensure that the
> connectionStatus is always accurately reported.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [nifi] markap14 merged pull request #6298: NIFI-10052 Avoid obtaining any locks when creating/sending heartbeats
markap14 merged PR #6298: URL: https://github.com/apache/nifi/pull/6298 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] markap14 commented on pull request #6298: NIFI-10052 Avoid obtaining any locks when creating/sending heartbeats
markap14 commented on PR #6298: URL: https://github.com/apache/nifi/pull/6298#issuecomment-1218337579 Thanks for putting up the PR @s9514171 ! The code looks good. Addresses all issues outlined in ticket. All unit & system tests pass. +1 will merge to main -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (NIFI-10022) InvokeHTTP. Retry Relationships
[
https://issues.apache.org/jira/browse/NIFI-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann resolved NIFI-10022.
-
Resolution: Information Provided
The framework retry relationships configuration will work based on the
configuration, but after eventual failure, FlowFiles will be routed to the
appropriate relationship. In this case, an HTTP 404 would be routed to the No
Retry relationship after exceeded the configured number of retry attempts. That
may not be the most intuitive approach for every processor, such as InvokeHTTP
where there are multiple types of failures, but that is the intended behavior.
> InvokeHTTP. Retry Relationships
> ---
>
> Key: NIFI-10022
> URL: https://issues.apache.org/jira/browse/NIFI-10022
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Zaytsev Sergey
>Priority: Major
> Attachments: Test_case_InvokeHTTP.xml
>
>
> Hi,
> I observe incorrect operation of repeated requests in the InvokeHTTP
> processor.
> Send a request to the address, ex. GET www.google.com/1.
> Get response 404.
> I go to repeat on the No Retry relationships.
> Setting retry:
> "Number Of Retry Attempts": 5
> "Retry Back Off Policy": Penalize
> "Retry Maximum Back Off Period": 10 sec
> If "Number Of Retry Attempts" is exceeded, attribute
> retryCount.\{processor_id} is reset.
> Requests will be repeated until a 2xx response is received or a timeout
> expires.
> I assume that when the connection limit is exceeded, the executable file of
> the stream should be sent along the Failure relationships.
> NiFI 1.16.1
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-9993) Processors cannot write to Content Repo: Unable to write flowfile content to content repository container default due to archive file size constraints; waiting for archi
[
https://issues.apache.org/jira/browse/NIFI-9993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17580919#comment-17580919
]
Mark Payne commented on NIFI-9993:
--
[~moghammed] there is no workaround that I'm aware of
[~malthe] I am not 100% sure but i suspect it was introduced somewhere around
version 1.12.0
> Processors cannot write to Content Repo: Unable to write flowfile content to
> content repository container default due to archive file size constraints;
> waiting for archive cleanup
> ---
>
> Key: NIFI-9993
> URL: https://issues.apache.org/jira/browse/NIFI-9993
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.17.0, 1.16.2
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Sometimes, we can see NiFi log the following log message over and over:
> {code:java}
> 2022-05-05 10:47:03,267 WARN [Timer-Driven Process Thread-142]
> o.a.n.c.repository.FileSystemRepository Unable to write to container default
> due to archive file size constraints; waiting for archive cleanup {code}
> However, we can also see that there's nothing in the archive to cleanup,
> based on the fact that we also continually see this log message:
> {code:java}
> 2022-05-05 10:01:02,515 INFO [Cleanup Archive for default]
> o.a.n.c.repository.FileSystemRepository Successfully deleted 0 files (0
> bytes) from archive {code}
> If there are any files in the archive directory, we should see them getting
> cleaned up. The fact that we constantly see "Successfully deleted 0 files (0
> bytes) from archive" implies there is nothing in the archive.
> But if there is nothing in the archive, then we should not be seeing any
> notices about "Unable to write to container default due to archive file size
> constraints."
> This results in processors getting blocked for indefinite periods of time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-9015) Ability to derive custom Rest API processes from InvokeHTTP
[ https://issues.apache.org/jira/browse/NIFI-9015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17580896#comment-17580896 ] David Handermann commented on NIFI-9015: Thanks [~otto]! > Ability to derive custom Rest API processes from InvokeHTTP > --- > > Key: NIFI-9015 > URL: https://issues.apache.org/jira/browse/NIFI-9015 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > In setups with custom development, there is a lot of boilerplate > customization to InvokeHTTP around some known set of rest apis. As such flow > developers / users have to 'know' and understand these things in order to > setup possibly multiple InvokeHTTP instances with these details. > Some users may instead want to create a top level derived processor with > custom setup and parameters for a specific rest api for easy configuration > and deployment. > The > [InvokeAWSGatewayApi|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-aws-nar/1.14.0/org.apache.nifi.processors.aws.wag.InvokeAWSGatewayApi/index.html] > processor ( which itself had to be copied from InvokeHTTP because of this > limitation ) offers this, such that you can create a derived processor for a > custom AWS Web Gateway service. > With a coming processor registry, the ability for people to contribute > specialized processors for certain APIs would be great. These may be a > subset or a super set of the existing properties. We may also move the > 'base' properties to a new UI tab in the configuration, such that you can add > custom extra configuration in the first focus tab. We could also allow for > removing properties as well from the base tab, etc etc > This task would involve refactoring the invokeHTTP processor such that there > is a reusable base, with the InvokeHTTP processor being a pass-through > default implementation ( IE> a new derived from base would have all the same > features as InvokeHTTP ). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-5232) HttpConnectionService controller service
[ https://issues.apache.org/jira/browse/NIFI-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17580894#comment-17580894 ] David Handermann commented on NIFI-5232: https://github.com/apache/nifi/pull/6268 is the pull request for NIFI-10244, which contains the implementation, now merged into main. > HttpConnectionService controller service > > > Key: NIFI-5232 > URL: https://issues.apache.org/jira/browse/NIFI-5232 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Mike Thomsen >Priority: Major > > The functionality of InvokeHttp and related processors should be copied over > to a controller service that can do much the same thing. This controller > service would be able to handle all of the common scenarios with HTTP > connections from processors going forward. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-9015) Ability to derive custom Rest API processes from InvokeHTTP
[ https://issues.apache.org/jira/browse/NIFI-9015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler resolved NIFI-9015. --- Resolution: Duplicate This is accomplished in NIFI-10244 > Ability to derive custom Rest API processes from InvokeHTTP > --- > > Key: NIFI-9015 > URL: https://issues.apache.org/jira/browse/NIFI-9015 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > In setups with custom development, there is a lot of boilerplate > customization to InvokeHTTP around some known set of rest apis. As such flow > developers / users have to 'know' and understand these things in order to > setup possibly multiple InvokeHTTP instances with these details. > Some users may instead want to create a top level derived processor with > custom setup and parameters for a specific rest api for easy configuration > and deployment. > The > [InvokeAWSGatewayApi|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-aws-nar/1.14.0/org.apache.nifi.processors.aws.wag.InvokeAWSGatewayApi/index.html] > processor ( which itself had to be copied from InvokeHTTP because of this > limitation ) offers this, such that you can create a derived processor for a > custom AWS Web Gateway service. > With a coming processor registry, the ability for people to contribute > specialized processors for certain APIs would be great. These may be a > subset or a super set of the existing properties. We may also move the > 'base' properties to a new UI tab in the configuration, such that you can add > custom extra configuration in the first focus tab. We could also allow for > removing properties as well from the base tab, etc etc > This task would involve refactoring the invokeHTTP processor such that there > is a reusable base, with the InvokeHTTP processor being a pass-through > default implementation ( IE> a new derived from base would have all the same > features as InvokeHTTP ). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (NIFI-9015) Ability to derive custom Rest API processes from InvokeHTTP
[ https://issues.apache.org/jira/browse/NIFI-9015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler closed NIFI-9015. - > Ability to derive custom Rest API processes from InvokeHTTP > --- > > Key: NIFI-9015 > URL: https://issues.apache.org/jira/browse/NIFI-9015 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > In setups with custom development, there is a lot of boilerplate > customization to InvokeHTTP around some known set of rest apis. As such flow > developers / users have to 'know' and understand these things in order to > setup possibly multiple InvokeHTTP instances with these details. > Some users may instead want to create a top level derived processor with > custom setup and parameters for a specific rest api for easy configuration > and deployment. > The > [InvokeAWSGatewayApi|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-aws-nar/1.14.0/org.apache.nifi.processors.aws.wag.InvokeAWSGatewayApi/index.html] > processor ( which itself had to be copied from InvokeHTTP because of this > limitation ) offers this, such that you can create a derived processor for a > custom AWS Web Gateway service. > With a coming processor registry, the ability for people to contribute > specialized processors for certain APIs would be great. These may be a > subset or a super set of the existing properties. We may also move the > 'base' properties to a new UI tab in the configuration, such that you can add > custom extra configuration in the first focus tab. We could also allow for > removing properties as well from the base tab, etc etc > This task would involve refactoring the invokeHTTP processor such that there > is a reusable base, with the InvokeHTTP processor being a pass-through > default implementation ( IE> a new derived from base would have all the same > features as InvokeHTTP ). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-9015) Ability to derive custom Rest API processes from InvokeHTTP
[ https://issues.apache.org/jira/browse/NIFI-9015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler reassigned NIFI-9015: - Assignee: Otto Fowler > Ability to derive custom Rest API processes from InvokeHTTP > --- > > Key: NIFI-9015 > URL: https://issues.apache.org/jira/browse/NIFI-9015 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > In setups with custom development, there is a lot of boilerplate > customization to InvokeHTTP around some known set of rest apis. As such flow > developers / users have to 'know' and understand these things in order to > setup possibly multiple InvokeHTTP instances with these details. > Some users may instead want to create a top level derived processor with > custom setup and parameters for a specific rest api for easy configuration > and deployment. > The > [InvokeAWSGatewayApi|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-aws-nar/1.14.0/org.apache.nifi.processors.aws.wag.InvokeAWSGatewayApi/index.html] > processor ( which itself had to be copied from InvokeHTTP because of this > limitation ) offers this, such that you can create a derived processor for a > custom AWS Web Gateway service. > With a coming processor registry, the ability for people to contribute > specialized processors for certain APIs would be great. These may be a > subset or a super set of the existing properties. We may also move the > 'base' properties to a new UI tab in the configuration, such that you can add > custom extra configuration in the first focus tab. We could also allow for > removing properties as well from the base tab, etc etc > This task would involve refactoring the invokeHTTP processor such that there > is a reusable base, with the InvokeHTTP processor being a pass-through > default implementation ( IE> a new derived from base would have all the same > features as InvokeHTTP ). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-9015) Ability to derive custom Rest API processes from InvokeHTTP
[ https://issues.apache.org/jira/browse/NIFI-9015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17580890#comment-17580890 ] Otto Fowler commented on NIFI-9015: --- [~exceptionfactory]: I have been following the PR review and I think that great work. I'll close this ticket. > Ability to derive custom Rest API processes from InvokeHTTP > --- > > Key: NIFI-9015 > URL: https://issues.apache.org/jira/browse/NIFI-9015 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Otto Fowler >Priority: Major > > In setups with custom development, there is a lot of boilerplate > customization to InvokeHTTP around some known set of rest apis. As such flow > developers / users have to 'know' and understand these things in order to > setup possibly multiple InvokeHTTP instances with these details. > Some users may instead want to create a top level derived processor with > custom setup and parameters for a specific rest api for easy configuration > and deployment. > The > [InvokeAWSGatewayApi|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-aws-nar/1.14.0/org.apache.nifi.processors.aws.wag.InvokeAWSGatewayApi/index.html] > processor ( which itself had to be copied from InvokeHTTP because of this > limitation ) offers this, such that you can create a derived processor for a > custom AWS Web Gateway service. > With a coming processor registry, the ability for people to contribute > specialized processors for certain APIs would be great. These may be a > subset or a super set of the existing properties. We may also move the > 'base' properties to a new UI tab in the configuration, such that you can add > custom extra configuration in the first focus tab. We could also allow for > removing properties as well from the base tab, etc etc > This task would involve refactoring the invokeHTTP processor such that there > is a reusable base, with the InvokeHTTP processor being a pass-through > default implementation ( IE> a new derived from base would have all the same > features as InvokeHTTP ). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-5232) HttpConnectionService controller service
[ https://issues.apache.org/jira/browse/NIFI-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17580885#comment-17580885 ] Mike Thomsen commented on NIFI-5232: Sure. Got a link? > HttpConnectionService controller service > > > Key: NIFI-5232 > URL: https://issues.apache.org/jira/browse/NIFI-5232 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Mike Thomsen >Priority: Major > > The functionality of InvokeHttp and related processors should be copied over > to a controller service that can do much the same thing. This controller > service would be able to handle all of the common scenarios with HTTP > connections from processors going forward. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] mr1716 opened a new pull request, #6311: NIFI-10367 update easy-rules-mvel to version 4.1.0
mr1716 opened a new pull request, #6311: URL: https://github.com/apache/nifi/pull/6311 # Summary [NIFI-10367](https://issues.apache.org/jira/browse/NIFI-10367) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI-10367) issue created ### Pull Request Tracking - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X] Pull Request based on current revision of the `main` branch - [X] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [X] JDK 8 - [X] JDK 11 - [X] JDK 17 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-10366) Make Default Run Duration configurable
[ https://issues.apache.org/jira/browse/NIFI-10366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lehel Boér updated NIFI-10366: -- Status: Patch Available (was: In Progress) > Make Default Run Duration configurable > -- > > Key: NIFI-10366 > URL: https://issues.apache.org/jira/browse/NIFI-10366 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Lehel Boér >Assignee: Lehel Boér >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > We should provide a duration parameter is @SupportsBatching to set default > run duration. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-10367) update easy-rules-mvel to version 4.1.0
Mike R created NIFI-10367: - Summary: update easy-rules-mvel to version 4.1.0 Key: NIFI-10367 URL: https://issues.apache.org/jira/browse/NIFI-10367 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.17.0 Reporter: Mike R update easy-rules-mvel to version 4.1.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] Lehel44 opened a new pull request, #6310: NIFI-10366: Make Default Run Duration configurable
Lehel44 opened a new pull request, #6310: URL: https://github.com/apache/nifi/pull/6310 # Summary [NIFI-10366](https://issues.apache.org/jira/browse/NIFI-10366) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI-10366) issue created ### Pull Request Tracking - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [ ] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 8 - [ ] JDK 11 - [ ] JDK 17 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[
https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17580868#comment-17580868
]
Joe Witt commented on NIFI-9572:
[~giocapi] Here again as the examples before the lsof output provided shows
only 4000 or so open files (4192 in your case). If you hit out of file limits
with only 4000 or so files open yet you think the limit on your system is 999K
then something isn't right. We've seen more and more confusion around what
people meant to set for max open files vs what they're actually seeing. You
will want to show more from your logs/output and dumps so we can get the full
context of what might be happening for you. Thanks
> Failed to index Provenance Events and (Too many Files)
> --
>
> Key: NIFI-9572
> URL: https://issues.apache.org/jira/browse/NIFI-9572
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
>Affects Versions: 1.15.2
>Reporter: mayki
>Priority: Major
> Attachments: bootstrap.conf, lsof (1).txt, nifi-app.log,
> nifi-app.log.tar.gz, nifi.properties, nifi_691106_pid.tar.gz
>
>
> Hello
> I have upgraded NIFI 1.15.2 since 2022/01/05
> No issue until this night 2022/01/13
> * nifi version 1.15.2
> * jdk-1.8.0_311
> And the limit is high
> {code:java}
> Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2
> -bash-4.2@nifi$ ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) rg
> pending signals (-i) 63278
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 5
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 1
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
> {code}
>
> We got a lot error about provenance_repository, it fill our filesystem logs ..
>
> {code:java}
> 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2]
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877)
> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891)
> at
> org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468)
> at
> org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444)
> at
> org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70)
> at
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202)
> at
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.nio.file.FileSystemException:
> /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd:
> Too many open files
> {code}
>
>
> We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability.
> But it is impossible to do that if we got this error.
>
> Thanks for you help.
>
> Regards
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-10358) Apply SSL Properties to JDBC Connection in CaptureChangeMySQL
[
https://issues.apache.org/jira/browse/NIFI-10358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-10358:
Fix Version/s: 1.18.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Apply SSL Properties to JDBC Connection in CaptureChangeMySQL
> -
>
> Key: NIFI-10358
> URL: https://issues.apache.org/jira/browse/NIFI-10358
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Fix For: 1.18.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The {{CaptureChangeMySQL}} Processor supports TLS for Binary Log connections
> using the {{SSL Mode}} and {{SSL Context Service}} properties, but these
> settings do not apply to the JDBC enrichment connection.
> Without apply the SSL properties to the JDBC connection,
> {{CaptureChangeMySQL}} depends on the default MySQL JDBC Connector
> configuration to negotiate TLS settings. MySQL JDBC Connector versions prior
> to 8.0.28 enable deprecated TLS versions 1.0 and 1.1, but Java 8 Update 292
> and following disable TLS 1.0 and 1.1 in the default java.security
> configuration. As a result of this behavior, {{CaptureChangeMySQL}} can fail
> to establish a JDBC connection when running on a newer version of Java and an
> older version of the MySQL JDBC Connector. It is possible to work around the
> problem by upgrading to MySQL JDBC Connector 8.0.28 and following, which
> selects TLS 1.2 as the default protocol version. Although this resolves TLS
> protocol negotiation issues, it does not support customization of the TLS
> keystore and truststore properties, which may be necessary for some MySQL
> installations.
> Configuring the JDBC connection properties based on the {{SSL Mode}} and
> {{SSL Context Service}} properties should provide a more intuitive and
> flexible configuration approach.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-10358) Apply SSL Properties to JDBC Connection in CaptureChangeMySQL
[
https://issues.apache.org/jira/browse/NIFI-10358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17580865#comment-17580865
]
ASF subversion and git services commented on NIFI-10358:
Commit 5df6efa0f2d5a21dd545976127f3643ded01d07f in nifi's branch
refs/heads/main from David Handermann
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=5df6efa0f2 ]
NIFI-10358 Updated CaptureChangeMySQL to use SSL JDBC properties
- Added Connection Properties Provider with implementation to translate SSL
Mode and SSL Context Service configuration to MySQL Connector properties
Signed-off-by: Matthew Burgess
This closes #6306
> Apply SSL Properties to JDBC Connection in CaptureChangeMySQL
> -
>
> Key: NIFI-10358
> URL: https://issues.apache.org/jira/browse/NIFI-10358
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The {{CaptureChangeMySQL}} Processor supports TLS for Binary Log connections
> using the {{SSL Mode}} and {{SSL Context Service}} properties, but these
> settings do not apply to the JDBC enrichment connection.
> Without apply the SSL properties to the JDBC connection,
> {{CaptureChangeMySQL}} depends on the default MySQL JDBC Connector
> configuration to negotiate TLS settings. MySQL JDBC Connector versions prior
> to 8.0.28 enable deprecated TLS versions 1.0 and 1.1, but Java 8 Update 292
> and following disable TLS 1.0 and 1.1 in the default java.security
> configuration. As a result of this behavior, {{CaptureChangeMySQL}} can fail
> to establish a JDBC connection when running on a newer version of Java and an
> older version of the MySQL JDBC Connector. It is possible to work around the
> problem by upgrading to MySQL JDBC Connector 8.0.28 and following, which
> selects TLS 1.2 as the default protocol version. Although this resolves TLS
> protocol negotiation issues, it does not support customization of the TLS
> keystore and truststore properties, which may be necessary for some MySQL
> installations.
> Configuring the JDBC connection properties based on the {{SSL Mode}} and
> {{SSL Context Service}} properties should provide a more intuitive and
> flexible configuration approach.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [nifi] mattyb149 closed pull request #6306: NIFI-10358 Update CaptureChangeMySQL to use SSL JDBC properties
mattyb149 closed pull request #6306: NIFI-10358 Update CaptureChangeMySQL to use SSL JDBC properties URL: https://github.com/apache/nifi/pull/6306 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] mattyb149 commented on pull request #6306: NIFI-10358 Update CaptureChangeMySQL to use SSL JDBC properties
mattyb149 commented on PR #6306: URL: https://github.com/apache/nifi/pull/6306#issuecomment-1218230875 +1 LGTM, thanks for the improvement! Merging to main -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-10362) Cluster can disconnect node as soon as it rejoins cluster upon restart
[
https://issues.apache.org/jira/browse/NIFI-10362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-10362:
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Cluster can disconnect node as soon as it rejoins cluster upon restart
> --
>
> Key: NIFI-10362
> URL: https://issues.apache.org/jira/browse/NIFI-10362
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.18.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When the Cluster Coordinator disconnects a node due to a user requesting that
> the node get disconnected, the node is immediately marked as DISCONNECTED,
> and then a background thread is responsible for notifying the node that it's
> been disconnected. The background task attempts several times if it cannot
> successfully send the notification.
> However, if the node is disconnected and then restarted before it's been
> notified, we have a situation in which the node becomes CONNECTING (and
> possibly then CONNECTED), and then the background task is triggered. This
> then results in the node being told that it's DISCONNECTED. But the Cluster
> Coordinator doesn't think so (because its already changed the state back to
> CONNECTING/CONNECTED).
> While the chances that this happens are slim in production and it's easily
> worked around (by simply waiting a few seconds after disconnecting a node
> before restarting it, or just restarting without disconnecting) it causes a
> lot of problems for system tests and potentially other automated activities.
> It results in the following log message in the Cluster Coordinator:
> {code:java}
> 2022-08-15 00:47:50,200 ERROR [Disconnect localhost:5672]
> org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator Failed to
> notify localhost:5672 that it has been disconnected from the cluster due to
> User anonymous requested that node be disconnected from cluster {code}
> And then we see confusing error messages such as:
> {code:java}
> 2022-08-15 00:48:01,461 INFO [Replicate Request Thread-23]
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator
> Received a status of 200 from localhost:5672 for request PUT
> /nifi-api/flow/process-groups/root when performing first stage of two-stage
> commit. The action will not occur. Node explanation:
> {"id":"root","state":"STOPPED"} {code}
> This is because when the cluster coordinator replicates the request to all
> nodes, the node that thinks it is disconnected receives the request and
> performs the action. It then responds with a "200 OK" but it should have
> noted that it's the first phase of a 2-phase action and responded with "201
> Continue".
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [nifi] exceptionfactory closed pull request #6308: NIFI-10362: When asynchronous node disconnect is issued, do not send …
exceptionfactory closed pull request #6308: NIFI-10362: When asynchronous node disconnect is issued, do not send … URL: https://github.com/apache/nifi/pull/6308 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-10362) Cluster can disconnect node as soon as it rejoins cluster upon restart
[
https://issues.apache.org/jira/browse/NIFI-10362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17580862#comment-17580862
]
ASF subversion and git services commented on NIFI-10362:
Commit 21503f6353c33063b7acff5915a94397aad72926 in nifi's branch
refs/heads/main from Mark Payne
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=21503f6353 ]
NIFI-10362: When asynchronous node disconnect is issued, do not send disconnect
to node if the node becomes reconnected in the interim. Also, addressed the
issue in which a disconnected node acts on a replicated request during the
first phase by detect that it's the first phase if configured for cluster, not
when only when connected to a cluster.
This closes #6308
Signed-off-by: David Handermann
> Cluster can disconnect node as soon as it rejoins cluster upon restart
> --
>
> Key: NIFI-10362
> URL: https://issues.apache.org/jira/browse/NIFI-10362
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.18.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When the Cluster Coordinator disconnects a node due to a user requesting that
> the node get disconnected, the node is immediately marked as DISCONNECTED,
> and then a background thread is responsible for notifying the node that it's
> been disconnected. The background task attempts several times if it cannot
> successfully send the notification.
> However, if the node is disconnected and then restarted before it's been
> notified, we have a situation in which the node becomes CONNECTING (and
> possibly then CONNECTED), and then the background task is triggered. This
> then results in the node being told that it's DISCONNECTED. But the Cluster
> Coordinator doesn't think so (because its already changed the state back to
> CONNECTING/CONNECTED).
> While the chances that this happens are slim in production and it's easily
> worked around (by simply waiting a few seconds after disconnecting a node
> before restarting it, or just restarting without disconnecting) it causes a
> lot of problems for system tests and potentially other automated activities.
> It results in the following log message in the Cluster Coordinator:
> {code:java}
> 2022-08-15 00:47:50,200 ERROR [Disconnect localhost:5672]
> org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator Failed to
> notify localhost:5672 that it has been disconnected from the cluster due to
> User anonymous requested that node be disconnected from cluster {code}
> And then we see confusing error messages such as:
> {code:java}
> 2022-08-15 00:48:01,461 INFO [Replicate Request Thread-23]
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator
> Received a status of 200 from localhost:5672 for request PUT
> /nifi-api/flow/process-groups/root when performing first stage of two-stage
> commit. The action will not occur. Node explanation:
> {"id":"root","state":"STOPPED"} {code}
> This is because when the cluster coordinator replicates the request to all
> nodes, the node that thinks it is disconnected receives the request and
> performs the action. It then responds with a "200 OK" but it should have
> noted that it's the first phase of a 2-phase action and responded with "201
> Continue".
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [nifi-minifi-cpp] fgerlits commented on a diff in pull request #1370: MINIFICPP-1866 Secure connection for ListenSyslog, ListenTCP
fgerlits commented on code in PR #1370:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1370#discussion_r948147376
##
extensions/standard-processors/tests/unit/ListenSyslogTests.cpp:
##
@@ -439,12 +439,13 @@ TEST_CASE("Test ListenSyslog via TCP with SSL
connection", "[ListenSyslog]") {
SingleProcessorTestController controller{listen_syslog};
auto ssl_context_service =
controller.plan->addController("SSLContextService", "SSLContextService");
+ const auto executable_dir =
minifi::utils::file::FileUtils::get_executable_dir();
REQUIRE(controller.plan->setProperty(ssl_context_service,
controllers::SSLContextService::CACertificate.getName(),
-minifi::utils::file::FileUtils::get_executable_dir() +
"/resources/ca_cert.crt"));
+minifi::utils::file::concat_path(executable_dir,
"resources/ca_cert.crt")));
Review Comment:
We need `concat_path` between `resources` and `ca_cert.crt`, too. Maybe
it's time for a 3-parameter (or variadic) `concat_path`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a diff in pull request #1370: MINIFICPP-1866 Secure connection for ListenSyslog, ListenTCP
lordgamez commented on code in PR #1370:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1370#discussion_r948129229
##
libminifi/include/utils/net/SessionHandlingServer.h:
##
@@ -0,0 +1,67 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#pragma once
+
+#include
+#include
+
+#include "Server.h"
+#include "asio/ssl.hpp"
+
+namespace org::apache::nifi::minifi::utils::net {
+
+template
+class SessionHandlingServer : public Server {
+ public:
+ SessionHandlingServer(std::optional max_queue_size, uint16_t port,
std::shared_ptr logger)
+ : Server(max_queue_size, std::move(logger)),
+acceptor_(io_context_, asio::ip::tcp::endpoint(asio::ip::tcp::v4(),
port)) {
+ }
+
+ void run() override {
+startAccept();
+Server::run();
+ }
+
+ protected:
+ void startAccept() {
+auto new_session = createSession();
+acceptor_.async_accept(new_session->getSocket(),
+ [this, new_session](const auto& error_code) -> void
{
+ handleAccept(new_session, error_code);
+ });
Review Comment:
Sounds logical, but for some reason changing it to unique_ptr causes a
segmentation fault in the asio library. I wasn't able to figure out what caused
it so far.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a diff in pull request #1370: MINIFICPP-1866 Secure connection for ListenSyslog, ListenTCP
lordgamez commented on code in PR #1370:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1370#discussion_r948126915
##
libminifi/test/Utils.h:
##
@@ -135,3 +142,37 @@ struct FlowFileQueueTestAccessor {
FIELD_ACCESSOR(load_task_);
FIELD_ACCESSOR(queue_);
};
+
+bool sendMessagesViaSSL(const std::vector& contents,
uint64_t port, const std::string& ca_cert_path, const
std::optional& ssl_data = std::nullopt) {
+ asio::ssl::context ctx(asio::ssl::context::sslv23);
+ ctx.load_verify_file(ca_cert_path);
+ if (ssl_data) {
+ctx.set_verify_mode(asio::ssl::verify_peer);
+ctx.use_certificate_file(ssl_data->cert_loc, asio::ssl::context::pem);
+ctx.use_private_key_file(ssl_data->key_loc, asio::ssl::context::pem);
Review Comment:
Added in f3fb107052af05c62635ba3d4a9c1906c714d6fe
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NIFI-10366) Make Default Run Duration configurable
Lehel Boér created NIFI-10366: - Summary: Make Default Run Duration configurable Key: NIFI-10366 URL: https://issues.apache.org/jira/browse/NIFI-10366 Project: Apache NiFi Issue Type: Improvement Reporter: Lehel Boér Assignee: Lehel Boér We should provide a duration parameter is @SupportsBatching to set default run duration. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi-minifi-cpp] lordgamez commented on a diff in pull request #1370: MINIFICPP-1866 Secure connection for ListenSyslog, ListenTCP
lordgamez commented on code in PR #1370:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1370#discussion_r948126571
##
extensions/standard-processors/tests/unit/ListenSyslogTests.cpp:
##
@@ -434,4 +434,35 @@ TEST_CASE("ListenSyslog max queue and max batch size
test", "[ListenSyslog]") {
CHECK(controller.trigger().at(ListenSyslog::Success).empty());
}
-} // namespace org::apache::nifi::minifi::processors::testing
+TEST_CASE("Test ListenSyslog via TCP with SSL connection", "[ListenSyslog]") {
+ const auto listen_syslog = std::make_shared("ListenSyslog");
+
+ SingleProcessorTestController controller{listen_syslog};
+ auto ssl_context_service =
controller.plan->addController("SSLContextService", "SSLContextService");
+ REQUIRE(controller.plan->setProperty(ssl_context_service,
controllers::SSLContextService::CACertificate.getName(),
+minifi::utils::file::FileUtils::get_executable_dir() +
"/resources/ca_cert.crt"));
Review Comment:
Good point, updated in f3fb107052af05c62635ba3d4a9c1906c714d6fe
##
libminifi/src/utils/net/SslServer.cpp:
##
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "utils/net/SslServer.h"
+
+namespace org::apache::nifi::minifi::utils::net {
+
+SslSession::SslSession(asio::io_context& io_context, asio::ssl::context&
context, utils::ConcurrentQueue& concurrent_queue,
+std::optional max_queue_size,
std::shared_ptr logger)
+ : concurrent_queue_(concurrent_queue),
+max_queue_size_(max_queue_size),
Review Comment:
Updated in f3fb107052af05c62635ba3d4a9c1906c714d6fe
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a diff in pull request #1370: MINIFICPP-1866 Secure connection for ListenSyslog, ListenTCP
lordgamez commented on code in PR #1370: URL: https://github.com/apache/nifi-minifi-cpp/pull/1370#discussion_r948126207 ## PROCESSORS.md: ## @@ -1303,11 +1305,26 @@ Listens for incoming TCP connections and reads data from each connection using a In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. -| Name | Default Value | Allowable Values | Description | -|---|---|--|---| -| **Listening Port**| | | The port to listen on for communication. | -| **Max Batch Size**| 500 | | The maximum number of messages to process at a time. | -| **Max Size of Message Queue** | 1 | | Maximum number of messages allowed to be buffered before processing them when the processor is triggered. If the buffer is full, the message is ignored. If set to zero the buffer is unlimited. | +| Name | Default Value | Allowable Values | Description | +|---|---||--| +| **Listening Port**| || The port to listen on for communication. | +| **Max Batch Size**| 500 || The maximum number of messages to process at a time. | +| **Max Size of Message Queue** | 1 || Maximum number of messages allowed to be buffered before processing them when the processor is triggered. If the buffer is full, the message is ignored. If set to zero the buffer is unlimited. | +| SSL Context Service | || The Controller Service to use in order to obtain an SSL Context. If this property is set, messages will be received over a secure connection. | +| Client Auth | NONE | NONEWANTREQUIRED | The client authentication policy to use for the SSL Context. Only used if an SSL Context Service is provided. | + +### Relationships + +| Name| Description | +|-|| +| success | Messages received successfully will be sent out this relationship. | + +### Output Attributes + +| Attribute| Description | Requirements | +|--||| +| _tcp.port_ | The sending host of the messages. | - | +| _tcp.sender_ | The sending port the messages were received. | - | Review Comment: Good catch, updated in f3fb107052af05c62635ba3d4a9c1906c714d6fe ## PROCESSORS.md: ## @@ -1254,13 +1254,15 @@ With parsing disabled all message will be routed to the success relationship, bu In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property sup
[jira] [Updated] (NIFI-10365) InvokeHTTP: support dynamic specification of "Request Header Attributes Pattern"
[ https://issues.apache.org/jira/browse/NIFI-10365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Grey updated NIFI-10365: - Description: The current implementation of processor `InvokeHTTP` supports specification of flowfile attributes to send as HTTP headers via processor property `Request Header Attributes Pattern`. This property is a regex, and is immutable while the processor is running. In some situations, it would be helpful to dynamically specify attributes to send as HTTP headers. So some InvokeHTTP invocations would send HTTP headers A and B, while others would send headers B and C. One possible implementation would involve the interpretation of an incoming flowfile attribute value to serve the role currently filled by `Request Header Attributes Pattern`. was: The current implementation of processor `InvokeHTTP` supports specification of flowfile attributes to send as HTTP headers via processor property `Request Header Attributes Pattern`. This property is a regex, and is immutable while the processor is running. In some situations, it would be helpful to dynamically specify attributes to send as HTTP headers. So some InvokeHTTP invocations would send HTTP headers A and B, while others would send headers B and C. One possible implementation would involve the interpretation of an incoming flowfile attribute value to serve the role currently filled by `Request Header Attributes Pattern`. > InvokeHTTP: support dynamic specification of "Request Header Attributes > Pattern" > > > Key: NIFI-10365 > URL: https://issues.apache.org/jira/browse/NIFI-10365 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Paul Grey >Priority: Minor > > The current implementation of processor `InvokeHTTP` supports specification > of > flowfile attributes to send as HTTP headers via processor property `Request > Header Attributes Pattern`. This property is a regex, and is immutable while > the processor is running. > In some situations, it would be helpful to dynamically specify attributes to > send as HTTP headers. So some InvokeHTTP invocations would send HTTP headers > A and B, while others would send headers B and C. > One possible implementation would involve the interpretation of an incoming > flowfile attribute value to serve the role currently filled by `Request > Header Attributes Pattern`. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-10365) InvokeHTTP: support dynamic specification of "Request Header Attributes Pattern"
[
https://issues.apache.org/jira/browse/NIFI-10365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17580839#comment-17580839
]
Paul Grey commented on NIFI-10365:
--
A possible variant:
processor configuration:
{noformat}
Request Header Attributes Pattern = request.header.(.*)
{noformat}
incoming flowfile attributes:
{noformat}
request.header.Authorization: Bearer 123123
request.header.Content-Type: application/json
request.header.Accept-Encoding: gzip, deflate, br
{noformat}
outgoing HTTP headers:
{noformat}
Authorization: Bearer 123123
Content-Type: application/json
Accept-Encoding: gzip, deflate, br
{noformat}
> InvokeHTTP: support dynamic specification of "Request Header Attributes
> Pattern"
>
>
> Key: NIFI-10365
> URL: https://issues.apache.org/jira/browse/NIFI-10365
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Paul Grey
>Priority: Minor
>
> The current implementation of processor `InvokeHTTP` supports specification
> of
> flowfile attributes to send as HTTP headers via processor property `Request
> Header Attributes Pattern`. This property is a regex, and is immutable while
> the processor is running.
> In some situations, it would be helpful to dynamically specify attributes to
> send as HTTP headers. So some InvokeHTTP invocations would send HTTP headers
> A and B, while others would send headers B and C.
> One possible implementation would involve the interpretation of an incoming
> flowfile attribute value to serve the role currently filled by `Request
> Header Attributes Pattern`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [nifi-minifi-cpp] fgerlits commented on a diff in pull request #1392: MINIFICPP-1900 Make the (Python part of the) script extension compile on Windows
fgerlits commented on code in PR #1392: URL: https://github.com/apache/nifi-minifi-cpp/pull/1392#discussion_r948082000 ## libminifi/include/agent/agent_version.h: ## @@ -18,6 +18,8 @@ #ifndef LIBMINIFI_INCLUDE_AGENT_AGENT_VERSION_H_ #define LIBMINIFI_INCLUDE_AGENT_AGENT_VERSION_H_ +#undef COMPILER + Review Comment: TBH, I don't know why it's needed. Something somewhere `#define`s `COMPILER` to the MSVC version, but I couldn't find what and where. I'll take another look and add the comment if I find out, but I didn't think `// something somewhere #defines COMPILE to the MSVC version on Windows` would be a useful comment. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-10365) InvokeHTTP: support dynamic specification of "Request Header Attributes Pattern"
[ https://issues.apache.org/jira/browse/NIFI-10365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17580836#comment-17580836 ] Paul Grey commented on NIFI-10365: -- https://apachenifi.slack.com/archives/C0L9VCD47/p1660229343884179 > InvokeHTTP: support dynamic specification of "Request Header Attributes > Pattern" > > > Key: NIFI-10365 > URL: https://issues.apache.org/jira/browse/NIFI-10365 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Paul Grey >Priority: Minor > > The current implementation of processor `InvokeHTTP` supports specification > of > flowfile attributes to send as HTTP headers via processor property `Request > Header Attributes Pattern`. This property is a regex, and is immutable while > the processor is running. > In some situations, it would be helpful to dynamically specify attributes to > send as HTTP headers. So some InvokeHTTP invocations would send HTTP headers > A and B, while others would send headers B and C. > One possible implementation would involve the interpretation of an incoming > flowfile attribute value to serve the role currently filled by `Request > Header Attributes Pattern`. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-10365) InvokeHTTP: support dynamic specification of "Request Header Attributes Pattern"
Paul Grey created NIFI-10365: Summary: InvokeHTTP: support dynamic specification of "Request Header Attributes Pattern" Key: NIFI-10365 URL: https://issues.apache.org/jira/browse/NIFI-10365 Project: Apache NiFi Issue Type: Improvement Reporter: Paul Grey The current implementation of processor `InvokeHTTP` supports specification of flowfile attributes to send as HTTP headers via processor property `Request Header Attributes Pattern`. This property is a regex, and is immutable while the processor is running. In some situations, it would be helpful to dynamically specify attributes to send as HTTP headers. So some InvokeHTTP invocations would send HTTP headers A and B, while others would send headers B and C. One possible implementation would involve the interpretation of an incoming flowfile attribute value to serve the role currently filled by `Request Header Attributes Pattern`. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi-minifi-cpp] martinzink opened a new pull request, #1393: MINIFICPP-1882 centos build default CA doesnt work on ubuntu
martinzink opened a new pull request, #1393: URL: https://github.com/apache/nifi-minifi-cpp/pull/1393 Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically main)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on pull request #1390: MINIFICPP-1908 - Set manifest description for script processors
adamdebreceni commented on PR #1390: URL: https://github.com/apache/nifi-minifi-cpp/pull/1390#issuecomment-1218126081 > Do we have any tests for this? If not, it would be good to add some, or to check the descriptions in existing manifest tests. If possible, Python processors should be checked, too. added test to check the python proc manifests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-9993) Processors cannot write to Content Repo: Unable to write flowfile content to content repository container default due to archive file size constraints; waiting for archi
[
https://issues.apache.org/jira/browse/NIFI-9993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17580827#comment-17580827
]
Frank De Ruiter commented on NIFI-9993:
---
Is there any workaround for this issue?
> Processors cannot write to Content Repo: Unable to write flowfile content to
> content repository container default due to archive file size constraints;
> waiting for archive cleanup
> ---
>
> Key: NIFI-9993
> URL: https://issues.apache.org/jira/browse/NIFI-9993
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.17.0, 1.16.2
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Sometimes, we can see NiFi log the following log message over and over:
> {code:java}
> 2022-05-05 10:47:03,267 WARN [Timer-Driven Process Thread-142]
> o.a.n.c.repository.FileSystemRepository Unable to write to container default
> due to archive file size constraints; waiting for archive cleanup {code}
> However, we can also see that there's nothing in the archive to cleanup,
> based on the fact that we also continually see this log message:
> {code:java}
> 2022-05-05 10:01:02,515 INFO [Cleanup Archive for default]
> o.a.n.c.repository.FileSystemRepository Successfully deleted 0 files (0
> bytes) from archive {code}
> If there are any files in the archive directory, we should see them getting
> cleaned up. The fact that we constantly see "Successfully deleted 0 files (0
> bytes) from archive" implies there is nothing in the archive.
> But if there is nothing in the archive, then we should not be seeing any
> notices about "Unable to write to container default due to archive file size
> constraints."
> This results in processors getting blocked for indefinite periods of time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-10276) Add database provider option for Registry Flow Provider in Docker
[ https://issues.apache.org/jira/browse/NIFI-10276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-10276: Issue Type: Improvement (was: New Feature) > Add database provider option for Registry Flow Provider in Docker > - > > Key: NIFI-10276 > URL: https://issues.apache.org/jira/browse/NIFI-10276 > Project: Apache NiFi > Issue Type: Improvement > Components: NiFi Registry >Affects Versions: 1.17.0 >Reporter: Julien G. >Assignee: Julien G. >Priority: Major > Fix For: 1.18.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > When using the NiFi registry docker image, scripts are used to transcribe > some environment variables to the actual configuration. > One of these variables is NIFI_REGISTRY_FLOW_PROVIDER which is used to > specify the provider used. > But we cannot use it to specify the database provider. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-10276) Add database provider option for Registry Flow Provider in Docker
[ https://issues.apache.org/jira/browse/NIFI-10276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-10276. - Fix Version/s: 1.18.0 Assignee: Julien G. Resolution: Fixed > Add database provider option for Registry Flow Provider in Docker > - > > Key: NIFI-10276 > URL: https://issues.apache.org/jira/browse/NIFI-10276 > Project: Apache NiFi > Issue Type: New Feature > Components: NiFi Registry >Affects Versions: 1.17.0 >Reporter: Julien G. >Assignee: Julien G. >Priority: Major > Fix For: 1.18.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > When using the NiFi registry docker image, scripts are used to transcribe > some environment variables to the actual configuration. > One of these variables is NIFI_REGISTRY_FLOW_PROVIDER which is used to > specify the provider used. > But we cannot use it to specify the database provider. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10276) Add database provider option for Registry Flow Provider in Docker
[ https://issues.apache.org/jira/browse/NIFI-10276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-10276: Priority: Minor (was: Major) > Add database provider option for Registry Flow Provider in Docker > - > > Key: NIFI-10276 > URL: https://issues.apache.org/jira/browse/NIFI-10276 > Project: Apache NiFi > Issue Type: Improvement > Components: NiFi Registry >Affects Versions: 1.17.0 >Reporter: Julien G. >Assignee: Julien G. >Priority: Minor > Fix For: 1.18.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > When using the NiFi registry docker image, scripts are used to transcribe > some environment variables to the actual configuration. > One of these variables is NIFI_REGISTRY_FLOW_PROVIDER which is used to > specify the provider used. > But we cannot use it to specify the database provider. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi-minifi-cpp] szaszm commented on a diff in pull request #1392: MINIFICPP-1900 Make the (Python part of the) script extension compile on Windows
szaszm commented on code in PR #1392:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1392#discussion_r948039984
##
extensions/script/python/PythonCreator.h:
##
@@ -129,7 +129,8 @@ class PythonCreator : public minifi::core::CoreComponent {
return "";
}
const auto python_package_path = std::filesystem::relative(pythonscript,
basePath).parent_path();
-std::vector path_elements{python_package_path.begin(),
python_package_path.end()};
+std::vector path_elements;
+std::transform(python_package_path.begin(), python_package_path.end(),
std::back_inserter(path_elements), [](const auto& path) { return path.string();
});
Review Comment:
```suggestion
std::vector path_elements;
path_elements.reserve(std::distance(python_package_path.begin(),
python_package_path.end()));
std::transform(python_package_path.begin(), python_package_path.end(),
std::back_inserter(path_elements), [](const auto& path) { return path.string();
});
```
##
libminifi/include/agent/agent_version.h:
##
@@ -18,6 +18,8 @@
#ifndef LIBMINIFI_INCLUDE_AGENT_AGENT_VERSION_H_
#define LIBMINIFI_INCLUDE_AGENT_AGENT_VERSION_H_
+#undef COMPILER
+
Review Comment:
Why is this needed? Could you add a comment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-10276) Add database provider option for Registry Flow Provider in Docker
[ https://issues.apache.org/jira/browse/NIFI-10276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-10276: Summary: Add database provider option for Registry Flow Provider in Docker (was: Add database provider management) > Add database provider option for Registry Flow Provider in Docker > - > > Key: NIFI-10276 > URL: https://issues.apache.org/jira/browse/NIFI-10276 > Project: Apache NiFi > Issue Type: New Feature > Components: NiFi Registry >Affects Versions: 1.17.0 >Reporter: Julien G. >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > When using the NiFi registry docker image, scripts are used to transcribe > some environment variables to the actual configuration. > One of these variables is NIFI_REGISTRY_FLOW_PROVIDER which is used to > specify the provider used. > But we cannot use it to specify the database provider. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-10276) Add database provider management
[ https://issues.apache.org/jira/browse/NIFI-10276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17580825#comment-17580825 ] ASF subversion and git services commented on NIFI-10276: Commit 0afdd32bb578d21c42599e621d5c6e474c223205 in nifi's branch refs/heads/main from Julien Guitton [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=0afdd32bb5 ] NIFI-10276 Support database option for Registry Flow Provider in Docker This closes #6244 Signed-off-by: David Handermann > Add database provider management > > > Key: NIFI-10276 > URL: https://issues.apache.org/jira/browse/NIFI-10276 > Project: Apache NiFi > Issue Type: New Feature > Components: NiFi Registry >Affects Versions: 1.17.0 >Reporter: Julien G. >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > When using the NiFi registry docker image, scripts are used to transcribe > some environment variables to the actual configuration. > One of these variables is NIFI_REGISTRY_FLOW_PROVIDER which is used to > specify the provider used. > But we cannot use it to specify the database provider. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] exceptionfactory closed pull request #6244: NIFI-10276 Add database provider management with NIFI_REGISTRY_FLOW_PROV…
exceptionfactory closed pull request #6244: NIFI-10276 Add database provider management with NIFI_REGISTRY_FLOW_PROV… URL: https://github.com/apache/nifi/pull/6244 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a diff in pull request #1370: MINIFICPP-1866 Secure connection for ListenSyslog, ListenTCP
lordgamez commented on code in PR #1370:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1370#discussion_r948040417
##
libminifi/src/utils/net/SslServer.cpp:
##
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "utils/net/SslServer.h"
+
+namespace org::apache::nifi::minifi::utils::net {
+
+SslSession::SslSession(asio::io_context& io_context, asio::ssl::context&
context, utils::ConcurrentQueue& concurrent_queue,
+std::optional max_queue_size,
std::shared_ptr logger)
+ : concurrent_queue_(concurrent_queue),
+max_queue_size_(max_queue_size),
+logger_(std::move(logger)),
+socket_(io_context, context) {
+}
+
+ssl_socket::lowest_layer_type& SslSession::getSocket() {
+ return socket_.lowest_layer();
+}
+
+void SslSession::start() {
+ socket_.async_handshake(asio::ssl::stream_base::server,
+[this, self = shared_from_this()](const std::error_code& error_code) {
+ if (error_code) {
+logger_->log_error("Error occured during SSL handshake: (%d) %s",
error_code.value(), error_code.message());
+return;
+ }
+ asio::async_read_until(socket_,
+ buffer_,
+ '\n',
+ [self](const auto& error_code, size_t) -> void {
+ self->handleReadUntilNewLine(error_code);
Review Comment:
I'm not sure as this was an older code, but I think shared_from_this is used
due to lifetime considerations, that the session might still be alive when
server is being destroyed, so we need to keep it alive until the session is
finished, but this is just a guess.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits opened a new pull request, #1392: MINIFICPP-1900 Make the (Python part of the) script extension compile on Windows
fgerlits opened a new pull request, #1392: URL: https://github.com/apache/nifi-minifi-cpp/pull/1392 Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically main)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a diff in pull request #1370: MINIFICPP-1866 Secure connection for ListenSyslog, ListenTCP
szaszm commented on code in PR #1370:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1370#discussion_r948021879
##
extensions/civetweb/tests/ListenHTTPTests.cpp:
##
@@ -60,15 +60,15 @@ class ListenHTTPTestsFixture {
LogTestController::getInstance().setTrace();
LogTestController::getInstance().setTrace();
LogTestController::getInstance().setDebug();
-LogTestController::getInstance().setDebug();
Review Comment:
You're right, I was wrong about the name lookup rules, sorry.
##
libminifi/include/utils/net/SessionHandlingServer.h:
##
@@ -0,0 +1,67 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#pragma once
+
+#include
+#include
+
+#include "Server.h"
+#include "asio/ssl.hpp"
+
+namespace org::apache::nifi::minifi::utils::net {
+
+template
+class SessionHandlingServer : public Server {
+ public:
+ SessionHandlingServer(std::optional max_queue_size, uint16_t port,
std::shared_ptr logger)
+ : Server(max_queue_size, std::move(logger)),
+acceptor_(io_context_, asio::ip::tcp::endpoint(asio::ip::tcp::v4(),
port)) {
+ }
+
+ void run() override {
+startAccept();
+Server::run();
+ }
+
+ protected:
+ void startAccept() {
+auto new_session = createSession();
+acceptor_.async_accept(new_session->getSocket(),
+ [this, new_session](const auto& error_code) -> void
{
+ handleAccept(new_session, error_code);
+ });
Review Comment:
Shouldn't the session be moved into the lambda? After querying the socket,
it's no longer used outside. It could be moved further into handleAccept later,
on the accepting thread. This would allow it to not be a shared_ptr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a diff in pull request #1370: MINIFICPP-1866 Secure connection for ListenSyslog, ListenTCP
fgerlits commented on code in PR #1370:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1370#discussion_r947938125
##
libminifi/src/utils/net/SslServer.cpp:
##
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "utils/net/SslServer.h"
+
+namespace org::apache::nifi::minifi::utils::net {
+
+SslSession::SslSession(asio::io_context& io_context, asio::ssl::context&
context, utils::ConcurrentQueue& concurrent_queue,
+std::optional max_queue_size,
std::shared_ptr logger)
+ : concurrent_queue_(concurrent_queue),
+max_queue_size_(max_queue_size),
Review Comment:
minor, but `max_queue_size` could be moved instead of copied
##
PROCESSORS.md:
##
@@ -1254,13 +1254,15 @@ With parsing disabled all message will be routed to the
success relationship, bu
In the list below, the names of required properties appear in bold. Any other
properties (not in bold) are considered optional. The table also indicates any
default values, and whether a property supports the NiFi Expression Language.
-| Name | Default Value | Allowable Values | Description
|
-|---|---|--|--|
-| Listening Port| 514 | | The port for
Syslog communication. (Well-known ports (0-1023) require root access)
|
-| Protocol | UDP | UDPTCP | The protocol
for Syslog communication.
|
-| Parse Messages| false | falsetrue| Indicates if
the processor should parse the Syslog messages. If set to false, each outgoing
FlowFile will only contain the sender, protocol, and port, and no additional
attributes. |
-| Max Batch Size| 500 | | The maximum
number of Syslog events to process at a time.
|
-| Max Size of Message Queue | 1 | | Maximum
number of Syslog messages allowed to be buffered before processing them when
the processor is triggered. If the buffer is full, the message is ignored. If
set to zero the buffer is unlimited. |
+| Name | Default Value | Allowable Values |
Description
|
+|---|---||-|
+| Listening Port| 514 || The
port for Syslog communication. (Well-known ports (0-1023) require root access)
|
+| Protocol | UDP | UDPTCP | The
protocol for Syslog communication.
|
+| Parse Messages| false | falsetrue |
Indicates if the processor should parse the Syslog messages. If set to false,
each outgoing FlowFile wil
[jira] [Updated] (NIFI-10361) Upgrade commons-jexl3 to version 3.2.1
[ https://issues.apache.org/jira/browse/NIFI-10361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-10361: Priority: Minor (was: Major) > Upgrade commons-jexl3 to version 3.2.1 > -- > > Key: NIFI-10361 > URL: https://issues.apache.org/jira/browse/NIFI-10361 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike R >Priority: Minor > Time Spent: 0.5h > Remaining Estimate: 0h > > Upgrade commons-jexl3 to version 3.2.1 from 3.1 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-10361) Upgrade commons-jexl3 to version 3.2.1
[ https://issues.apache.org/jira/browse/NIFI-10361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17580813#comment-17580813 ] ASF subversion and git services commented on NIFI-10361: Commit 99a0c050704ddd71c2bce0ead6a15046f7d0db12 in nifi's branch refs/heads/main from mr1716 [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=99a0c05070 ] NIFI-10361 Upgraded commons-jexl3 from 3.1 to 3.2.1 This closes #6305 Signed-off-by: David Handermann > Upgrade commons-jexl3 to version 3.2.1 > -- > > Key: NIFI-10361 > URL: https://issues.apache.org/jira/browse/NIFI-10361 > Project: Apache NiFi > Issue Type: Bug >Reporter: Mike R >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Upgrade commons-jexl3 to version 3.2.1 from 3.1 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] exceptionfactory closed pull request #6305: NIFI-10361: Upgrade commons-jexl3 to version 3.2.1
exceptionfactory closed pull request #6305: NIFI-10361: Upgrade commons-jexl3 to version 3.2.1 URL: https://github.com/apache/nifi/pull/6305 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-10361) Upgrade commons-jexl3 to version 3.2.1
[ https://issues.apache.org/jira/browse/NIFI-10361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-10361: Issue Type: Improvement (was: Bug) > Upgrade commons-jexl3 to version 3.2.1 > -- > > Key: NIFI-10361 > URL: https://issues.apache.org/jira/browse/NIFI-10361 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike R >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Upgrade commons-jexl3 to version 3.2.1 from 3.1 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-10361) Upgrade commons-jexl3 to version 3.2.1
[ https://issues.apache.org/jira/browse/NIFI-10361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-10361. - Fix Version/s: 1.18.0 Assignee: Mike R Resolution: Fixed > Upgrade commons-jexl3 to version 3.2.1 > -- > > Key: NIFI-10361 > URL: https://issues.apache.org/jira/browse/NIFI-10361 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Mike R >Assignee: Mike R >Priority: Minor > Fix For: 1.18.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Upgrade commons-jexl3 to version 3.2.1 from 3.1 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10361) Upgrade commons-jexl3 to version 3.2.1
[ https://issues.apache.org/jira/browse/NIFI-10361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-10361: Component/s: Extensions > Upgrade commons-jexl3 to version 3.2.1 > -- > > Key: NIFI-10361 > URL: https://issues.apache.org/jira/browse/NIFI-10361 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Mike R >Priority: Minor > Time Spent: 0.5h > Remaining Estimate: 0h > > Upgrade commons-jexl3 to version 3.2.1 from 3.1 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-9015) Ability to derive custom Rest API processes from InvokeHTTP
[
https://issues.apache.org/jira/browse/NIFI-9015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17580809#comment-17580809
]
David Handermann commented on NIFI-9015:
NIFI-10244 introduced a new {{WebClientServiceProvider}} Controller Service and
{{WebClientService}} interface for HTTP communication. It seems like building
on these components would provide a better fit for extensible components as
opposed to refactoring InvokeHTTP for extension. What do you think [~otto]?
> Ability to derive custom Rest API processes from InvokeHTTP
> ---
>
> Key: NIFI-9015
> URL: https://issues.apache.org/jira/browse/NIFI-9015
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Otto Fowler
>Priority: Major
>
> In setups with custom development, there is a lot of boilerplate
> customization to InvokeHTTP around some known set of rest apis. As such flow
> developers / users have to 'know' and understand these things in order to
> setup possibly multiple InvokeHTTP instances with these details.
> Some users may instead want to create a top level derived processor with
> custom setup and parameters for a specific rest api for easy configuration
> and deployment.
> The
> [InvokeAWSGatewayApi|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-aws-nar/1.14.0/org.apache.nifi.processors.aws.wag.InvokeAWSGatewayApi/index.html]
> processor ( which itself had to be copied from InvokeHTTP because of this
> limitation ) offers this, such that you can create a derived processor for a
> custom AWS Web Gateway service.
> With a coming processor registry, the ability for people to contribute
> specialized processors for certain APIs would be great. These may be a
> subset or a super set of the existing properties. We may also move the
> 'base' properties to a new UI tab in the configuration, such that you can add
> custom extra configuration in the first focus tab. We could also allow for
> removing properties as well from the base tab, etc etc
> This task would involve refactoring the invokeHTTP processor such that there
> is a reusable base, with the InvokeHTTP processor being a pass-through
> default implementation ( IE> a new derived from base would have all the same
> features as InvokeHTTP ).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-5232) HttpConnectionService controller service
[
https://issues.apache.org/jira/browse/NIFI-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17580804#comment-17580804
]
David Handermann commented on NIFI-5232:
NIFI-10244 implemented something similar to the capabilities described in this
issue in a {{WebClientServiceProvider}} Controller Service and
{{WebClientService}} abstraction.
[~mike.thomsen] Perhaps this issue could be closed in light of that
implementation?
> HttpConnectionService controller service
>
>
> Key: NIFI-5232
> URL: https://issues.apache.org/jira/browse/NIFI-5232
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Mike Thomsen
>Priority: Major
>
> The functionality of InvokeHttp and related processors should be copied over
> to a controller service that can do much the same thing. This controller
> service would be able to handle all of the common scenarios with HTTP
> connections from processors going forward.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [nifi] taz1988 opened a new pull request, #6309: NIFI-10363 Extend lib/spp dependencySet to prevent NoClassDefFoundError
taz1988 opened a new pull request, #6309: URL: https://github.com/apache/nifi/pull/6309 # Summary [NIFI-10363](https://issues.apache.org/jira/browse/NIFI-10363) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI-10363) issue created ### Pull Request Tracking - [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [x] Pull Request based on current revision of the `main` branch - [x] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. Build with `mvn clean install -P contrib-check`. After build I tried to reproduce the error mentioned on the jira ticket. The error not appeared, nifi-registry started properly. ### Build - [x] Build completed using `mvn clean install -P contrib-check` - [x] JDK 8 - [ ] JDK 11 - [ ] JDK 17 ### Licensing - N/A New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - N/A New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - N/A Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-10362) Cluster can disconnect node as soon as it rejoins cluster upon restart
[
https://issues.apache.org/jira/browse/NIFI-10362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-10362:
--
Fix Version/s: 1.18.0
Status: Patch Available (was: Open)
> Cluster can disconnect node as soon as it rejoins cluster upon restart
> --
>
> Key: NIFI-10362
> URL: https://issues.apache.org/jira/browse/NIFI-10362
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.18.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When the Cluster Coordinator disconnects a node due to a user requesting that
> the node get disconnected, the node is immediately marked as DISCONNECTED,
> and then a background thread is responsible for notifying the node that it's
> been disconnected. The background task attempts several times if it cannot
> successfully send the notification.
> However, if the node is disconnected and then restarted before it's been
> notified, we have a situation in which the node becomes CONNECTING (and
> possibly then CONNECTED), and then the background task is triggered. This
> then results in the node being told that it's DISCONNECTED. But the Cluster
> Coordinator doesn't think so (because its already changed the state back to
> CONNECTING/CONNECTED).
> While the chances that this happens are slim in production and it's easily
> worked around (by simply waiting a few seconds after disconnecting a node
> before restarting it, or just restarting without disconnecting) it causes a
> lot of problems for system tests and potentially other automated activities.
> It results in the following log message in the Cluster Coordinator:
> {code:java}
> 2022-08-15 00:47:50,200 ERROR [Disconnect localhost:5672]
> org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator Failed to
> notify localhost:5672 that it has been disconnected from the cluster due to
> User anonymous requested that node be disconnected from cluster {code}
> And then we see confusing error messages such as:
> {code:java}
> 2022-08-15 00:48:01,461 INFO [Replicate Request Thread-23]
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator
> Received a status of 200 from localhost:5672 for request PUT
> /nifi-api/flow/process-groups/root when performing first stage of two-stage
> commit. The action will not occur. Node explanation:
> {"id":"root","state":"STOPPED"} {code}
> This is because when the cluster coordinator replicates the request to all
> nodes, the node that thinks it is disconnected receives the request and
> performs the action. It then responds with a "200 OK" but it should have
> noted that it's the first phase of a 2-phase action and responded with "201
> Continue".
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [nifi] markap14 opened a new pull request, #6308: NIFI-10362: When asynchronous node disconnect is issued, do not send …
markap14 opened a new pull request, #6308: URL: https://github.com/apache/nifi/pull/6308 …disconnect to node if the node becomes reconnected in the interim. Also, addressed the issue in which a disconnected node acts on a replicated request during the first phase by detect that it's the first phase if configured for cluster, not when only when connected to a cluster. # Summary [NIFI-0](https://issues.apache.org/jira/browse/NIFI-0) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [ ] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 8 - [ ] JDK 11 - [ ] JDK 17 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a diff in pull request #1377: MINIFICPP-1865 Integrate performance clang-tidy checks in CI
lordgamez commented on code in PR #1377:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1377#discussion_r947942230
##
libminifi/src/c2/C2Agent.cpp:
##
@@ -739,7 +739,7 @@ utils::TaskRescheduleInfo C2Agent::produce() {
std::make_move_iterator(payload_batch.end()),
[&] (C2Payload&& payload) {
try {
-C2Payload && response =
protocol_.load()->consumePayload(std::move(payload));
+C2Payload && response = protocol_.load()->consumePayload(payload);
Review Comment:
Updated in 7d86623adbe9d1cf018a62e72536bce62cd4332d
##
libminifi/test/unit/ExpectedTest.cpp:
##
@@ -52,7 +52,7 @@ TEST_CASE("expected map", "[expected][map]") {
{
const nonstd::expected e = 21;
-auto ret = std::move(e) | utils::map(mul2);
+auto ret = e | utils::map(mul2);
Review Comment:
Good catch, updated in 7d86623adbe9d1cf018a62e72536bce62cd4332d
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a diff in pull request #1377: MINIFICPP-1865 Integrate performance clang-tidy checks in CI
lordgamez commented on code in PR #1377:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1377#discussion_r947942605
##
libminifi/test/unit/IdTests.cpp:
##
@@ -77,7 +77,7 @@ TEST_CASE("Test Generate Move", "[id]") {
auto generated = generator->generate();
auto str = generated.to_string();
- utils::Identifier moved = std::move(generated);
+ utils::Identifier moved = generated;
Review Comment:
Updated in 7d86623adbe9d1cf018a62e72536bce62cd4332d
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[
https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Giovanna Cazelato Pires updated NIFI-9572:
--
Attachment: (was: lsof (1).txt)
> Failed to index Provenance Events and (Too many Files)
> --
>
> Key: NIFI-9572
> URL: https://issues.apache.org/jira/browse/NIFI-9572
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
>Affects Versions: 1.15.2
>Reporter: mayki
>Priority: Major
> Attachments: bootstrap.conf, lsof (1).txt, nifi-app.log,
> nifi-app.log.tar.gz, nifi.properties, nifi_691106_pid.tar.gz
>
>
> Hello
> I have upgraded NIFI 1.15.2 since 2022/01/05
> No issue until this night 2022/01/13
> * nifi version 1.15.2
> * jdk-1.8.0_311
> And the limit is high
> {code:java}
> Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2
> -bash-4.2@nifi$ ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) rg
> pending signals (-i) 63278
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 5
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 1
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
> {code}
>
> We got a lot error about provenance_repository, it fill our filesystem logs ..
>
> {code:java}
> 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2]
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877)
> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891)
> at
> org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468)
> at
> org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444)
> at
> org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70)
> at
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202)
> at
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.nio.file.FileSystemException:
> /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd:
> Too many open files
> {code}
>
>
> We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability.
> But it is impossible to do that if we got this error.
>
> Thanks for you help.
>
> Regards
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[
https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17580785#comment-17580785
]
Giovanna Cazelato Pires commented on NIFI-9572:
---
Hi everyone! I'm experience this same issue for a while now.
1. The specific version of apache nifi. - 1.15.3
2. The specific os/kernel version in use (uname -a) --
4.14.268-139.500.amzn1.x86_64
3. The specific java runtime actually being used by nifi - 11.0.16.8.1
4. A dump from 'lsof -p ' --- attachment (
lsof(1).txt )
5. What is your ulimit for files? (ulimit -a) -- 999k
[^lsof (1).txt]
> Failed to index Provenance Events and (Too many Files)
> --
>
> Key: NIFI-9572
> URL: https://issues.apache.org/jira/browse/NIFI-9572
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
>Affects Versions: 1.15.2
>Reporter: mayki
>Priority: Major
> Attachments: bootstrap.conf, lsof (1).txt, nifi-app.log,
> nifi-app.log.tar.gz, nifi.properties, nifi_691106_pid.tar.gz
>
>
> Hello
> I have upgraded NIFI 1.15.2 since 2022/01/05
> No issue until this night 2022/01/13
> * nifi version 1.15.2
> * jdk-1.8.0_311
> And the limit is high
> {code:java}
> Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2
> -bash-4.2@nifi$ ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) rg
> pending signals (-i) 63278
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 5
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 1
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
> {code}
>
> We got a lot error about provenance_repository, it fill our filesystem logs ..
>
> {code:java}
> 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2]
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877)
> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891)
> at
> org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468)
> at
> org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444)
> at
> org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70)
> at
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202)
> at
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.nio.file.FileSystemException:
> /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd:
> Too many open files
> {code}
>
>
> We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability.
> But it is impossible to do that if we got this error.
>
> Thanks for you help.
>
> Regards
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[
https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Giovanna Cazelato Pires updated NIFI-9572:
--
Attachment: lsof (1).txt
> Failed to index Provenance Events and (Too many Files)
> --
>
> Key: NIFI-9572
> URL: https://issues.apache.org/jira/browse/NIFI-9572
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
>Affects Versions: 1.15.2
>Reporter: mayki
>Priority: Major
> Attachments: bootstrap.conf, lsof (1).txt, nifi-app.log,
> nifi-app.log.tar.gz, nifi.properties, nifi_691106_pid.tar.gz
>
>
> Hello
> I have upgraded NIFI 1.15.2 since 2022/01/05
> No issue until this night 2022/01/13
> * nifi version 1.15.2
> * jdk-1.8.0_311
> And the limit is high
> {code:java}
> Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2
> -bash-4.2@nifi$ ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) rg
> pending signals (-i) 63278
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 5
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 1
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
> {code}
>
> We got a lot error about provenance_repository, it fill our filesystem logs ..
>
> {code:java}
> 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2]
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877)
> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891)
> at
> org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468)
> at
> org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444)
> at
> org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70)
> at
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202)
> at
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.nio.file.FileSystemException:
> /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd:
> Too many open files
> {code}
>
>
> We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability.
> But it is impossible to do that if we got this error.
>
> Thanks for you help.
>
> Regards
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-9572) Failed to index Provenance Events and (Too many Files)
[
https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Giovanna Cazelato Pires updated NIFI-9572:
--
Attachment: lsof (1).txt
> Failed to index Provenance Events and (Too many Files)
> --
>
> Key: NIFI-9572
> URL: https://issues.apache.org/jira/browse/NIFI-9572
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
>Affects Versions: 1.15.2
>Reporter: mayki
>Priority: Major
> Attachments: bootstrap.conf, lsof (1).txt, nifi-app.log,
> nifi-app.log.tar.gz, nifi.properties, nifi_691106_pid.tar.gz
>
>
> Hello
> I have upgraded NIFI 1.15.2 since 2022/01/05
> No issue until this night 2022/01/13
> * nifi version 1.15.2
> * jdk-1.8.0_311
> And the limit is high
> {code:java}
> Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2
> -bash-4.2@nifi$ ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) rg
> pending signals (-i) 63278
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 5
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 1
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
> {code}
>
> We got a lot error about provenance_repository, it fill our filesystem logs ..
>
> {code:java}
> 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2]
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877)
> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891)
> at
> org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468)
> at
> org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444)
> at
> org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70)
> at
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202)
> at
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.nio.file.FileSystemException:
> /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd:
> Too many open files
> {code}
>
>
> We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability.
> But it is impossible to do that if we got this error.
>
> Thanks for you help.
>
> Regards
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [nifi-minifi-cpp] lordgamez commented on a diff in pull request #1370: MINIFICPP-1866 Secure connection for ListenSyslog, ListenTCP
lordgamez commented on code in PR #1370:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1370#discussion_r947909188
##
extensions/civetweb/tests/ListenHTTPTests.cpp:
##
@@ -60,15 +60,15 @@ class ListenHTTPTestsFixture {
LogTestController::getInstance().setTrace();
LogTestController::getInstance().setTrace();
LogTestController::getInstance().setDebug();
-LogTestController::getInstance().setDebug();
Review Comment:
Those changes were actually needed, as we now have a header using
`minifi::test::utils` namespace included and the tests are in the
`minifi::test` namespace. Due to this the compiler expects HTTPClient to be in
the `minifi::test::utils` namespace if we only specify it as
`utils::HTTPClient`. It returns a compiler error: `error: ‘HTTPClient’ is not a
member of ‘org::apache::nifi::minifi::test::utils’; did you mean
‘org::apache::nifi::minifi::utils::HTTPClient’`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
