[nifi] branch main updated (df2f1f3 -> df00cc6)
This is an automated email from the ASF dual-hosted git repository. mattyb149 pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git. from df2f1f3 NIFI-9780 updating docker file versions new 23132fb NIFI-6047 new df00cc6 NIFI-6047 Cleaned up code to allow tests to run against 1.13.0-snapshot Removed DMC. NIFI-6047 Started integrating changes from NIFI-6014. NIFI-6047 Added DMC tests. NIFI-6047 Added cache identifier recordpath test. NIFI-6047 Added additional details. NIFI-6047 Removed old additional details. NIFI-6047 made some changes requested in a follow up review. NIFI-6047 latest. NIFI-6047 Finished updates First round of code review cleanup Latest Removed EL from the dynamic prope [...] The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../processors/standard/DeduplicateRecord.java | 666 + .../services/org.apache.nifi.processor.Processor | 1 + .../additionalDetails.html | 70 +++ .../processors/standard/TestDeduplicateRecord.java | 321 ++ 4 files changed, 1058 insertions(+) create mode 100644 nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DeduplicateRecord.java create mode 100644 nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/resources/docs/org.apache.nifi.processors.standard.DeduplicateRecords/additionalDetails.html create mode 100644 nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestDeduplicateRecord.java
[nifi] 01/02: NIFI-6047
This is an automated email from the ASF dual-hosted git repository. mattyb149 pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git commit 23132fb89f63b8eb1305103934cb5aaed061eefe Author: Adam AuthorDate: Sat Feb 16 19:30:39 2019 -0500 NIFI-6047 Added NiFi DetectDuplicateRecord standard processor. Adding some documentation and PR review tweaks. Exposing processor Documentation updates, exception handling consolidation, added support for record path field variables. Added tests. Build bump. Migrated cache service to groovy folder. Moved declarations for properties to @BeforeClass lifecycle method. Adding some documentation and PR review tweaks. Documentation updates, exception handling consolidation, added support for record path field variables. Added tests. Build bump. Migrated cache service to groovy folder. Fixed variable type bug. Fixed mapping of test params to usage. Fixed potential illegal state exception bug. --- .../processors/standard/DetectDuplicateRecord.java | 648 + .../services/org.apache.nifi.processor.Processor | 1 + .../DetectDuplicateRecord/additionalDetails.html | 96 +++ .../processors/standard/MockCacheService.groovy| 77 +++ .../standard/TestDetectDuplicateRecord.java| 209 +++ 5 files changed, 1031 insertions(+) diff --git a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DetectDuplicateRecord.java b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DetectDuplicateRecord.java new file mode 100644 index 000..191a675 --- /dev/null +++ b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DetectDuplicateRecord.java @@ -0,0 +1,648 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard; + +import com.google.common.base.Joiner; +import com.google.common.hash.BloomFilter; +import com.google.common.hash.Funnels; +import org.apache.commons.codec.binary.Hex; +import org.apache.commons.codec.digest.DigestUtils; +import org.apache.commons.codec.digest.MessageDigestAlgorithms; +import org.apache.nifi.annotation.behavior.*; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.SeeAlso; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.*; +import org.apache.nifi.distributed.cache.client.Deserializer; +import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient; +import org.apache.nifi.distributed.cache.client.Serializer; +import org.apache.nifi.distributed.cache.client.exception.DeserializationException; +import org.apache.nifi.distributed.cache.client.exception.SerializationException; +import org.apache.nifi.expression.AttributeExpression.ResultType; +import org.apache.nifi.expression.ExpressionLanguageScope; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.flowfile.attributes.CoreAttributes; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.processor.*; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.record.path.FieldValue; +import org.apache.nifi.record.path.RecordPath; +import org.apache.nifi.record.path.RecordPathResult; +import org.apache.nifi.record.path.util.RecordPathCache; +import org.apache.nifi.record.path.validation.RecordPathPropertyNameValidator; +import org.apache.nifi.record.path.validation.RecordPathValidator; +import org.apache.nifi.schema.access.SchemaNotFoundException; +import org.apache.nifi.serialization.*; +import org.apache.nifi.serialization.record.Record; +import org.apache.nifi.serialization.record.RecordSchema; +import org.apache.nifi.serialization.record.util.DataTypeUtils; + +import java.io.*; +import java.nio.charset.Charset; +import
[nifi] 02/02: NIFI-6047 Cleaned up code to allow tests to run against 1.13.0-snapshot Removed DMC. NIFI-6047 Started integrating changes from NIFI-6014. NIFI-6047 Added DMC tests. NIFI-6047 Added cach
This is an automated email from the ASF dual-hosted git repository. mattyb149 pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git commit df00cc6cb576c11ae3ef0f1c6f64454598298936 Author: Mike Thomsen AuthorDate: Thu Oct 29 13:52:12 2020 -0400 NIFI-6047 Cleaned up code to allow tests to run against 1.13.0-snapshot Removed DMC. NIFI-6047 Started integrating changes from NIFI-6014. NIFI-6047 Added DMC tests. NIFI-6047 Added cache identifier recordpath test. NIFI-6047 Added additional details. NIFI-6047 Removed old additional details. NIFI-6047 made some changes requested in a follow up review. NIFI-6047 latest. NIFI-6047 Finished updates First round of code review cleanup Latest Removed EL from the dynamic properties. Finished code review requested refactoring. Checkstyle fix. Removed a Java 11 API NIFI-6047 Renamed processor to DeduplicateRecord Signed-off-by: Matthew Burgess This closes #4646 --- ...DuplicateRecord.java => DeduplicateRecord.java} | 562 +++-- .../services/org.apache.nifi.processor.Processor | 2 +- .../additionalDetails.html | 70 +++ .../DetectDuplicateRecord/additionalDetails.html | 96 .../processors/standard/MockCacheService.groovy| 77 --- .../processors/standard/TestDeduplicateRecord.java | 321 .../standard/TestDetectDuplicateRecord.java| 209 7 files changed, 682 insertions(+), 655 deletions(-) diff --git a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DetectDuplicateRecord.java b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DeduplicateRecord.java similarity index 53% rename from nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DetectDuplicateRecord.java rename to nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DeduplicateRecord.java index 191a675..b055a75 100644 --- a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DetectDuplicateRecord.java +++ b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DeduplicateRecord.java @@ -22,94 +22,111 @@ import com.google.common.hash.Funnels; import org.apache.commons.codec.binary.Hex; import org.apache.commons.codec.digest.DigestUtils; import org.apache.commons.codec.digest.MessageDigestAlgorithms; -import org.apache.nifi.annotation.behavior.*; +import org.apache.nifi.annotation.behavior.DynamicProperty; +import org.apache.nifi.annotation.behavior.EventDriven; +import org.apache.nifi.annotation.behavior.InputRequirement; import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.SystemResource; +import org.apache.nifi.annotation.behavior.SystemResourceConsideration; +import org.apache.nifi.annotation.behavior.WritesAttribute; import org.apache.nifi.annotation.documentation.CapabilityDescription; import org.apache.nifi.annotation.documentation.SeeAlso; import org.apache.nifi.annotation.documentation.Tags; import org.apache.nifi.annotation.lifecycle.OnScheduled; -import org.apache.nifi.components.*; -import org.apache.nifi.distributed.cache.client.Deserializer; +import org.apache.nifi.components.AllowableValue; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.Validator; import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient; import org.apache.nifi.distributed.cache.client.Serializer; -import org.apache.nifi.distributed.cache.client.exception.DeserializationException; -import org.apache.nifi.distributed.cache.client.exception.SerializationException; -import org.apache.nifi.expression.AttributeExpression.ResultType; import org.apache.nifi.expression.ExpressionLanguageScope; import org.apache.nifi.flowfile.FlowFile; import org.apache.nifi.flowfile.attributes.CoreAttributes; import org.apache.nifi.logging.ComponentLog; -import org.apache.nifi.processor.*; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessorInitializationContext; +import org.apache.nifi.processor.Relationship; import org.apache.nifi.processor.exception.ProcessException; import org.apache.nifi.processor.util.StandardValidators; import org.apache.nifi.record.path.FieldValue; import org.apache.nifi.record.path.RecordPath; import
[nifi] branch main updated: NIFI-9780 updating docker file versions
This is an automated email from the ASF dual-hosted git repository.
joewitt pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/main by this push:
new df2f1f3 NIFI-9780 updating docker file versions
df2f1f3 is described below
commit df2f1f3b49cf90d2c9a97e41cea67aae40598681
Author: Joe Witt
AuthorDate: Wed Mar 9 13:35:25 2022 -0700
NIFI-9780 updating docker file versions
---
nifi-docker/dockerhub/DockerImage.txt | 2 +-
nifi-docker/dockerhub/Dockerfile | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/nifi-docker/dockerhub/DockerImage.txt
b/nifi-docker/dockerhub/DockerImage.txt
index 44f4ac7..e254633 100644
--- a/nifi-docker/dockerhub/DockerImage.txt
+++ b/nifi-docker/dockerhub/DockerImage.txt
@@ -13,4 +13,4 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-apache/nifi:1.15.0
+apache/nifi:1.16.0
diff --git a/nifi-docker/dockerhub/Dockerfile b/nifi-docker/dockerhub/Dockerfile
index f3de2f4..e967805 100644
--- a/nifi-docker/dockerhub/Dockerfile
+++ b/nifi-docker/dockerhub/Dockerfile
@@ -24,7 +24,7 @@ LABEL site="https://nifi.apache.org;
ARG UID=1000
ARG GID=1000
-ARG NIFI_VERSION=1.15.0
+ARG NIFI_VERSION=1.16.0
ARG BASE_URL=https://archive.apache.org/dist
ARG MIRROR_BASE_URL=${MIRROR_BASE_URL:-${BASE_URL}}
ARG DISTRO_PATH=${DISTRO_PATH:-${NIFI_VERSION}}
[nifi] 02/02: NIFI-9782 This closes #5854. Excluded H2 DB from nifi-druid-bundle
This is an automated email from the ASF dual-hosted git repository. joewitt pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git commit 4a46b087b866205fb85d92853fc82398612d35c9 Author: exceptionfactory AuthorDate: Wed Mar 9 11:18:46 2022 -0600 NIFI-9782 This closes #5854. Excluded H2 DB from nifi-druid-bundle Signed-off-by: Joe Witt --- nifi-nar-bundles/nifi-druid-bundle/pom.xml | 4 1 file changed, 4 insertions(+) diff --git a/nifi-nar-bundles/nifi-druid-bundle/pom.xml b/nifi-nar-bundles/nifi-druid-bundle/pom.xml index fcc34be..cff1cd4 100644 --- a/nifi-nar-bundles/nifi-druid-bundle/pom.xml +++ b/nifi-nar-bundles/nifi-druid-bundle/pom.xml @@ -97,6 +97,10 @@ commons-logging commons-logging + +com.h2database +h2 +
[nifi] 01/02: NIFI-9783: This closes #5855. When migrating FlowFiles from one ProcessSession to another, if any FlowFile had already been transferred, and the Relationship to which it was transferred
This is an automated email from the ASF dual-hosted git repository.
joewitt pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi.git
commit 73356ea448dced6a789f2d71c5cb5aceac45520e
Author: Mark Payne
AuthorDate: Wed Mar 9 13:44:08 2022 -0500
NIFI-9783: This closes #5855. When migrating FlowFiles from one
ProcessSession to another, if any FlowFile had already been transferred, and
the Relationship to which it was transferred was auto-terminated, we were
updating the wrong member variable, which threw off our stats for the
processor. Fixed that.
Signed-off-by: Joe Witt
---
.../repository/StandardProcessSession.java | 21 +++---
.../repository/StandardProcessSessionIT.java | 33 ++
2 files changed, 50 insertions(+), 4 deletions(-)
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/repository/StandardProcessSession.java
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/repository/StandardProcessSession.java
index 5bec8a6..bf1a1ab 100644
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/repository/StandardProcessSession.java
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/repository/StandardProcessSession.java
@@ -1599,11 +1599,24 @@ public class StandardProcessSession implements
ProcessSession, ProvenanceEventEn
}
if (repoRecord.getTransferRelationship() != null) {
-flowFilesOut--;
-contentSizeOut -= flowFile.getSize();
+final Relationship transferRelationship =
repoRecord.getTransferRelationship();
+final Collection destinations =
context.getConnections(transferRelationship);
+final int numDestinations = destinations.size();
+final boolean autoTerminated = numDestinations == 0 &&
context.getConnectable().isAutoTerminated(transferRelationship);
-newOwner.flowFilesOut++;
-newOwner.contentSizeOut += flowFile.getSize();
+if (autoTerminated) {
+removedCount--;
+removedBytes -= flowFile.getSize();
+
+newOwner.removedCount++;
+newOwner.removedBytes += flowFile.getSize();
+} else {
+flowFilesOut--;
+contentSizeOut -= flowFile.getSize();
+
+newOwner.flowFilesOut++;
+newOwner.contentSizeOut += flowFile.getSize();
+}
}
final List events =
generatedProvenanceEvents.remove(flowFile);
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/repository/StandardProcessSessionIT.java
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/repository/StandardProcessSessionIT.java
index dc51885..4e300d1 100644
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/repository/StandardProcessSessionIT.java
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/repository/StandardProcessSessionIT.java
@@ -2453,6 +2453,39 @@ public class StandardProcessSessionIT {
}
@Test
+public void testMigrateAfterTransferToAutoTerminatedRelationship() {
+final long start = System.currentTimeMillis();
+
+FlowFile flowFile = session.create();
+flowFile = session.write(flowFile, out ->
out.write("Hello".getBytes(StandardCharsets.UTF_8)));
+
+final StandardProcessSession newSession = new
StandardProcessSession(context, () -> false);
+
+
when(connectable.getConnections(any(Relationship.class))).thenReturn(Collections.emptySet());
+
when(connectable.isAutoTerminated(any(Relationship.class))).thenReturn(true);
+
+session.transfer(flowFile, new
Relationship.Builder().name("success").build());
+session.migrate(newSession, Collections.singleton(flowFile));
+
+session.commit();
+
+RepositoryStatusReport report =
flowFileEventRepository.reportTransferEvents(start - 1);
+FlowFileEvent event =
report.getReportEntries().values().iterator().next();
+assertEquals(0, event.getFlowFilesRemoved());
+assertEquals(0, event.getContentSizeRemoved());
+assertEquals(0, event.getFlowFilesOut());
+assertEquals(0, event.getContentSizeOut());
+
+newSession.commit();
+report = flowFileEventRepository.reportTransferEvents(start - 1);
+event =
[nifi] branch main updated (c73573b -> 4a46b08)
This is an automated email from the ASF dual-hosted git repository. joewitt pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git. from c73573b NIFI-9761 Correct PeerChannel processing for TLS 1.3 (#5836) new 73356ea NIFI-9783: This closes #5855. When migrating FlowFiles from one ProcessSession to another, if any FlowFile had already been transferred, and the Relationship to which it was transferred was auto-terminated, we were updating the wrong member variable, which threw off our stats for the processor. Fixed that. new 4a46b08 NIFI-9782 This closes #5854. Excluded H2 DB from nifi-druid-bundle The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: nifi-nar-bundles/nifi-druid-bundle/pom.xml | 4 +++ .../repository/StandardProcessSession.java | 21 +++--- .../repository/StandardProcessSessionIT.java | 33 ++ 3 files changed, 54 insertions(+), 4 deletions(-)
[nifi] branch main updated: NIFI-9761 Correct PeerChannel processing for TLS 1.3 (#5836)
This is an automated email from the ASF dual-hosted git repository.
markap14 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/main by this push:
new c73573b NIFI-9761 Correct PeerChannel processing for TLS 1.3 (#5836)
c73573b is described below
commit c73573b325a6e370bb3031e6906c786bcbf72ebc
Author: exceptionfactory
AuthorDate: Wed Mar 9 14:15:52 2022 -0600
NIFI-9761 Correct PeerChannel processing for TLS 1.3 (#5836)
* NIFI-9761 Corrected PeerChannel processing for TLS 1.3
- Added TestPeerChannel with methods for TLS 1.2 and TLS 1.3
- Updated PeerChannel.close() to process SSLEngine close notification
- Improved logging and corrected handling after decryption
---
.../nifi-framework/nifi-framework-core/pom.xml | 5 +
.../clustered/client/async/nio/PeerChannel.java| 169 ++
.../client/async/nio/TestPeerChannel.java | 254 +
3 files changed, 385 insertions(+), 43 deletions(-)
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/pom.xml
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/pom.xml
index 7a39b22..1eb6665 100644
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/pom.xml
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/pom.xml
@@ -265,6 +265,11 @@
test
+io.netty
+netty-handler
+test
+
+
com.github.ben-manes.caffeine
caffeine
2.8.1
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/client/async/nio/PeerChannel.java
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/client/async/nio/PeerChannel.java
index 67afb4a..5bee319 100644
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/client/async/nio/PeerChannel.java
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/client/async/nio/PeerChannel.java
@@ -23,6 +23,7 @@ import org.slf4j.LoggerFactory;
import javax.net.ssl.SSLEngine;
import javax.net.ssl.SSLEngineResult;
import javax.net.ssl.SSLException;
+import javax.net.ssl.SSLSession;
import java.io.Closeable;
import java.io.IOException;
import java.nio.ByteBuffer;
@@ -30,6 +31,10 @@ import java.nio.channels.SocketChannel;
import java.util.OptionalInt;
public class PeerChannel implements Closeable {
+private static final int END_OF_FILE = -1;
+
+private static final int EMPTY_BUFFER = 0;
+
private static final Logger logger =
LoggerFactory.getLogger(PeerChannel.class);
private final SocketChannel socketChannel;
@@ -38,7 +43,7 @@ public class PeerChannel implements Closeable {
private final ByteBuffer singleByteBuffer = ByteBuffer.allocate(1);
private ByteBuffer destinationBuffer = ByteBuffer.allocate(16 * 1024); //
buffer that SSLEngine is to write into
-private ByteBuffer streamBuffer = ByteBuffer.allocate(16 * 1024); //
buffer for data that is read from SocketChannel
+private final ByteBuffer streamBuffer = ByteBuffer.allocate(16 * 1024); //
buffer for data that is read from SocketChannel
private ByteBuffer applicationBuffer = ByteBuffer.allocate(0); // buffer
for application-level data that is ready to be served up (i.e., already
decrypted if necessary)
public PeerChannel(final SocketChannel socketChannel, final SSLEngine
sslEngine, final String peerDescription) {
@@ -47,10 +52,45 @@ public class PeerChannel implements Closeable {
this.peerDescription = peerDescription;
}
-
+/**
+ * Close Socket Channel and process SSLEngine close notifications when
configured
+ *
+ * @throws IOException Thrown on failure to close Socket Channel or
process SSLEngine operations
+ */
@Override
public void close() throws IOException {
-socketChannel.close();
+try {
+if (sslEngine == null) {
+logger.debug("Closing Peer Channel [{}] SSLEngine not
configured", peerDescription);
+} else {
+logger.debug("Closing Peer Channel [{}] SSLEngine close
started", peerDescription);
+sslEngine.closeOutbound();
+
+// Send TLS close notification packets available after
initiating SSLEngine.closeOutbound()
+final ByteBuffer inputBuffer = ByteBuffer.allocate(0);
+final ByteBuffer outputBuffer =
ByteBuffer.allocate(sslEngine.getSession().getPacketBufferSize());
+
+SSLEngineResult wrapResult =
[nifi] branch main updated: NIFI-9781: Fix handling when selecting array element via QueryRecord
This is an automated email from the ASF dual-hosted git repository.
jgresock pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/main by this push:
new 6a1c7c7 NIFI-9781: Fix handling when selecting array element via
QueryRecord
6a1c7c7 is described below
commit 6a1c7c72d5b91b9ce5d5cb5b86e3155d21e2c19b
Author: Mark Payne
AuthorDate: Wed Mar 9 11:04:00 2022 -0500
NIFI-9781: Fix handling when selecting array element via QueryRecord
Signed-off-by: Joe Gresock
This closes #5853.
---
.../nifi/serialization/record/ResultSetRecordSet.java | 19 +--
.../apache/nifi/queryrecord/FlowFileEnumerator.java | 2 +-
2 files changed, 18 insertions(+), 3 deletions(-)
diff --git
a/nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/ResultSetRecordSet.java
b/nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/ResultSetRecordSet.java
index cfb1e4b..8ce2255 100644
---
a/nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/ResultSetRecordSet.java
+++
b/nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/ResultSetRecordSet.java
@@ -18,6 +18,7 @@
package org.apache.nifi.serialization.record;
import org.apache.nifi.serialization.SimpleRecordSchema;
+import org.apache.nifi.serialization.record.type.ArrayDataType;
import org.apache.nifi.serialization.record.util.DataTypeUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -201,7 +202,7 @@ public class ResultSetRecordSet implements RecordSet,
Closeable {
throws SQLException {
switch (sqlType) {
case Types.ARRAY:
-return getArrayDataType(rs, columnIndex, useLogicalTypes);
+return getArrayDataType(rs, readerSchema, columnIndex,
useLogicalTypes);
case Types.BINARY:
case Types.LONGVARBINARY:
case Types.VARBINARY:
@@ -282,7 +283,21 @@ public class ResultSetRecordSet implements RecordSet,
Closeable {
}
}
-private DataType getArrayDataType(final ResultSet rs, final int
columnIndex, final boolean useLogicalTypes) throws SQLException {
+private DataType getArrayDataType(final ResultSet rs, final RecordSchema
readerSchema, final int columnIndex, final boolean useLogicalTypes) throws
SQLException {
+// We first want to check if the Reader Schema can tell us what the
type of the array is.
+final String columnName = rs.getMetaData().getColumnName(columnIndex);
+final Optional optionalRecordField =
readerSchema.getField(columnName);
+if (optionalRecordField.isPresent()) {
+final RecordField recordField = optionalRecordField.get();
+final DataType dataType = recordField.getDataType();
+if (dataType.getFieldType() == RecordFieldType.ARRAY) {
+final ArrayDataType arrayDataType = (ArrayDataType) dataType;
+if (arrayDataType.getElementType() != null) {
+return dataType;
+}
+}
+}
+
// The JDBC API does not allow us to know what the base type of an
array is through the metadata.
// As a result, we have to obtain the actual Array for this record.
Once we have this, we can determine
// the base type. However, if the base type is, itself, an array, we
will simply return a base type of
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/queryrecord/FlowFileEnumerator.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/queryrecord/FlowFileEnumerator.java
index e4814ec..db66c5a 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/queryrecord/FlowFileEnumerator.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/queryrecord/FlowFileEnumerator.java
@@ -107,7 +107,7 @@ public class FlowFileEnumerator implements
Enumerator {
// the actual value, NOT a 1-element array of values.
if (fields.length == 1) {
final int desiredCellIndex = fields[0];
-return row[desiredCellIndex];
+return cast(row[desiredCellIndex]);
}
// Create a new Object array that contains only the desired fields.
[nifi] branch main updated: NIFI-9728: Added support for User Assigned Managed Identity authentication for Azure ADLS and Blob_v12 processors
This is an automated email from the ASF dual-hosted git repository.
pvillard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/main by this push:
new 3219c10 NIFI-9728: Added support for User Assigned Managed Identity
authentication for Azure ADLS and Blob_v12 processors
3219c10 is described below
commit 3219c105ebd6d3a648c84684b2cb78efe81da438
Author: Peter Turcsanyi
AuthorDate: Sat Mar 5 13:49:46 2022 +0100
NIFI-9728: Added support for User Assigned Managed Identity authentication
for Azure ADLS and Blob_v12 processors
Also bumped Azure dependencies.
Signed-off-by: Pierre Villard
This closes #5846.
---
.../nifi-azure-processors/pom.xml | 6 +--
.../azure/AbstractAzureBlobProcessor_v12.java | 10 ++--
.../AbstractAzureDataLakeStorageProcessor.java | 2 +
.../azure/storage/utils/AzureStorageUtils.java | 11
.../storage/ADLSCredentialsControllerService.java | 61 ++
...ureStorageCredentialsControllerService_v12.java | 17 --
.../TestADLSCredentialsControllerService.java | 47 -
...ureStorageCredentialsControllerService_v12.java | 24 -
.../azure/storage/ADLSCredentialsDetails.java | 15 +-
.../AzureStorageCredentialsDetails_v12.java| 54 +++
nifi-nar-bundles/nifi-azure-bundle/pom.xml | 7 ++-
11 files changed, 178 insertions(+), 76 deletions(-)
diff --git a/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/pom.xml
b/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/pom.xml
index ceccc48..4a14fd2 100644
--- a/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/pom.xml
+++ b/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/pom.xml
@@ -22,9 +22,9 @@
3.3.0
3.3.0
-1.2.4
-
12.7.1
-12.14.1
+1.2.6
+
12.7.4
+12.14.4
diff --git
a/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/AbstractAzureBlobProcessor_v12.java
b/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/AbstractAzureBlobProcessor_v12.java
index 0186452..91e4420 100644
---
a/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/AbstractAzureBlobProcessor_v12.java
+++
b/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/AbstractAzureBlobProcessor_v12.java
@@ -130,13 +130,15 @@ public abstract class AbstractAzureBlobProcessor_v12
extends AbstractProcessor {
clientBuilder.credential(new
AzureSasCredential(credentialsDetails.getSasToken()));
break;
case MANAGED_IDENTITY:
-clientBuilder.credential(new
ManagedIdentityCredentialBuilder().build());
+clientBuilder.credential(new ManagedIdentityCredentialBuilder()
+
.clientId(credentialsDetails.getManagedIdentityClientId())
+.build());
break;
case SERVICE_PRINCIPAL:
clientBuilder.credential(new ClientSecretCredentialBuilder()
-.tenantId(credentialsDetails.getTenantId())
-.clientId(credentialsDetails.getClientId())
-.clientSecret(credentialsDetails.getClientSecret())
+
.tenantId(credentialsDetails.getServicePrincipalTenantId())
+
.clientId(credentialsDetails.getServicePrincipalClientId())
+
.clientSecret(credentialsDetails.getServicePrincipalClientSecret())
.build());
break;
case ACCESS_TOKEN:
diff --git
a/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/AbstractAzureDataLakeStorageProcessor.java
b/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/AbstractAzureDataLakeStorageProcessor.java
index d371e12..39e2ad2 100644
---
a/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/AbstractAzureDataLakeStorageProcessor.java
+++
b/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/AbstractAzureDataLakeStorageProcessor.java
@@ -129,6 +129,7 @@ public abstract class AbstractAzureDataLakeStorageProcessor
extends AbstractProc
final AccessToken accessToken = credentialsDetails.getAccessToken();
final String endpointSuffix = credentialsDetails.getEndpointSuffix();
final boolean useManagedIdentity =
credentialsDetails.getUseManagedIdentity();
+final String managedIdentityClientId =
[nifi] branch main updated: NIFI-9669 Adding PutDynamoDBRecord processor
This is an automated email from the ASF dual-hosted git repository.
turcsanyi pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/main by this push:
new b86466b NIFI-9669 Adding PutDynamoDBRecord processor
b86466b is described below
commit b86466b4a5eae655a0a708430da722a164cef8aa
Author: Bence Simon
AuthorDate: Fri Feb 11 16:57:57 2022 +0100
NIFI-9669 Adding PutDynamoDBRecord processor
This closes #5761.
Signed-off-by: Peter Turcsanyi
---
.../AbstractAWSCredentialsProviderProcessor.java | 1 +
.../nifi/processors/aws/AbstractAWSProcessor.java | 27 +-
.../nifi-aws-bundle/nifi-aws-processors/pom.xml| 9 +
.../processors/aws/dynamodb/DeleteDynamoDB.java| 24 +-
.../nifi/processors/aws/dynamodb/GetDynamoDB.java | 24 +-
.../nifi/processors/aws/dynamodb/PutDynamoDB.java | 24 +-
.../processors/aws/dynamodb/PutDynamoDBRecord.java | 378 +
.../aws/dynamodb/RecordToItemConverter.java| 119 +++
.../services/org.apache.nifi.processor.Processor | 1 +
.../additionalDetails.html | 260 ++
.../aws/dynamodb/PutDynamoDBRecordTest.java| 325 ++
.../aws/dynamodb/RecordToItemConverterTest.java| 265 +++
.../test/resources/dynamodb/multipleChunks.json| 31 ++
.../test/resources/dynamodb/multipleInputs.json| 5 +
.../test/resources/dynamodb/nonRecordOriented.txt | 1 +
.../src/test/resources/dynamodb/singleInput.json | 1 +
.../nifi/serialization/SplitRecordSetHandler.java | 102 ++
.../SplitRecordSetHandlerException.java| 28 ++
.../serialization/SplitRecordSetHandlerTest.java | 251 ++
19 files changed, 1832 insertions(+), 44 deletions(-)
diff --git
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSCredentialsProviderProcessor.java
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSCredentialsProviderProcessor.java
index 8b2c560..91cf927 100644
---
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSCredentialsProviderProcessor.java
+++
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSCredentialsProviderProcessor.java
@@ -50,6 +50,7 @@ public abstract class
AbstractAWSCredentialsProviderProcessor customValidate(final
ValidationContext validationContext) {
-final List problems = new
ArrayList<>(super.customValidate(validationContext));
+final List validationResults = new
ArrayList<>(super.customValidate(validationContext));
final boolean accessKeySet =
validationContext.getProperty(ACCESS_KEY).isSet();
final boolean secretKeySet =
validationContext.getProperty(SECRET_KEY).isSet();
if ((accessKeySet && !secretKeySet) || (secretKeySet &&
!accessKeySet)) {
-problems.add(new ValidationResult.Builder().input("Access
Key").valid(false).explanation("If setting Secret Key or Access Key, must set
both").build());
+validationResults.add(new ValidationResult.Builder().input("Access
Key").valid(false).explanation("If setting Secret Key or Access Key, must set
both").build());
}
final boolean credentialsFileSet =
validationContext.getProperty(CREDENTIALS_FILE).isSet();
if ((secretKeySet || accessKeySet) && credentialsFileSet) {
-problems.add(new ValidationResult.Builder().input("Access
Key").valid(false).explanation("Cannot set both Credentials File and Secret
Key/Access Key").build());
+validationResults.add(new ValidationResult.Builder().input("Access
Key").valid(false).explanation("Cannot set both Credentials File and Secret
Key/Access Key").build());
}
final boolean proxyHostSet =
validationContext.getProperty(PROXY_HOST).isSet();
final boolean proxyPortSet =
validationContext.getProperty(PROXY_HOST_PORT).isSet();
+final boolean proxyConfigServiceSet =
validationContext.getProperty(ProxyConfigurationService.PROXY_CONFIGURATION_SERVICE).isSet();
if ((proxyHostSet && !proxyPortSet) || (!proxyHostSet &&
proxyPortSet)) {
-problems.add(new ValidationResult.Builder().subject("Proxy Host
and Port").valid(false).explanation("If Proxy Host or Proxy Port is set, both
must be set").build());
+validationResults.add(new
ValidationResult.Builder().subject("Proxy Host and
Port").valid(false).explanation("If Proxy Host or Proxy Port is set, both must
be set").build());
}
final boolean proxyUserSet =
validationContext.getProperty(PROXY_USERNAME).isSet();
final boolean proxyPwdSet =
[nifi] branch main updated: NIFI-9774 Upgraded Netty from 4.1.73 to 4.1.74
This is an automated email from the ASF dual-hosted git repository. mthomsen pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/main by this push: new c8f8747 NIFI-9774 Upgraded Netty from 4.1.73 to 4.1.74 c8f8747 is described below commit c8f8747dbc6075be4feb3549fa5cc4b2e8a0fb7c Author: exceptionfactory AuthorDate: Tue Mar 8 12:27:14 2022 -0600 NIFI-9774 Upgraded Netty from 4.1.73 to 4.1.74 This closes #5851 Signed-off-by: Mike Thomsen --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index ec59fc0..3d6c253 100644 --- a/pom.xml +++ b/pom.xml @@ -121,7 +121,7 @@ 1.2.10 3.11.2 3.10.6.Final -4.1.73.Final +4.1.74.Final 5.3.16 5.6.2 2.1.210
[nifi] branch main updated: NIFI-9778 Fixing additional details for ScriptedPartitionRecord
This is an automated email from the ASF dual-hosted git repository. pvillard pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/main by this push: new 3b8afee NIFI-9778 Fixing additional details for ScriptedPartitionRecord 3b8afee is described below commit 3b8afee9d6b84f4b4467a81ddac6e3d5bae573ac Author: Bence Simon AuthorDate: Tue Mar 8 22:58:41 2022 +0100 NIFI-9778 Fixing additional details for ScriptedPartitionRecord Signed-off-by: Pierre Villard This closes #5852. --- .../additionalDetails.html| 0 1 file changed, 0 insertions(+), 0 deletions(-) diff --git a/nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/resources/docs/org.apache.nifi.processors.script.ScriptedPartitonRecord/additionalDetails.html b/nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/resources/docs/org.apache.nifi.processors.script.ScriptedPartitionRecord/additionalDetails.html similarity index 100% rename from nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/resources/docs/org.apache.nifi.processors.script.ScriptedPartitonRecord/additionalDetails.html rename to nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/resources/docs/org.apache.nifi.processors.script.ScriptedPartitionRecord/additionalDetails.html
