[GitHub] [nifi] mans2singh commented on issue #2956: NIFI-5537 Create Neo4J cypher execution processor
mans2singh commented on issue #2956: NIFI-5537 Create Neo4J cypher execution processor URL: https://github.com/apache/nifi/pull/2956#issuecomment-504636656 Thanks @MikeThomsen @ottobackwards @alopresto @mattyb149 @thenatog @joewitt and everyone for your advice/guidance. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] mans2singh commented on issue #2956: NIFI-5537 Create Neo4J cypher execution processor
mans2singh commented on issue #2956: NIFI-5537 Create Neo4J cypher execution processor URL: https://github.com/apache/nifi/pull/2956#issuecomment-504636451 Closing this request This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] mans2singh closed pull request #2956: NIFI-5537 Create Neo4J cypher execution processor
mans2singh closed pull request #2956: NIFI-5537 Create Neo4J cypher execution processor URL: https://github.com/apache/nifi/pull/2956 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] jtstorck edited a comment on issue #3015: NIFI-5254 [WIP] Updated to Groovy 2.5.2
jtstorck edited a comment on issue #3015: NIFI-5254 [WIP] Updated to Groovy 2.5.2 URL: https://github.com/apache/nifi/pull/3015#issuecomment-504563475 @MikeThomsen I'll be pushing a rebased [PR 3404](https://github.com/apache/nifi/pull/3404) fairly soon, which already includes replacing `groovy-all` with all the individual modules, and adds the needed modules to each lowest-level POM. It is that way on the current PR, if you want to take a look, though some of the changes are spread across a few commits, mainly [this one](https://github.com/apache/nifi/pull/3404/commits/4cd8de1f1753d460072e5ae94b5a50493e8beda6). Right now, I'm working on fixing some tests on my local rebase, and currently have the `groovy` dependency version set to 2.5.7. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] jtstorck commented on issue #3015: NIFI-5254 [WIP] Updated to Groovy 2.5.2
jtstorck commented on issue #3015: NIFI-5254 [WIP] Updated to Groovy 2.5.2 URL: https://github.com/apache/nifi/pull/3015#issuecomment-504563475 @MikeThomsen I'll be pushing a rebased [PR 3404](https://github.com/apache/nifi/pull/3404) fairly soon, which replaces `groovy-all` with all the individual modules, and adds the needed modules to each lowest-level POM. It is that way on the current PR, if you want to take a look, though some of the changes are spread across a few commits, mainly [this one](https://github.com/apache/nifi/pull/3404/commits/4cd8de1f1753d460072e5ae94b5a50493e8beda6). Right now, I'm working on fixing some tests on my local rebase, and currently have the `groovy` dependency version set to 2.5.7. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (NIFI-6391) ConsumeJMS/PublishJMS - Excessive connections created when get/publish operations continually fail
Steven Youtsey created NIFI-6391: Summary: ConsumeJMS/PublishJMS - Excessive connections created when get/publish operations continually fail Key: NIFI-6391 URL: https://issues.apache.org/jira/browse/NIFI-6391 Project: Apache NiFi Issue Type: Bug Components: Extensions Affects Versions: 1.8.0 Reporter: Steven Youtsey PublishJMS repeatedly threw "ResourceAllocationException: too many open connections" after successive publish failures due to issues on the remote JMS Broker. The cause of this is the specification and implementation of the JMSConnectionFactoryProviderDefinition.resetConnectionFactory(ConnectionFactory cf) method. Setting the ConnectionFactory to null may well indeed close connections upon 'destruction', but it may take awhile for the GC to run; in the meantime, more connections are opened. The connections need to be manually closed rather than waiting for a GC. >From AbstractJMSProcessor.onTrigger(), need to call worker.shutdown() prior to >resetConnectionFactory(). Also, noticed some problems iwth the ConnectionFactoryProviderDefinition implementatoins wrt the resetConnectionFactory methods. The factory is nulled but never re-initialized but for onEnabled(); which will lead to a NPE at some point. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [nifi] MikeThomsen commented on issue #3015: NIFI-5254 [WIP] Updated to Groovy 2.5.2
MikeThomsen commented on issue #3015: NIFI-5254 [WIP] Updated to Groovy 2.5.2 URL: https://github.com/apache/nifi/pull/3015#issuecomment-504560446 @jtstorck I'm going to try to pare down the use of groovy-all. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] alopresto commented on issue #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on issue #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP. URL: https://github.com/apache/nifi/pull/3542#issuecomment-504548386 Reviewing... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296367647
##
File path:
nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy
##
@@ -1568,7 +1572,7 @@ class ConfigEncryptionTool {
// If the tool is not going to encrypt NiFiProperties
and the existing file is already encrypted, encrypt and update the new
sensitive props key
if (!tool.handlingNiFiProperties &&
existingNiFiPropertiesAreEncrypted) {
-AESSensitivePropertyProvider spp = new
AESSensitivePropertyProvider(tool.keyHex)
+SensitivePropertyProvider spp =
StandardSensitivePropertyProvider.fromKey(tool.keyHex)
Review comment:
The field `tool.keyHex` probably needs to be re-examined as it is no longer
_just_ the AES key. When the internal logic was written, it was designed with
extensibility, but this tool was written as a helper script and not designed
for the injection of the provider. This should be re-evaluated due to the
introduction of the AWS KMS provider. (Another subtask - NIFI-6363).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296367647
##
File path:
nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy
##
@@ -1568,7 +1572,7 @@ class ConfigEncryptionTool {
// If the tool is not going to encrypt NiFiProperties
and the existing file is already encrypted, encrypt and update the new
sensitive props key
if (!tool.handlingNiFiProperties &&
existingNiFiPropertiesAreEncrypted) {
-AESSensitivePropertyProvider spp = new
AESSensitivePropertyProvider(tool.keyHex)
+SensitivePropertyProvider spp =
StandardSensitivePropertyProvider.fromKey(tool.keyHex)
Review comment:
The field `tool.keyHex` probably needs to be re-examined as it is no longer
_just_ the AES key. When the internal logic was written, it was designed with
extensibility, but this tool was written as a helper script and not designed
for the injection of the provider. This should be re-evaluated due to the
introduction of the AWS KMS provider.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296367647
##
File path:
nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy
##
@@ -1568,7 +1572,7 @@ class ConfigEncryptionTool {
// If the tool is not going to encrypt NiFiProperties
and the existing file is already encrypted, encrypt and update the new
sensitive props key
if (!tool.handlingNiFiProperties &&
existingNiFiPropertiesAreEncrypted) {
-AESSensitivePropertyProvider spp = new
AESSensitivePropertyProvider(tool.keyHex)
+SensitivePropertyProvider spp =
StandardSensitivePropertyProvider.fromKey(tool.keyHex)
Review comment:
The field `tool.keyHex` probably needs to be re-examined as it is no longer
_just_ the AES key. When the internal logic was written, it was designed with
extensibility, but this tool was written as a helper script and not designed
for the injection of the provider. This should be re-evaluated due to the
introduction of the AWS KMS provider. (Another subtask).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296367274
##
File path:
nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy
##
@@ -1269,10 +1273,10 @@ class ConfigEncryptionTool {
}
private
-static List
serializeNiFiPropertiesAndPreserveFormat(NiFiProperties niFiProperties, File
originalPropertiesFile) {
+static List
serializeNiFiPropertiesAndPreserveFormat(NiFiProperties niFiProperties, File
originalPropertiesFile, String keyHex) {
Review comment:
This would change to a provider instance due to the change in
`ProtectedNiFiProperties` constructors.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296365858
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/ProtectedNiFiProperties.java
##
@@ -39,7 +43,7 @@
* This encapsulates the sensitive property access logic from external
consumers
* of {@code NiFiProperties}.
*/
-class ProtectedNiFiProperties extends StandardNiFiProperties {
+public class ProtectedNiFiProperties extends StandardNiFiProperties {
Review comment:
The `localProviderCache` at Line 51 can be removed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296365532
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/test/groovy/org/apache/nifi/properties/ProtectedNiFiPropertiesGroovyTest.groovy
##
@@ -662,7 +630,7 @@ class ProtectedNiFiPropertiesGroovyTest extends
GroovyTestCase {
@Test
void testInstanceWithNoProtectedPropertiesShouldNotLoadSPP() throws
Exception {
// Arrange
-ProtectedNiFiProperties properties =
loadFromFile("/conf/nifi.properties")
+ProtectedNiFiProperties properties =
loadFromFile("/conf/nifi.properties", KEY_HEX)
assert properties.@localProviderCache?.isEmpty()
Review comment:
I think `localProviderCache` can be removed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296365134
##
File path:
nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy
##
@@ -1269,10 +1273,10 @@ class ConfigEncryptionTool {
}
private
-static List
serializeNiFiPropertiesAndPreserveFormat(NiFiProperties niFiProperties, File
originalPropertiesFile) {
+static List
serializeNiFiPropertiesAndPreserveFormat(NiFiProperties niFiProperties, File
originalPropertiesFile, String keyHex) {
List lines = originalPropertiesFile.readLines()
-ProtectedNiFiProperties protectedNiFiProperties = new
ProtectedNiFiProperties(niFiProperties)
+ProtectedNiFiProperties protectedNiFiProperties = new
ProtectedNiFiProperties(niFiProperties, keyHex) // BOOM WHAT
Review comment:
Seems indicative of an accomplishment.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296364937
##
File path:
nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy
##
@@ -1091,7 +1095,7 @@ class ConfigEncryptionTool {
encryptedProperties.setProperty(key,
plainProperties.getProperty(key))
}
NiFiProperties mergedProperties = new
StandardNiFiProperties(encryptedProperties)
-logger.info("Final result: ${mergedProperties.size()} keys including
${ProtectedNiFiProperties.countProtectedProperties(mergedProperties)} protected
keys")
+logger.info("Final result: ${mergedProperties.size()} keys including
${ProtectedNiFiProperties.countProtectedProperties(mergedProperties, keyHex)}
protected keys")
Review comment:
I think the implementation of
`ProtectedNiFiProperties#countProtectedProperties()` needs to change so that it
doesn't require the key hex or ID in order to count the protected properties.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296364540
##
File path:
nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy
##
@@ -1043,12 +1050,12 @@ class ConfigEncryptionTool {
* @param plainProperties the NiFiProperties instance containing the raw
values
* @return the NiFiProperties containing protected values
*/
-private NiFiProperties encryptSensitiveProperties(NiFiProperties
plainProperties) {
+private NiFiProperties encryptSensitiveProperties(NiFiProperties
plainProperties, String keyHex) {
Review comment:
Rename `keyHex` to `keyHexOrKeyId` to indicate this can support AWS KMS as
well.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296364129
##
File path:
nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy
##
@@ -873,12 +877,14 @@ class ConfigEncryptionTool {
}
String decryptLoginIdentityProviders(String encryptedXml, String
existingKeyHex = keyHex) {
-AESSensitivePropertyProvider sensitivePropertyProvider = new
AESSensitivePropertyProvider(existingKeyHex)
+SensitivePropertyProvider sensitivePropertyProvider =
StandardSensitivePropertyProvider.fromKey(existingKeyHex)
try {
def doc = getXmlSlurper().parseText(encryptedXml)
// Find the provider element by class even if it has been renamed
def passwords = doc.provider.find { it.'class' as String ==
LDAP_PROVIDER_CLASS }.property.findAll {
+// TODO: Update filter to include AWS identifier
Review comment:
Possibly to be done as subtask (needs new Jira).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296363703
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/main/java/org/apache/nifi/web/security/spring/LoginIdentityProviderFactoryBean.java
##
@@ -219,16 +217,14 @@ private LoginIdentityProviderConfigurationContext
loadLoginIdentityProviderConfi
}
private String decryptValue(String cipherText, String encryptionScheme)
throws SensitivePropertyProtectionException {
-initializeSensitivePropertyProvider(encryptionScheme);
+initializeSensitivePropertyProvider(encryptionScheme);
return SENSITIVE_PROPERTY_PROVIDER.unprotect(cipherText);
}
private static void initializeSensitivePropertyProvider(String
encryptionScheme) throws SensitivePropertyProtectionException {
-if (SENSITIVE_PROPERTY_PROVIDER == null ||
!SENSITIVE_PROPERTY_PROVIDER.getIdentifierKey().equalsIgnoreCase(encryptionScheme))
{
+if (SENSITIVE_PROPERTY_PROVIDER == null) {
try {
-String keyHex = getMasterKey();
-SENSITIVE_PROPERTY_PROVIDER_FACTORY = new
AESSensitivePropertyProviderFactory(keyHex);
-SENSITIVE_PROPERTY_PROVIDER =
SENSITIVE_PROPERTY_PROVIDER_FACTORY.getProvider();
+SENSITIVE_PROPERTY_PROVIDER =
StandardSensitivePropertyProvider.fromKey(getMasterKey(), encryptionScheme);
Review comment:
May need to write test cases around a `login-identity-providers.xml` file
with the AWS KMS key ID embedded in the `encryption="..."` attribute of the
individual encrypted elements. (Possibly as subtask; requires new Jira).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296363703
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/main/java/org/apache/nifi/web/security/spring/LoginIdentityProviderFactoryBean.java
##
@@ -219,16 +217,14 @@ private LoginIdentityProviderConfigurationContext
loadLoginIdentityProviderConfi
}
private String decryptValue(String cipherText, String encryptionScheme)
throws SensitivePropertyProtectionException {
-initializeSensitivePropertyProvider(encryptionScheme);
+initializeSensitivePropertyProvider(encryptionScheme);
return SENSITIVE_PROPERTY_PROVIDER.unprotect(cipherText);
}
private static void initializeSensitivePropertyProvider(String
encryptionScheme) throws SensitivePropertyProtectionException {
-if (SENSITIVE_PROPERTY_PROVIDER == null ||
!SENSITIVE_PROPERTY_PROVIDER.getIdentifierKey().equalsIgnoreCase(encryptionScheme))
{
+if (SENSITIVE_PROPERTY_PROVIDER == null) {
try {
-String keyHex = getMasterKey();
-SENSITIVE_PROPERTY_PROVIDER_FACTORY = new
AESSensitivePropertyProviderFactory(keyHex);
-SENSITIVE_PROPERTY_PROVIDER =
SENSITIVE_PROPERTY_PROVIDER_FACTORY.getProvider();
+SENSITIVE_PROPERTY_PROVIDER =
StandardSensitivePropertyProvider.fromKey(getMasterKey(), encryptionScheme);
Review comment:
May need to write test cases around a `login-identity-providers.xml` file
with the AWS KMS key ID embedded in the `encryption="..."` attribute of the
individual encrypted elements.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296362992
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-runtime/src/test/groovy/org/apache/nifi/NiFiGroovyTest.groovy
##
@@ -212,7 +211,7 @@ class NiFiGroovyTest extends GroovyTestCase {
}
private static NiFiProperties decrypt(NiFiProperties encryptedProperties,
String keyHex) {
Review comment:
Rename `keyHex` to `keyOrKeyId` as this could be AWS KMS key ID or AES key
in hex.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296361928
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/test/groovy/org/apache/nifi/properties/sensitive/aws/kms/AWSKMSSensitivePropertyProviderIT.groovy
##
@@ -0,0 +1,253 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties.sensitive.aws.kms
+
+import com.amazonaws.auth.PropertiesCredentials
+import com.amazonaws.services.kms.AWSKMSClient
+import com.amazonaws.services.kms.AWSKMSClientBuilder
+import com.amazonaws.services.kms.model.CreateAliasRequest
+import com.amazonaws.services.kms.model.CreateKeyRequest
+import com.amazonaws.services.kms.model.CreateKeyResult
+import com.amazonaws.services.kms.model.DescribeKeyRequest
+import com.amazonaws.services.kms.model.DescribeKeyResult
+import com.amazonaws.services.kms.model.GenerateDataKeyRequest
+import com.amazonaws.services.kms.model.GenerateDataKeyResult
+import com.amazonaws.services.kms.model.ScheduleKeyDeletionRequest
+import org.apache.nifi.properties.StandardNiFiProperties
+import org.apache.nifi.properties.sensitive.ProtectedNiFiProperties
+import
org.apache.nifi.properties.sensitive.SensitivePropertyProtectionException
+import org.apache.nifi.properties.sensitive.SensitivePropertyProvider
+import org.apache.nifi.util.NiFiProperties
+import org.junit.After
+import org.junit.AfterClass
+import org.junit.Before
+import org.junit.BeforeClass
+import org.junit.Test
+import org.junit.runner.RunWith
+import org.junit.runners.JUnit4
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+import java.security.SecureRandom
+
+
+@RunWith(JUnit4.class)
+class AWSKMSSensitivePropertyProviderIT extends GroovyTestCase {
+private static final Logger logger =
LoggerFactory.getLogger(AWSKMSSensitivePropertyProviderIT.class)
+protected final static String CREDENTIALS_FILE =
System.getProperty("user.home") + "/aws-credentials.properties";
+private static String[] knownGoodKeys = []
+private static AWSKMSClient client
+
+/**
+ * This method creates a CMK, DEK, and an alias to that DEK for exercising
the AWS KMS calls.
+ *
+ * @throws Exception
+ */
+@BeforeClass
+static void setUpOnce() throws Exception {
+final FileInputStream fis
+try {
+fis = new FileInputStream(CREDENTIALS_FILE)
+} catch (FileNotFoundException e1) {
+fail("Could not open credentials file " + CREDENTIALS_FILE + ": "
+ e1.getLocalizedMessage());
+return
+}
+final PropertiesCredentials credentials = new
PropertiesCredentials(fis)
+
+// We're copying the properties directly so the standard builder works.
+System.setProperty("aws.accessKeyId", credentials.AWSAccessKeyId)
+System.setProperty("aws.secretKey", credentials.AWSSecretKey)
+System.setProperty("aws.region", "us-east-2")
+
+client = AWSKMSClientBuilder.standard().build() as AWSKMSClient
+
+// generate a cmk
+CreateKeyRequest cmkRequest = new
CreateKeyRequest().withDescription("CMK for unit tests")
+CreateKeyResult cmkResult = client.createKey(cmkRequest)
+
+// from the cmk, generate a dek
+GenerateDataKeyRequest dekRequest = new
GenerateDataKeyRequest().withKeyId(cmkResult.keyMetadata.getKeyId()).withKeySpec("AES_128")
+GenerateDataKeyResult dekResult = client.generateDataKey(dekRequest)
+
+// add an alias to the dek
+final String aliasName = "alias/aws-kms-spp-integration-test-" +
UUID.randomUUID().toString()
+CreateAliasRequest aliasReq = new
CreateAliasRequest().withAliasName(aliasName).withTargetKeyId(dekResult.getKeyId())
+client.createAlias(aliasReq)
+
+// re-read the dek so we have the arn
+DescribeKeyRequest descRequest = new
DescribeKeyRequest().withKeyId(dekResult.getKeyId())
+DescribeKeyResult descResult = client.describeKey(descRequest)
+
+knownGoodKeys = [
+dekResult.getKeyId(),
+descResult.keyM
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296361258
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/test/groovy/org/apache/nifi/properties/sensitive/aws/kms/AWSKMSSensitivePropertyProviderIT.groovy
##
@@ -0,0 +1,253 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties.sensitive.aws.kms
+
+import com.amazonaws.auth.PropertiesCredentials
+import com.amazonaws.services.kms.AWSKMSClient
+import com.amazonaws.services.kms.AWSKMSClientBuilder
+import com.amazonaws.services.kms.model.CreateAliasRequest
+import com.amazonaws.services.kms.model.CreateKeyRequest
+import com.amazonaws.services.kms.model.CreateKeyResult
+import com.amazonaws.services.kms.model.DescribeKeyRequest
+import com.amazonaws.services.kms.model.DescribeKeyResult
+import com.amazonaws.services.kms.model.GenerateDataKeyRequest
+import com.amazonaws.services.kms.model.GenerateDataKeyResult
+import com.amazonaws.services.kms.model.ScheduleKeyDeletionRequest
+import org.apache.nifi.properties.StandardNiFiProperties
+import org.apache.nifi.properties.sensitive.ProtectedNiFiProperties
+import
org.apache.nifi.properties.sensitive.SensitivePropertyProtectionException
+import org.apache.nifi.properties.sensitive.SensitivePropertyProvider
+import org.apache.nifi.util.NiFiProperties
+import org.junit.After
+import org.junit.AfterClass
+import org.junit.Before
+import org.junit.BeforeClass
+import org.junit.Test
+import org.junit.runner.RunWith
+import org.junit.runners.JUnit4
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+import java.security.SecureRandom
+
+
+@RunWith(JUnit4.class)
+class AWSKMSSensitivePropertyProviderIT extends GroovyTestCase {
+private static final Logger logger =
LoggerFactory.getLogger(AWSKMSSensitivePropertyProviderIT.class)
+protected final static String CREDENTIALS_FILE =
System.getProperty("user.home") + "/aws-credentials.properties";
+private static String[] knownGoodKeys = []
+private static AWSKMSClient client
+
+/**
+ * This method creates a CMK, DEK, and an alias to that DEK for exercising
the AWS KMS calls.
+ *
+ * @throws Exception
+ */
+@BeforeClass
+static void setUpOnce() throws Exception {
+final FileInputStream fis
+try {
+fis = new FileInputStream(CREDENTIALS_FILE)
+} catch (FileNotFoundException e1) {
+fail("Could not open credentials file " + CREDENTIALS_FILE + ": "
+ e1.getLocalizedMessage());
+return
+}
+final PropertiesCredentials credentials = new
PropertiesCredentials(fis)
+
+// We're copying the properties directly so the standard builder works.
+System.setProperty("aws.accessKeyId", credentials.AWSAccessKeyId)
+System.setProperty("aws.secretKey", credentials.AWSSecretKey)
+System.setProperty("aws.region", "us-east-2")
+
+client = AWSKMSClientBuilder.standard().build() as AWSKMSClient
+
+// generate a cmk
+CreateKeyRequest cmkRequest = new
CreateKeyRequest().withDescription("CMK for unit tests")
+CreateKeyResult cmkResult = client.createKey(cmkRequest)
+
+// from the cmk, generate a dek
+GenerateDataKeyRequest dekRequest = new
GenerateDataKeyRequest().withKeyId(cmkResult.keyMetadata.getKeyId()).withKeySpec("AES_128")
+GenerateDataKeyResult dekResult = client.generateDataKey(dekRequest)
+
+// add an alias to the dek
+final String aliasName = "alias/aws-kms-spp-integration-test-" +
UUID.randomUUID().toString()
+CreateAliasRequest aliasReq = new
CreateAliasRequest().withAliasName(aliasName).withTargetKeyId(dekResult.getKeyId())
+client.createAlias(aliasReq)
+
+// re-read the dek so we have the arn
+DescribeKeyRequest descRequest = new
DescribeKeyRequest().withKeyId(dekResult.getKeyId())
+DescribeKeyResult descResult = client.describeKey(descRequest)
+
+knownGoodKeys = [
+dekResult.getKeyId(),
+descResult.keyM
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296360868
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/test/groovy/org/apache/nifi/properties/sensitive/aws/kms/AWSKMSSensitivePropertyProviderIT.groovy
##
@@ -0,0 +1,253 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties.sensitive.aws.kms
+
+import com.amazonaws.auth.PropertiesCredentials
+import com.amazonaws.services.kms.AWSKMSClient
+import com.amazonaws.services.kms.AWSKMSClientBuilder
+import com.amazonaws.services.kms.model.CreateAliasRequest
+import com.amazonaws.services.kms.model.CreateKeyRequest
+import com.amazonaws.services.kms.model.CreateKeyResult
+import com.amazonaws.services.kms.model.DescribeKeyRequest
+import com.amazonaws.services.kms.model.DescribeKeyResult
+import com.amazonaws.services.kms.model.GenerateDataKeyRequest
+import com.amazonaws.services.kms.model.GenerateDataKeyResult
+import com.amazonaws.services.kms.model.ScheduleKeyDeletionRequest
+import org.apache.nifi.properties.StandardNiFiProperties
+import org.apache.nifi.properties.sensitive.ProtectedNiFiProperties
+import
org.apache.nifi.properties.sensitive.SensitivePropertyProtectionException
+import org.apache.nifi.properties.sensitive.SensitivePropertyProvider
+import org.apache.nifi.util.NiFiProperties
+import org.junit.After
+import org.junit.AfterClass
+import org.junit.Before
+import org.junit.BeforeClass
+import org.junit.Test
+import org.junit.runner.RunWith
+import org.junit.runners.JUnit4
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+import java.security.SecureRandom
+
+
+@RunWith(JUnit4.class)
+class AWSKMSSensitivePropertyProviderIT extends GroovyTestCase {
+private static final Logger logger =
LoggerFactory.getLogger(AWSKMSSensitivePropertyProviderIT.class)
+protected final static String CREDENTIALS_FILE =
System.getProperty("user.home") + "/aws-credentials.properties";
+private static String[] knownGoodKeys = []
+private static AWSKMSClient client
+
+/**
+ * This method creates a CMK, DEK, and an alias to that DEK for exercising
the AWS KMS calls.
+ *
+ * @throws Exception
+ */
+@BeforeClass
+static void setUpOnce() throws Exception {
+final FileInputStream fis
+try {
+fis = new FileInputStream(CREDENTIALS_FILE)
+} catch (FileNotFoundException e1) {
+fail("Could not open credentials file " + CREDENTIALS_FILE + ": "
+ e1.getLocalizedMessage());
+return
+}
+final PropertiesCredentials credentials = new
PropertiesCredentials(fis)
+
+// We're copying the properties directly so the standard builder works.
+System.setProperty("aws.accessKeyId", credentials.AWSAccessKeyId)
+System.setProperty("aws.secretKey", credentials.AWSSecretKey)
+System.setProperty("aws.region", "us-east-2")
+
+client = AWSKMSClientBuilder.standard().build() as AWSKMSClient
+
+// generate a cmk
+CreateKeyRequest cmkRequest = new
CreateKeyRequest().withDescription("CMK for unit tests")
+CreateKeyResult cmkResult = client.createKey(cmkRequest)
+
Review comment:
I would add more logging around the process occurring here to illustrate
progress.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296360246
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/test/groovy/org/apache/nifi/properties/sensitive/aws/kms/AWSKMSSensitivePropertyProviderIT.groovy
##
@@ -0,0 +1,253 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties.sensitive.aws.kms
+
+import com.amazonaws.auth.PropertiesCredentials
+import com.amazonaws.services.kms.AWSKMSClient
+import com.amazonaws.services.kms.AWSKMSClientBuilder
+import com.amazonaws.services.kms.model.CreateAliasRequest
+import com.amazonaws.services.kms.model.CreateKeyRequest
+import com.amazonaws.services.kms.model.CreateKeyResult
+import com.amazonaws.services.kms.model.DescribeKeyRequest
+import com.amazonaws.services.kms.model.DescribeKeyResult
+import com.amazonaws.services.kms.model.GenerateDataKeyRequest
+import com.amazonaws.services.kms.model.GenerateDataKeyResult
+import com.amazonaws.services.kms.model.ScheduleKeyDeletionRequest
+import org.apache.nifi.properties.StandardNiFiProperties
+import org.apache.nifi.properties.sensitive.ProtectedNiFiProperties
+import
org.apache.nifi.properties.sensitive.SensitivePropertyProtectionException
+import org.apache.nifi.properties.sensitive.SensitivePropertyProvider
+import org.apache.nifi.util.NiFiProperties
+import org.junit.After
+import org.junit.AfterClass
+import org.junit.Before
+import org.junit.BeforeClass
+import org.junit.Test
+import org.junit.runner.RunWith
+import org.junit.runners.JUnit4
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+import java.security.SecureRandom
+
+
+@RunWith(JUnit4.class)
+class AWSKMSSensitivePropertyProviderIT extends GroovyTestCase {
+private static final Logger logger =
LoggerFactory.getLogger(AWSKMSSensitivePropertyProviderIT.class)
+protected final static String CREDENTIALS_FILE =
System.getProperty("user.home") + "/aws-credentials.properties";
+private static String[] knownGoodKeys = []
+private static AWSKMSClient client
+
+/**
+ * This method creates a CMK, DEK, and an alias to that DEK for exercising
the AWS KMS calls.
+ *
+ * @throws Exception
+ */
+@BeforeClass
+static void setUpOnce() throws Exception {
+final FileInputStream fis
+try {
+fis = new FileInputStream(CREDENTIALS_FILE)
+} catch (FileNotFoundException e1) {
+fail("Could not open credentials file " + CREDENTIALS_FILE + ": "
+ e1.getLocalizedMessage());
+return
+}
+final PropertiesCredentials credentials = new
PropertiesCredentials(fis)
+
+// We're copying the properties directly so the standard builder works.
+System.setProperty("aws.accessKeyId", credentials.AWSAccessKeyId)
+System.setProperty("aws.secretKey", credentials.AWSSecretKey)
+System.setProperty("aws.region", "us-east-2")
+
+client = AWSKMSClientBuilder.standard().build() as AWSKMSClient
+
+// generate a cmk
+CreateKeyRequest cmkRequest = new
CreateKeyRequest().withDescription("CMK for unit tests")
+CreateKeyResult cmkResult = client.createKey(cmkRequest)
+
+// from the cmk, generate a dek
+GenerateDataKeyRequest dekRequest = new
GenerateDataKeyRequest().withKeyId(cmkResult.keyMetadata.getKeyId()).withKeySpec("AES_128")
+GenerateDataKeyResult dekResult = client.generateDataKey(dekRequest)
+
+// add an alias to the dek
+final String aliasName = "alias/aws-kms-spp-integration-test-" +
UUID.randomUUID().toString()
+CreateAliasRequest aliasReq = new
CreateAliasRequest().withAliasName(aliasName).withTargetKeyId(dekResult.getKeyId())
+client.createAlias(aliasReq)
+
+// re-read the dek so we have the arn
+DescribeKeyRequest descRequest = new
DescribeKeyRequest().withKeyId(dekResult.getKeyId())
+DescribeKeyResult descResult = client.describeKey(descRequest)
+
+knownGoodKeys = [
+dekResult.getKeyId(),
+descResult.keyM
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296357769
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/test/groovy/org/apache/nifi/properties/ProtectedNiFiPropertiesGroovyTest.groovy
##
@@ -77,7 +83,7 @@ class ProtectedNiFiPropertiesGroovyTest extends
GroovyTestCase {
}
}
-private static ProtectedNiFiProperties loadFromFile(String
propertiesFilePath) {
+private static ProtectedNiFiProperties loadFromFile(String
propertiesFilePath, String defaultKeyHex) {
Review comment:
Change `keyHex` to provider instance as with application code.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296357351
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/test/groovy/org/apache/nifi/properties/NiFiPropertiesLoaderGroovyTest.groovy
##
@@ -119,6 +115,7 @@ class NiFiPropertiesLoaderGroovyTest extends
GroovyTestCase {
assert niFiPropertiesLoader.@keyHex == KEY_HEX
}
+@Ignore // functionality / test should move to
StandardSensitivePropertyProvider class
Review comment:
Implement comment by moving functionality to referenced test.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296357031
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/aws/kms/AWSKMSSensitivePropertyProvider.java
##
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties.sensitive.aws.kms;
+
+import com.amazonaws.services.kms.AWSKMS;
+import com.amazonaws.services.kms.AWSKMSClientBuilder;
+import com.amazonaws.services.kms.model.DecryptRequest;
+import com.amazonaws.services.kms.model.DecryptResult;
+import com.amazonaws.services.kms.model.EncryptRequest;
+import com.amazonaws.services.kms.model.EncryptResult;
+import java.nio.ByteBuffer;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.properties.sensitive.SensitivePropertyMetadata;
+import
org.apache.nifi.properties.sensitive.SensitivePropertyProtectionException;
+import org.apache.nifi.properties.sensitive.SensitivePropertyProvider;
+import org.bouncycastle.util.encoders.Base64;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * This provider uses the AWS SDK to interact with the AWS KMS. Values are
encoded/decoded base64, using the
+ * standard encoders from bouncycastle.
+ */
+public class AWSKMSSensitivePropertyProvider implements
SensitivePropertyProvider {
+private static final Logger logger =
LoggerFactory.getLogger(AWSKMSSensitivePropertyProvider.class);
+
+private static final String IMPLEMENTATION_NAME = "AWS KMS Sensitive
Property Provider";
+protected static final String IMPLEMENTATION_KEY = "aws/kms/";
+
+private AWSKMS client;
+private final String key;
+
+public AWSKMSSensitivePropertyProvider(String keyId) {
+this.key = validateKey(keyId);
+this.client = AWSKMSClientBuilder.standard().build();
+}
+
+private String validateKey(String keyId) {
+if (keyId == null || StringUtils.isBlank(keyId)) {
+throw new SensitivePropertyProtectionException("The key cannot be
empty");
+}
+if (keyId.startsWith(IMPLEMENTATION_KEY)) {
+keyId = keyId.substring(IMPLEMENTATION_KEY.length());
+}
+return keyId;
+}
+
+/**
+ * Returns the name of the underlying implementation.
+ *
+ * @return the name of this sensitive property provider
+ */
+@Override
+public String getName() {
+return IMPLEMENTATION_NAME;
+
+}
+
+/**
+ * Returns the key used to identify the provider implementation in {@code
nifi.properties}.
+ *
+ * @return the key to persist in the sibling property
+ */
+@Override
+public String getIdentifierKey() {
+return IMPLEMENTATION_KEY + key; // getIdentifierKey() has to include
the kms key id/alias/arn
+}
+
+
+/**
+ * Returns the encrypted cipher text.
+ *
+ * @param unprotectedValue the sensitive value
+ * @return the value to persist in the {@code nifi.properties} file
+ * @throws SensitivePropertyProtectionException if there is an exception
encrypting the value
+ */
+@Override
+public String protect(String unprotectedValue) throws
SensitivePropertyProtectionException {
+if (unprotectedValue == null || unprotectedValue.trim().length() == 0)
{
+throw new IllegalArgumentException("Cannot encrypt an empty
value");
+}
+
+EncryptRequest request = new EncryptRequest()
+.withKeyId(key)
+.withPlaintext(ByteBuffer.wrap(unprotectedValue.getBytes()));
+
+EncryptResult response = client.encrypt(request);
+return Base64.toBase64String(response.getCiphertextBlob().array());
+}
+
+/**
+ * Returns the "protected" form of this value. This is a form which can
safely be persisted in the {@code nifi.properties} file without compromising
the value.
+ * An encryption-based provider would return a cipher text, while a
remote-lookup provider could return a unique ID to retrieve the secured value.
+ *
+ * @param unprotected
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296356717
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/aws/kms/AWSKMSSensitivePropertyProvider.java
##
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties.sensitive.aws.kms;
+
+import com.amazonaws.services.kms.AWSKMS;
+import com.amazonaws.services.kms.AWSKMSClientBuilder;
+import com.amazonaws.services.kms.model.DecryptRequest;
+import com.amazonaws.services.kms.model.DecryptResult;
+import com.amazonaws.services.kms.model.EncryptRequest;
+import com.amazonaws.services.kms.model.EncryptResult;
+import java.nio.ByteBuffer;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.properties.sensitive.SensitivePropertyMetadata;
+import
org.apache.nifi.properties.sensitive.SensitivePropertyProtectionException;
+import org.apache.nifi.properties.sensitive.SensitivePropertyProvider;
+import org.bouncycastle.util.encoders.Base64;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * This provider uses the AWS SDK to interact with the AWS KMS. Values are
encoded/decoded base64, using the
+ * standard encoders from bouncycastle.
+ */
+public class AWSKMSSensitivePropertyProvider implements
SensitivePropertyProvider {
+private static final Logger logger =
LoggerFactory.getLogger(AWSKMSSensitivePropertyProvider.class);
+
+private static final String IMPLEMENTATION_NAME = "AWS KMS Sensitive
Property Provider";
+protected static final String IMPLEMENTATION_KEY = "aws/kms/";
+
+private AWSKMS client;
+private final String key;
+
+public AWSKMSSensitivePropertyProvider(String keyId) {
+this.key = validateKey(keyId);
+this.client = AWSKMSClientBuilder.standard().build();
+}
+
+private String validateKey(String keyId) {
+if (keyId == null || StringUtils.isBlank(keyId)) {
+throw new SensitivePropertyProtectionException("The key cannot be
empty");
+}
+if (keyId.startsWith(IMPLEMENTATION_KEY)) {
+keyId = keyId.substring(IMPLEMENTATION_KEY.length());
+}
+return keyId;
+}
+
+/**
+ * Returns the name of the underlying implementation.
+ *
+ * @return the name of this sensitive property provider
+ */
+@Override
+public String getName() {
+return IMPLEMENTATION_NAME;
+
+}
+
+/**
+ * Returns the key used to identify the provider implementation in {@code
nifi.properties}.
+ *
+ * @return the key to persist in the sibling property
+ */
+@Override
+public String getIdentifierKey() {
+return IMPLEMENTATION_KEY + key; // getIdentifierKey() has to include
the kms key id/alias/arn
+}
+
+
+/**
+ * Returns the encrypted cipher text.
+ *
+ * @param unprotectedValue the sensitive value
+ * @return the value to persist in the {@code nifi.properties} file
+ * @throws SensitivePropertyProtectionException if there is an exception
encrypting the value
+ */
+@Override
+public String protect(String unprotectedValue) throws
SensitivePropertyProtectionException {
+if (unprotectedValue == null || unprotectedValue.trim().length() == 0)
{
Review comment:
Using `StringUtils.isBlank(unprotectedValue)` is beneficial because it
encapsulates the trim and also checks if the content is all whitespace.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296356394
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/aws/kms/AWSKMSSensitivePropertyProvider.java
##
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties.sensitive.aws.kms;
+
+import com.amazonaws.services.kms.AWSKMS;
+import com.amazonaws.services.kms.AWSKMSClientBuilder;
+import com.amazonaws.services.kms.model.DecryptRequest;
+import com.amazonaws.services.kms.model.DecryptResult;
+import com.amazonaws.services.kms.model.EncryptRequest;
+import com.amazonaws.services.kms.model.EncryptResult;
+import java.nio.ByteBuffer;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.properties.sensitive.SensitivePropertyMetadata;
+import
org.apache.nifi.properties.sensitive.SensitivePropertyProtectionException;
+import org.apache.nifi.properties.sensitive.SensitivePropertyProvider;
+import org.bouncycastle.util.encoders.Base64;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * This provider uses the AWS SDK to interact with the AWS KMS. Values are
encoded/decoded base64, using the
+ * standard encoders from bouncycastle.
+ */
+public class AWSKMSSensitivePropertyProvider implements
SensitivePropertyProvider {
+private static final Logger logger =
LoggerFactory.getLogger(AWSKMSSensitivePropertyProvider.class);
+
+private static final String IMPLEMENTATION_NAME = "AWS KMS Sensitive
Property Provider";
+protected static final String IMPLEMENTATION_KEY = "aws/kms/";
+
+private AWSKMS client;
+private final String key;
+
+public AWSKMSSensitivePropertyProvider(String keyId) {
+this.key = validateKey(keyId);
+this.client = AWSKMSClientBuilder.standard().build();
+}
+
+private String validateKey(String keyId) {
Review comment:
I would add a Javadoc explanation that this doesn't just "validate" the key
by checking true/false, but also strips the key identifier prefix, returning
just the AWS KMS lookup ID.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296356212
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/aws/kms/AWSKMSSensitivePropertyProvider.java
##
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties.sensitive.aws.kms;
+
+import com.amazonaws.services.kms.AWSKMS;
+import com.amazonaws.services.kms.AWSKMSClientBuilder;
+import com.amazonaws.services.kms.model.DecryptRequest;
+import com.amazonaws.services.kms.model.DecryptResult;
+import com.amazonaws.services.kms.model.EncryptRequest;
+import com.amazonaws.services.kms.model.EncryptResult;
+import java.nio.ByteBuffer;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.properties.sensitive.SensitivePropertyMetadata;
+import
org.apache.nifi.properties.sensitive.SensitivePropertyProtectionException;
+import org.apache.nifi.properties.sensitive.SensitivePropertyProvider;
+import org.bouncycastle.util.encoders.Base64;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * This provider uses the AWS SDK to interact with the AWS KMS. Values are
encoded/decoded base64, using the
+ * standard encoders from bouncycastle.
+ */
+public class AWSKMSSensitivePropertyProvider implements
SensitivePropertyProvider {
+private static final Logger logger =
LoggerFactory.getLogger(AWSKMSSensitivePropertyProvider.class);
+
+private static final String IMPLEMENTATION_NAME = "AWS KMS Sensitive
Property Provider";
+protected static final String IMPLEMENTATION_KEY = "aws/kms/";
+
+private AWSKMS client;
+private final String key;
Review comment:
Possibly rename to `keyId` as this contains the identifier of the
KMS-provided key, not the actual key value.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296355311
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/aes/AESSensitivePropertyMetadata.java
##
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties.sensitive.aes;
+
+import java.security.NoSuchAlgorithmException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import javax.crypto.Cipher;
+import org.apache.nifi.properties.sensitive.SensitivePropertyMetadata;
+import
org.apache.nifi.properties.sensitive.SensitivePropertyProtectionException;
+import org.apache.nifi.util.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * An implementation of the metadata container for AES/GCM encryption using
{@link AESSensitivePropertyProvider} which contains the algorithm & mode
(fixed) and the key length used (variable).
+ */
+public class AESSensitivePropertyMetadata implements SensitivePropertyMetadata
{
Review comment:
If implementing the AWS KMS provider didn't require the concept of a
separate metadata companion object, we can remove the entire approach, as the
AES implementation did not need it previously.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296354919
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/StandardSensitivePropertyProvider.java
##
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties.sensitive;
+
+import org.apache.nifi.properties.sensitive.aes.AESSensitivePropertyProvider;
+import
org.apache.nifi.properties.sensitive.aws.kms.AWSKMSSensitivePropertyProvider;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * This class hides the various SPP subclass construction from clients.
+ *
+ */
+public class StandardSensitivePropertyProvider {
+private static final Logger logger =
LoggerFactory.getLogger(StandardSensitivePropertyProvider.class);
+
+/**
+ * Creates a {@link SensitivePropertyProvider} suitable for a given key.
+ *
+ * If no provider recognizes a key, this implementation still returns an
{@link AESSensitivePropertyProvider} with
+ * the supplied key.
+ *
+ * @param hex provider encryption key
+ * @param options array of string options
+ * @return concrete instance of SensitivePropertyProvider
+ */
+public static SensitivePropertyProvider fromKey(String hex, String...
options) {
+
+String scheme = "";
+if (options.length > 0) {
+scheme = options[0];
+}
+
+if (AWSKMSSensitivePropertyProvider.isProviderFor(hex, options)) {
+logger.debug("StandardSensitivePropertyProvider selected specific
AWS KMS for key: " + hex + " scheme: " + scheme);
+return new AWSKMSSensitivePropertyProvider(hex);
+
+} else if (AESSensitivePropertyProvider.isProviderFor(hex, options)) {
+logger.debug("StandardSensitivePropertyProvider selected specific
AES provider for key: " + hex + " scheme: " + scheme);
+return new AESSensitivePropertyProvider(hex);
+
+} else {
Review comment:
If the previous conditional fails (i.e. the AES provider does not recognize
the provided key), this should throw an error.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296353589
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/StandardSensitivePropertyProvider.java
##
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties.sensitive;
+
+import org.apache.nifi.properties.sensitive.aes.AESSensitivePropertyProvider;
+import
org.apache.nifi.properties.sensitive.aws.kms.AWSKMSSensitivePropertyProvider;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * This class hides the various SPP subclass construction from clients.
+ *
+ */
+public class StandardSensitivePropertyProvider {
+private static final Logger logger =
LoggerFactory.getLogger(StandardSensitivePropertyProvider.class);
+
+/**
+ * Creates a {@link SensitivePropertyProvider} suitable for a given key.
+ *
+ * If no provider recognizes a key, this implementation still returns an
{@link AESSensitivePropertyProvider} with
+ * the supplied key.
+ *
+ * @param hex provider encryption key
+ * @param options array of string options
+ * @return concrete instance of SensitivePropertyProvider
+ */
+public static SensitivePropertyProvider fromKey(String hex, String...
options) {
+
+String scheme = "";
+if (options.length > 0) {
+scheme = options[0];
+}
+
+if (AWSKMSSensitivePropertyProvider.isProviderFor(hex, options)) {
+logger.debug("StandardSensitivePropertyProvider selected specific
AWS KMS for key: " + hex + " scheme: " + scheme);
Review comment:
This was probably for debugging purposes, but we don't want to log the key.
For debugging purposes, perhaps a truncated, masked, or hashed version would
allow for "uniqueness" tracking across the application without revealing the
actual key value.
I.e. if this is a key of format `aws-key:arn:1243` which references a key in
the KMS, logging that is "ok" but not ideal. If it is the raw hex value of the
AES key `0123`, that is very bad. Hashing that value to a unique identifier
would be preferred.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296353589
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/StandardSensitivePropertyProvider.java
##
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties.sensitive;
+
+import org.apache.nifi.properties.sensitive.aes.AESSensitivePropertyProvider;
+import
org.apache.nifi.properties.sensitive.aws.kms.AWSKMSSensitivePropertyProvider;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * This class hides the various SPP subclass construction from clients.
+ *
+ */
+public class StandardSensitivePropertyProvider {
+private static final Logger logger =
LoggerFactory.getLogger(StandardSensitivePropertyProvider.class);
+
+/**
+ * Creates a {@link SensitivePropertyProvider} suitable for a given key.
+ *
+ * If no provider recognizes a key, this implementation still returns an
{@link AESSensitivePropertyProvider} with
+ * the supplied key.
+ *
+ * @param hex provider encryption key
+ * @param options array of string options
+ * @return concrete instance of SensitivePropertyProvider
+ */
+public static SensitivePropertyProvider fromKey(String hex, String...
options) {
+
+String scheme = "";
+if (options.length > 0) {
+scheme = options[0];
+}
+
+if (AWSKMSSensitivePropertyProvider.isProviderFor(hex, options)) {
+logger.debug("StandardSensitivePropertyProvider selected specific
AWS KMS for key: " + hex + " scheme: " + scheme);
Review comment:
This was probably for debugging purposes, but we don't want to log the key.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296353080
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/StandardSensitivePropertyProvider.java
##
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties.sensitive;
+
+import org.apache.nifi.properties.sensitive.aes.AESSensitivePropertyProvider;
+import
org.apache.nifi.properties.sensitive.aws.kms.AWSKMSSensitivePropertyProvider;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * This class hides the various SPP subclass construction from clients.
+ *
+ */
+public class StandardSensitivePropertyProvider {
+private static final Logger logger =
LoggerFactory.getLogger(StandardSensitivePropertyProvider.class);
+
+/**
+ * Creates a {@link SensitivePropertyProvider} suitable for a given key.
+ *
+ * If no provider recognizes a key, this implementation still returns an
{@link AESSensitivePropertyProvider} with
+ * the supplied key.
+ *
+ * @param hex provider encryption key
+ * @param options array of string options
+ * @return concrete instance of SensitivePropertyProvider
+ */
+public static SensitivePropertyProvider fromKey(String hex, String...
options) {
Review comment:
I think we can be more specific with the enumerated options and expect
stricter compliance via typed named parameters.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296352124
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/SensitivePropertyMetadata.java
##
@@ -14,10 +14,16 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.nifi.properties;
+package org.apache.nifi.properties.sensitive;
-public interface SensitivePropertyProviderFactory {
-
-SensitivePropertyProvider getProvider();
+/**
+ * Parent interface to allow provider-specific encryption metadata around
protected property values.
+ */
+public interface SensitivePropertyMetadata {
Review comment:
We may be able to remove the concept of encryption metadata as the change
from a long-lived, multiple key-handling provider implementation to a
flyweight, single-use provider might have rendered this unnecessary.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296350601
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/ProtectedNiFiProperties.java
##
@@ -510,22 +448,21 @@ private boolean isProviderAvailable(String
protectionScheme) {
*/
private String unprotectValue(String key, String retrievedValue) {
// Checks if the key is sensitive and marked as protected
-if (isPropertyProtected(key)) {
-final String protectionScheme = getProperty(getProtectionKey(key));
+if (!isPropertyProtected(key)) {
+return retrievedValue;
+}
+final String protectionScheme = getProperty(getProtectionKey(key));
-// No provider registered for this scheme, so just return the value
-if (!isProviderAvailable(protectionScheme)) {
-logger.warn("No provider available for {} so passing the
protected {} value back", protectionScheme, key);
-return retrievedValue;
-}
+if (protectionScheme.equals("unknown")) {
+return retrievedValue;
+}
-try {
-SensitivePropertyProvider sensitivePropertyProvider =
getSensitivePropertyProvider(protectionScheme);
-return sensitivePropertyProvider.unprotect(retrievedValue);
-} catch (SensitivePropertyProtectionException e) {
-throw new SensitivePropertyProtectionException("Error
unprotecting value for " + key, e.getCause());
-}
+// try and make one to unprotect, and if that fails...
Review comment:
Please add the guard check using
`StandardSensitivePropertyProvider.handlesProtectionScheme()` or similar to log
warning message and return protected value without attempting (and failing) to
create a provider implementation for a protection scheme we can't handle.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296349525
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/ProtectedNiFiProperties.java
##
@@ -510,22 +448,21 @@ private boolean isProviderAvailable(String
protectionScheme) {
*/
private String unprotectValue(String key, String retrievedValue) {
// Checks if the key is sensitive and marked as protected
-if (isPropertyProtected(key)) {
-final String protectionScheme = getProperty(getProtectionKey(key));
+if (!isPropertyProtected(key)) {
+return retrievedValue;
+}
+final String protectionScheme = getProperty(getProtectionKey(key));
-// No provider registered for this scheme, so just return the value
-if (!isProviderAvailable(protectionScheme)) {
-logger.warn("No provider available for {} so passing the
protected {} value back", protectionScheme, key);
-return retrievedValue;
-}
+if (protectionScheme.equals("unknown")) {
+return retrievedValue;
+}
-try {
-SensitivePropertyProvider sensitivePropertyProvider =
getSensitivePropertyProvider(protectionScheme);
-return sensitivePropertyProvider.unprotect(retrievedValue);
-} catch (SensitivePropertyProtectionException e) {
-throw new SensitivePropertyProtectionException("Error
unprotecting value for " + key, e.getCause());
-}
+// try and make one to unprotect, and if that fails...
+try {
+return
StandardSensitivePropertyProvider.fromKey(defaultKeyHex).unprotect(retrievedValue);
+} catch (final SensitivePropertyProtectionException e) {
+throw new SensitivePropertyProtectionException(e);
Review comment:
I think the prior behavior of wrapping the cause in a generic exception for
visibility safety was desired.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296349226
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/ProtectedNiFiProperties.java
##
@@ -510,22 +448,21 @@ private boolean isProviderAvailable(String
protectionScheme) {
*/
private String unprotectValue(String key, String retrievedValue) {
// Checks if the key is sensitive and marked as protected
-if (isPropertyProtected(key)) {
-final String protectionScheme = getProperty(getProtectionKey(key));
+if (!isPropertyProtected(key)) {
+return retrievedValue;
+}
+final String protectionScheme = getProperty(getProtectionKey(key));
-// No provider registered for this scheme, so just return the value
-if (!isProviderAvailable(protectionScheme)) {
-logger.warn("No provider available for {} so passing the
protected {} value back", protectionScheme, key);
-return retrievedValue;
-}
+if (protectionScheme.equals("unknown")) {
+return retrievedValue;
+}
-try {
-SensitivePropertyProvider sensitivePropertyProvider =
getSensitivePropertyProvider(protectionScheme);
-return sensitivePropertyProvider.unprotect(retrievedValue);
-} catch (SensitivePropertyProtectionException e) {
-throw new SensitivePropertyProtectionException("Error
unprotecting value for " + key, e.getCause());
-}
+// try and make one to unprotect, and if that fails...
+try {
+return
StandardSensitivePropertyProvider.fromKey(defaultKeyHex).unprotect(retrievedValue);
Review comment:
This should handle AWS KMS as well.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296348457
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/ProtectedNiFiProperties.java
##
@@ -459,46 +435,8 @@ public static int countProtectedProperties(NiFiProperties
plainProperties) {
* @param plainProperties the instance to count sensitive properties
* @return the number of sensitive properties
*/
-public static int countSensitiveProperties(NiFiProperties plainProperties)
{
Review comment:
The method was just moved during refactoring, but the invocation was removed
from `#toString()` so please restore that.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296348028
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/ProtectedNiFiProperties.java
##
@@ -459,46 +435,8 @@ public static int countProtectedProperties(NiFiProperties
plainProperties) {
* @param plainProperties the instance to count sensitive properties
* @return the number of sensitive properties
*/
-public static int countSensitiveProperties(NiFiProperties plainProperties)
{
Review comment:
Please restore this functionality and include it in the `#toString()` for
visibility into how many _sensitive_ properties are actually _protected_.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS KMS SPP. Refactors SSPP.
alopresto commented on a change in pull request #3542: NIFI-6363 Integrates AWS
KMS SPP. Refactors SSPP.
URL: https://github.com/apache/nifi/pull/3542#discussion_r296346445
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/ProtectedNiFiProperties.java
##
@@ -53,17 +57,17 @@
public static final List DEFAULT_SENSITIVE_PROPERTIES = new
ArrayList<>(asList(SECURITY_KEY_PASSWD,
SECURITY_KEYSTORE_PASSWD, SECURITY_TRUSTSTORE_PASSWD,
SENSITIVE_PROPS_KEY, PROVENANCE_REPO_ENCRYPTION_KEY));
-public ProtectedNiFiProperties() {
-this(new StandardNiFiProperties());
-}
+// Default encryption key, hex encoded.
+private String defaultKeyHex;
Review comment:
We should change this to the default provider rather than loading the raw
key and maintaining it throughout the lifecycle of this object.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (MINIFICPP-929) Create memory map interface to flow files in ProcessSession/ContentRepository
Andrew Christianson created MINIFICPP-929: - Summary: Create memory map interface to flow files in ProcessSession/ContentRepository Key: MINIFICPP-929 URL: https://issues.apache.org/jira/browse/MINIFICPP-929 Project: Apache NiFi MiNiFi C++ Issue Type: Improvement Reporter: Andrew Christianson Assignee: Andrew Christianson Currently, MiNiFi - C++ only support stream-oriented i/o to FlowFile payloads. This can limit performance in cases where in-place access to the payload is desirable. In cases where data can be accessed randomly and in-place, a significant speedup can be realized by mapping the payload into system memory address space. This is natively supported at the kernel level in Linux, MacOS, and Windows via the mmap() interface on files. Other repositories, such as the VolatileRepository, already store the entire payload in memory, so it is natural to pass through this memory block as if it were a memory-mapped file. While the DatabaseContentRepostory does not appear to natively support a memory map interface, accesses via an emulated memory-map interface should be possible with no performance degradation with respect to a full read via the streaming interface. Cases where in-place, random access is beneficial include, but are not limited to: * in-place parsing of JSON (e.g. RapidJSON supports parsing in-place, at least for strings). * access of payload via protocol buffers * random access of large files on disk, where it would otherwise require many seek() and read() syscalls The interface should be accessible by processors via a mmap() call on ProcessSession (adjacent to read() and write()). A MemoryMapCallback should be provided, which is called back via a process() call where the argument is an instance of BaseMemoryMap. The BaseMemoryMap is extended for each type of repository that MiNiFi - C++ supports, including: FileSystemRepository, VolatileRepository, and DatabaseContentRepository. As part of the change, in addition to extensive unit test coverage, benchmarks should be written such that the performance impact can be empirically measured and evaluated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [nifi] mattyb149 commented on a change in pull request #3317: NIFI-6047 Add DetectDuplicateRecord Processor
mattyb149 commented on a change in pull request #3317: NIFI-6047 Add
DetectDuplicateRecord Processor
URL: https://github.com/apache/nifi/pull/3317#discussion_r296316300
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestDetectDuplicateRecord.java
##
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.serialization.record.MockRecordParser;
+import org.apache.nifi.serialization.record.MockRecordWriter;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.util.*;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.nifi.processors.standard.DetectDuplicateRecord.*;
+import static org.junit.Assert.assertEquals;
+
+public class TestDetectDuplicateRecord {
+
+private TestRunner runner;
+private MockCacheService cache;
+private MockRecordParser reader;
+private MockRecordWriter writer;
+
+@BeforeClass
+public static void beforeClass() {
+System.setProperty("org.slf4j.simpleLogger.defaultLogLevel", "info");
+System.setProperty("org.slf4j.simpleLogger.showDateTime", "true");
+System.setProperty("org.slf4j.simpleLogger.log.nifi.io.nio", "debug");
+
System.setProperty("org.slf4j.simpleLogger.log.nifi.processors.standard.DetectDuplicateRecord",
"debug");
+
System.setProperty("org.slf4j.simpleLogger.log.nifi.processors.standard.TestDetectDuplicateRecord",
"debug");
+}
+
+@Before
+public void setup() throws InitializationException {
+runner = TestRunners.newTestRunner(DetectDuplicateRecord.class);
+
+// RECORD_READER, RECORD_WRITER
+reader = new MockRecordParser();
+writer = new MockRecordWriter("header", false);
+
+runner.addControllerService("reader", reader);
+runner.enableControllerService(reader);
+runner.addControllerService("writer", writer);
+runner.enableControllerService(writer);
+
+runner.setProperty(RECORD_READER, "reader");
+runner.setProperty(RECORD_WRITER, "writer");
+
+reader.addSchemaField("firstName", RecordFieldType.STRING);
+reader.addSchemaField("middleName", RecordFieldType.STRING);
+reader.addSchemaField("lastName", RecordFieldType.STRING);
+
+// INCLUDE_ZERO_RECORD_FLOWFILES
+runner.setProperty(INCLUDE_ZERO_RECORD_FLOWFILES, "true");
+
+// CACHE_IDENTIFIER
+runner.setProperty(CACHE_IDENTIFIER, "true");
+
+// DISTRIBUTED_CACHE_SERVICE
+cache = new MockCacheService();
+runner.addControllerService("cache", cache);
+runner.setProperty(DISTRIBUTED_CACHE_SERVICE, "cache");
+runner.enableControllerService(cache);
+
+// CACHE_ENTRY_IDENTIFIER
+final Map props = new HashMap<>();
+props.put("hash.value", "1000");
+runner.enqueue(new byte[]{}, props);
+
+// AGE_OFF_DURATION
+runner.setProperty(AGE_OFF_DURATION, "48 hours");
+
+runner.assertValid();
+}
+
+ @Test
+ public void testDetectDuplicatesHashSet() {
+runner.setProperty(FILTER_TYPE, HASH_SET_VALUE);
+runner.setProperty("/middleName", "${field.value}");
+reader.addRecord("John", "Q", "Smith");
+reader.addRecord("John", "Q", "Smith");
+reader.addRecord("Jane", "X", "Doe");
+
+runner.enqueue("");
+runner.run();
+
+doCountTests(0, 1, 1, 1, 2, 1);
+}
+
+@Test
+public void testDetectDuplicatesBloomFilter() {
+runner.setProperty(FILTER_TYPE, BLOOM_FILTER_VALUE);
+runner.setProperty(BLOOM_FILTER_FPP, "0.10");
+runner.setProperty("/middleName", "${field.value}");
+reader.addRecord("John", "Q", "Smith");
+reader.addRecord("John", "Q", "Smith");
+reader.addRecord("Jane", "X", "Doe");
+
+runner.enqueue("");
+runner.run();
+
+doCountTests(0
[GitHub] [nifi] mattyb149 commented on a change in pull request #3317: NIFI-6047 Add DetectDuplicateRecord Processor
mattyb149 commented on a change in pull request #3317: NIFI-6047 Add
DetectDuplicateRecord Processor
URL: https://github.com/apache/nifi/pull/3317#discussion_r296311336
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DetectDuplicateRecord.java
##
@@ -0,0 +1,646 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import com.google.common.base.Joiner;
+import com.google.common.hash.BloomFilter;
+import com.google.common.hash.Funnels;
+import org.apache.commons.codec.binary.Hex;
+import org.apache.commons.codec.digest.DigestUtils;
+import org.apache.commons.codec.digest.MessageDigestAlgorithms;
+import org.apache.nifi.annotation.behavior.*;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.*;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import
org.apache.nifi.distributed.cache.client.exception.DeserializationException;
+import
org.apache.nifi.distributed.cache.client.exception.SerializationException;
+import org.apache.nifi.expression.AttributeExpression.ResultType;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.*;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathPropertyNameValidator;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.serialization.*;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.util.DataTypeUtils;
+
+import java.io.*;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.util.*;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+import static java.util.stream.Collectors.toList;
+import static org.apache.commons.codec.binary.StringUtils.getBytesUtf8;
+import static org.apache.commons.lang3.StringUtils.*;
+
+@EventDriven
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@SystemResourceConsideration(resource = SystemResource.MEMORY,
+description = "The HashSet filter type will grow memory space
proportionate to the number of unique records processed. " +
+"The BloomFilter type will use constant memory regardless of the
number of records processed.")
+@Tags({"text", "record", "update", "change", "replace", "modify", "distinct",
"unique",
+"filter", "hash", "dupe", "duplicate", "dedupe"})
+@CapabilityDescription("Caches records from each incoming FlowFile and
determines if the record " +
+"has already been seen. If so, routes the record to 'duplicate'. If the
record is " +
+"not determined to be a duplicate, it is routed to 'non-duplicate'."
+)
+@WritesAttribute(attribute = "record.count", description = "The number of
records processed.")
+@DynamicProperty(
+name = "RecordPath",
+value = "An expression language statement used to determine how the
RecordPath is resolved. " +
+"The following variables are availble: ${field.name},
${field.value}, ${field.type}",
+description = "The name of each user-defined property must be a valid
RecordPath.")
+@SeeAlso(classNames = {
+
"org.a
[GitHub] [nifi] mattyb149 commented on a change in pull request #3317: NIFI-6047 Add DetectDuplicateRecord Processor
mattyb149 commented on a change in pull request #3317: NIFI-6047 Add
DetectDuplicateRecord Processor
URL: https://github.com/apache/nifi/pull/3317#discussion_r296306737
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DetectDuplicateRecord.java
##
@@ -0,0 +1,646 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import com.google.common.base.Joiner;
+import com.google.common.hash.BloomFilter;
+import com.google.common.hash.Funnels;
+import org.apache.commons.codec.binary.Hex;
+import org.apache.commons.codec.digest.DigestUtils;
+import org.apache.commons.codec.digest.MessageDigestAlgorithms;
+import org.apache.nifi.annotation.behavior.*;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.*;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import
org.apache.nifi.distributed.cache.client.exception.DeserializationException;
+import
org.apache.nifi.distributed.cache.client.exception.SerializationException;
+import org.apache.nifi.expression.AttributeExpression.ResultType;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.*;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathPropertyNameValidator;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.serialization.*;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.util.DataTypeUtils;
+
+import java.io.*;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.util.*;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+import static java.util.stream.Collectors.toList;
+import static org.apache.commons.codec.binary.StringUtils.getBytesUtf8;
+import static org.apache.commons.lang3.StringUtils.*;
+
+@EventDriven
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@SystemResourceConsideration(resource = SystemResource.MEMORY,
+description = "The HashSet filter type will grow memory space
proportionate to the number of unique records processed. " +
+"The BloomFilter type will use constant memory regardless of the
number of records processed.")
+@Tags({"text", "record", "update", "change", "replace", "modify", "distinct",
"unique",
+"filter", "hash", "dupe", "duplicate", "dedupe"})
+@CapabilityDescription("Caches records from each incoming FlowFile and
determines if the record " +
+"has already been seen. If so, routes the record to 'duplicate'. If the
record is " +
+"not determined to be a duplicate, it is routed to 'non-duplicate'."
+)
+@WritesAttribute(attribute = "record.count", description = "The number of
records processed.")
+@DynamicProperty(
+name = "RecordPath",
+value = "An expression language statement used to determine how the
RecordPath is resolved. " +
+"The following variables are availble: ${field.name},
${field.value}, ${field.type}",
+description = "The name of each user-defined property must be a valid
RecordPath.")
+@SeeAlso(classNames = {
+
"org.a
[GitHub] [nifi] mattyb149 commented on a change in pull request #3317: NIFI-6047 Add DetectDuplicateRecord Processor
mattyb149 commented on a change in pull request #3317: NIFI-6047 Add
DetectDuplicateRecord Processor
URL: https://github.com/apache/nifi/pull/3317#discussion_r296316553
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestDetectDuplicateRecord.java
##
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.serialization.record.MockRecordParser;
+import org.apache.nifi.serialization.record.MockRecordWriter;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.util.*;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.nifi.processors.standard.DetectDuplicateRecord.*;
+import static org.junit.Assert.assertEquals;
+
+public class TestDetectDuplicateRecord {
+
+private TestRunner runner;
+private MockCacheService cache;
+private MockRecordParser reader;
+private MockRecordWriter writer;
+
+@BeforeClass
+public static void beforeClass() {
+System.setProperty("org.slf4j.simpleLogger.defaultLogLevel", "info");
+System.setProperty("org.slf4j.simpleLogger.showDateTime", "true");
+System.setProperty("org.slf4j.simpleLogger.log.nifi.io.nio", "debug");
+
System.setProperty("org.slf4j.simpleLogger.log.nifi.processors.standard.DetectDuplicateRecord",
"debug");
+
System.setProperty("org.slf4j.simpleLogger.log.nifi.processors.standard.TestDetectDuplicateRecord",
"debug");
+}
+
+@Before
+public void setup() throws InitializationException {
+runner = TestRunners.newTestRunner(DetectDuplicateRecord.class);
+
+// RECORD_READER, RECORD_WRITER
+reader = new MockRecordParser();
+writer = new MockRecordWriter("header", false);
+
+runner.addControllerService("reader", reader);
+runner.enableControllerService(reader);
+runner.addControllerService("writer", writer);
+runner.enableControllerService(writer);
+
+runner.setProperty(RECORD_READER, "reader");
+runner.setProperty(RECORD_WRITER, "writer");
+
+reader.addSchemaField("firstName", RecordFieldType.STRING);
+reader.addSchemaField("middleName", RecordFieldType.STRING);
+reader.addSchemaField("lastName", RecordFieldType.STRING);
+
+// INCLUDE_ZERO_RECORD_FLOWFILES
+runner.setProperty(INCLUDE_ZERO_RECORD_FLOWFILES, "true");
+
+// CACHE_IDENTIFIER
+runner.setProperty(CACHE_IDENTIFIER, "true");
+
+// DISTRIBUTED_CACHE_SERVICE
+cache = new MockCacheService();
+runner.addControllerService("cache", cache);
+runner.setProperty(DISTRIBUTED_CACHE_SERVICE, "cache");
+runner.enableControllerService(cache);
+
+// CACHE_ENTRY_IDENTIFIER
+final Map props = new HashMap<>();
+props.put("hash.value", "1000");
+runner.enqueue(new byte[]{}, props);
+
+// AGE_OFF_DURATION
+runner.setProperty(AGE_OFF_DURATION, "48 hours");
+
+runner.assertValid();
+}
+
+ @Test
+ public void testDetectDuplicatesHashSet() {
+runner.setProperty(FILTER_TYPE, HASH_SET_VALUE);
+runner.setProperty("/middleName", "${field.value}");
+reader.addRecord("John", "Q", "Smith");
+reader.addRecord("John", "Q", "Smith");
+reader.addRecord("Jane", "X", "Doe");
+
+runner.enqueue("");
+runner.run();
+
+doCountTests(0, 1, 1, 1, 2, 1);
+}
+
+@Test
+public void testDetectDuplicatesBloomFilter() {
+runner.setProperty(FILTER_TYPE, BLOOM_FILTER_VALUE);
+runner.setProperty(BLOOM_FILTER_FPP, "0.10");
+runner.setProperty("/middleName", "${field.value}");
+reader.addRecord("John", "Q", "Smith");
+reader.addRecord("John", "Q", "Smith");
+reader.addRecord("Jane", "X", "Doe");
+
+runner.enqueue("");
+runner.run();
+
+doCountTests(0
[GitHub] [nifi] mattyb149 commented on a change in pull request #3317: NIFI-6047 Add DetectDuplicateRecord Processor
mattyb149 commented on a change in pull request #3317: NIFI-6047 Add
DetectDuplicateRecord Processor
URL: https://github.com/apache/nifi/pull/3317#discussion_r296314665
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DetectDuplicateRecord.java
##
@@ -0,0 +1,646 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import com.google.common.base.Joiner;
+import com.google.common.hash.BloomFilter;
+import com.google.common.hash.Funnels;
+import org.apache.commons.codec.binary.Hex;
+import org.apache.commons.codec.digest.DigestUtils;
+import org.apache.commons.codec.digest.MessageDigestAlgorithms;
+import org.apache.nifi.annotation.behavior.*;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.*;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import
org.apache.nifi.distributed.cache.client.exception.DeserializationException;
+import
org.apache.nifi.distributed.cache.client.exception.SerializationException;
+import org.apache.nifi.expression.AttributeExpression.ResultType;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.*;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathPropertyNameValidator;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.serialization.*;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.util.DataTypeUtils;
+
+import java.io.*;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.util.*;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+import static java.util.stream.Collectors.toList;
+import static org.apache.commons.codec.binary.StringUtils.getBytesUtf8;
+import static org.apache.commons.lang3.StringUtils.*;
+
+@EventDriven
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@SystemResourceConsideration(resource = SystemResource.MEMORY,
+description = "The HashSet filter type will grow memory space
proportionate to the number of unique records processed. " +
+"The BloomFilter type will use constant memory regardless of the
number of records processed.")
+@Tags({"text", "record", "update", "change", "replace", "modify", "distinct",
"unique",
+"filter", "hash", "dupe", "duplicate", "dedupe"})
+@CapabilityDescription("Caches records from each incoming FlowFile and
determines if the record " +
+"has already been seen. If so, routes the record to 'duplicate'. If the
record is " +
+"not determined to be a duplicate, it is routed to 'non-duplicate'."
+)
+@WritesAttribute(attribute = "record.count", description = "The number of
records processed.")
+@DynamicProperty(
+name = "RecordPath",
+value = "An expression language statement used to determine how the
RecordPath is resolved. " +
+"The following variables are availble: ${field.name},
${field.value}, ${field.type}",
+description = "The name of each user-defined property must be a valid
RecordPath.")
+@SeeAlso(classNames = {
+
"org.a
[GitHub] [nifi] mattyb149 commented on a change in pull request #3317: NIFI-6047 Add DetectDuplicateRecord Processor
mattyb149 commented on a change in pull request #3317: NIFI-6047 Add
DetectDuplicateRecord Processor
URL: https://github.com/apache/nifi/pull/3317#discussion_r296316843
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestDetectDuplicateRecord.java
##
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.serialization.record.MockRecordParser;
+import org.apache.nifi.serialization.record.MockRecordWriter;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.util.*;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.nifi.processors.standard.DetectDuplicateRecord.*;
+import static org.junit.Assert.assertEquals;
+
+public class TestDetectDuplicateRecord {
+
+private TestRunner runner;
+private MockCacheService cache;
+private MockRecordParser reader;
+private MockRecordWriter writer;
+
+@BeforeClass
+public static void beforeClass() {
+System.setProperty("org.slf4j.simpleLogger.defaultLogLevel", "info");
+System.setProperty("org.slf4j.simpleLogger.showDateTime", "true");
+System.setProperty("org.slf4j.simpleLogger.log.nifi.io.nio", "debug");
+
System.setProperty("org.slf4j.simpleLogger.log.nifi.processors.standard.DetectDuplicateRecord",
"debug");
+
System.setProperty("org.slf4j.simpleLogger.log.nifi.processors.standard.TestDetectDuplicateRecord",
"debug");
+}
+
+@Before
+public void setup() throws InitializationException {
+runner = TestRunners.newTestRunner(DetectDuplicateRecord.class);
+
+// RECORD_READER, RECORD_WRITER
+reader = new MockRecordParser();
+writer = new MockRecordWriter("header", false);
+
+runner.addControllerService("reader", reader);
+runner.enableControllerService(reader);
+runner.addControllerService("writer", writer);
+runner.enableControllerService(writer);
+
+runner.setProperty(RECORD_READER, "reader");
+runner.setProperty(RECORD_WRITER, "writer");
+
+reader.addSchemaField("firstName", RecordFieldType.STRING);
+reader.addSchemaField("middleName", RecordFieldType.STRING);
+reader.addSchemaField("lastName", RecordFieldType.STRING);
+
+// INCLUDE_ZERO_RECORD_FLOWFILES
+runner.setProperty(INCLUDE_ZERO_RECORD_FLOWFILES, "true");
+
+// CACHE_IDENTIFIER
+runner.setProperty(CACHE_IDENTIFIER, "true");
+
+// DISTRIBUTED_CACHE_SERVICE
+cache = new MockCacheService();
+runner.addControllerService("cache", cache);
+runner.setProperty(DISTRIBUTED_CACHE_SERVICE, "cache");
+runner.enableControllerService(cache);
+
+// CACHE_ENTRY_IDENTIFIER
+final Map props = new HashMap<>();
+props.put("hash.value", "1000");
+runner.enqueue(new byte[]{}, props);
+
+// AGE_OFF_DURATION
+runner.setProperty(AGE_OFF_DURATION, "48 hours");
+
+runner.assertValid();
+}
+
+ @Test
+ public void testDetectDuplicatesHashSet() {
+runner.setProperty(FILTER_TYPE, HASH_SET_VALUE);
+runner.setProperty("/middleName", "${field.value}");
+reader.addRecord("John", "Q", "Smith");
+reader.addRecord("John", "Q", "Smith");
+reader.addRecord("Jane", "X", "Doe");
+
+runner.enqueue("");
+runner.run();
+
+doCountTests(0, 1, 1, 1, 2, 1);
+}
+
+@Test
+public void testDetectDuplicatesBloomFilter() {
+runner.setProperty(FILTER_TYPE, BLOOM_FILTER_VALUE);
+runner.setProperty(BLOOM_FILTER_FPP, "0.10");
+runner.setProperty("/middleName", "${field.value}");
+reader.addRecord("John", "Q", "Smith");
+reader.addRecord("John", "Q", "Smith");
+reader.addRecord("Jane", "X", "Doe");
+
+runner.enqueue("");
+runner.run();
+
+doCountTests(0
[GitHub] [nifi] mattyb149 commented on a change in pull request #3317: NIFI-6047 Add DetectDuplicateRecord Processor
mattyb149 commented on a change in pull request #3317: NIFI-6047 Add
DetectDuplicateRecord Processor
URL: https://github.com/apache/nifi/pull/3317#discussion_r296306386
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DetectDuplicateRecord.java
##
@@ -0,0 +1,646 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import com.google.common.base.Joiner;
+import com.google.common.hash.BloomFilter;
+import com.google.common.hash.Funnels;
+import org.apache.commons.codec.binary.Hex;
+import org.apache.commons.codec.digest.DigestUtils;
+import org.apache.commons.codec.digest.MessageDigestAlgorithms;
+import org.apache.nifi.annotation.behavior.*;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.*;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import
org.apache.nifi.distributed.cache.client.exception.DeserializationException;
+import
org.apache.nifi.distributed.cache.client.exception.SerializationException;
+import org.apache.nifi.expression.AttributeExpression.ResultType;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.*;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathPropertyNameValidator;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.serialization.*;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.util.DataTypeUtils;
+
+import java.io.*;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.util.*;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+import static java.util.stream.Collectors.toList;
+import static org.apache.commons.codec.binary.StringUtils.getBytesUtf8;
+import static org.apache.commons.lang3.StringUtils.*;
+
+@EventDriven
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@SystemResourceConsideration(resource = SystemResource.MEMORY,
+description = "The HashSet filter type will grow memory space
proportionate to the number of unique records processed. " +
+"The BloomFilter type will use constant memory regardless of the
number of records processed.")
+@Tags({"text", "record", "update", "change", "replace", "modify", "distinct",
"unique",
+"filter", "hash", "dupe", "duplicate", "dedupe"})
+@CapabilityDescription("Caches records from each incoming FlowFile and
determines if the record " +
+"has already been seen. If so, routes the record to 'duplicate'. If the
record is " +
+"not determined to be a duplicate, it is routed to 'non-duplicate'."
+)
+@WritesAttribute(attribute = "record.count", description = "The number of
records processed.")
+@DynamicProperty(
+name = "RecordPath",
+value = "An expression language statement used to determine how the
RecordPath is resolved. " +
+"The following variables are availble: ${field.name},
${field.value}, ${field.type}",
+description = "The name of each user-defined property must be a valid
RecordPath.")
+@SeeAlso(classNames = {
+
"org.a
[GitHub] [nifi] mattyb149 commented on a change in pull request #3317: NIFI-6047 Add DetectDuplicateRecord Processor
mattyb149 commented on a change in pull request #3317: NIFI-6047 Add
DetectDuplicateRecord Processor
URL: https://github.com/apache/nifi/pull/3317#discussion_r296309936
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DetectDuplicateRecord.java
##
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import com.google.common.base.Joiner;
+import com.google.common.hash.BloomFilter;
+import com.google.common.hash.Funnels;
+import org.apache.commons.codec.binary.Hex;
+import org.apache.commons.codec.digest.DigestUtils;
+import org.apache.commons.codec.digest.MessageDigestAlgorithms;
+import org.apache.nifi.annotation.behavior.*;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.*;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import
org.apache.nifi.distributed.cache.client.exception.DeserializationException;
+import
org.apache.nifi.distributed.cache.client.exception.SerializationException;
+import org.apache.nifi.expression.AttributeExpression.ResultType;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.*;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathPropertyNameValidator;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.serialization.*;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import java.io.*;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.util.*;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+import static java.util.stream.Collectors.toList;
+import static org.apache.commons.codec.binary.StringUtils.getBytesUtf8;
+import static org.apache.commons.lang3.StringUtils.*;
+
+@EventDriven
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@SystemResourceConsideration(resource = SystemResource.MEMORY,
+description = "Caches records from each incoming FlowFile and
determines if the cached record has " +
+"already been seen. The name of user-defined properties
determines the RecordPath values used to " +
+"determine if a record is unique. If no user-defined
properties are present, the entire record is " +
+"used as the input to determine uniqueness. All duplicate
records are routed to 'duplicate'. " +
+"If the record is not determined to be a duplicate, the
Processor routes the record to 'non-duplicate'.")
+@Tags({"text", "record", "update", "change", "replace", "modify", "distinct",
"unique",
+"filter", "hash", "dupe", "duplicate", "dedupe"})
+@CapabilityDescription("Caches records from each incoming FlowFile and
determines if the cached record has " +
+"already been seen. The name of user-defined properties determines the
RecordPath values used to " +
+"determine if a record is unique. If no user-defined properties are
present, the entire record is " +
+"used as the input to determine uniqueness. All duplicate records are
routed to 'duplicate'. " +
+
[jira] [Updated] (NIFI-6028) Upgrading NiFi can put versioned flows into a conflict state
[ https://issues.apache.org/jira/browse/NIFI-6028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-6028: - Status: Patch Available (was: Open) > Upgrading NiFi can put versioned flows into a conflict state > > > Key: NIFI-6028 > URL: https://issues.apache.org/jira/browse/NIFI-6028 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.7.1, 1.8.0, 1.7.0, 1.6.0, 1.5.0 >Reporter: Bryan Bende >Priority: Major > Labels: SDLC > Time Spent: 10m > Remaining Estimate: 0h > > When you upgrade NiFi, existing processors may have new properties and > relationships. If any of those processors are part of a versioned flow then > these changes will trigger a local modification indicating that a new version > needs to be saved to registry to track the new properties or relationships. > The issue is when you have multiple environments... > * Start in dev with NiFi version X > * Upgrade dev to NiFi version Y > * Now commit versioned PGs in dev that had local changes > * Go to staging and upgrade NiFi to version Y > * The versioned PGs are now in conflict because there is an upgrade > available, but there are local changes detected from upgrading NiFI, even > though these are the same changes available in the upgrade -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-6028) Upgrading NiFi can put versioned flows into a conflict state
[ https://issues.apache.org/jira/browse/NIFI-6028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne reassigned NIFI-6028: Assignee: Mark Payne > Upgrading NiFi can put versioned flows into a conflict state > > > Key: NIFI-6028 > URL: https://issues.apache.org/jira/browse/NIFI-6028 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.7.1 >Reporter: Bryan Bende >Assignee: Mark Payne >Priority: Major > Labels: SDLC > Time Spent: 10m > Remaining Estimate: 0h > > When you upgrade NiFi, existing processors may have new properties and > relationships. If any of those processors are part of a versioned flow then > these changes will trigger a local modification indicating that a new version > needs to be saved to registry to track the new properties or relationships. > The issue is when you have multiple environments... > * Start in dev with NiFi version X > * Upgrade dev to NiFi version Y > * Now commit versioned PGs in dev that had local changes > * Go to staging and upgrade NiFi to version Y > * The versioned PGs are now in conflict because there is an upgrade > available, but there are local changes detected from upgrading NiFI, even > though these are the same changes available in the upgrade -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-6028) Upgrading NiFi can put versioned flows into a conflict state
[ https://issues.apache.org/jira/browse/NIFI-6028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-6028: - Fix Version/s: 1.10.0 > Upgrading NiFi can put versioned flows into a conflict state > > > Key: NIFI-6028 > URL: https://issues.apache.org/jira/browse/NIFI-6028 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.7.1 >Reporter: Bryan Bende >Assignee: Mark Payne >Priority: Major > Labels: SDLC > Fix For: 1.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > > When you upgrade NiFi, existing processors may have new properties and > relationships. If any of those processors are part of a versioned flow then > these changes will trigger a local modification indicating that a new version > needs to be saved to registry to track the new properties or relationships. > The issue is when you have multiple environments... > * Start in dev with NiFi version X > * Upgrade dev to NiFi version Y > * Now commit versioned PGs in dev that had local changes > * Go to staging and upgrade NiFi to version Y > * The versioned PGs are now in conflict because there is an upgrade > available, but there are local changes detected from upgrading NiFI, even > though these are the same changes available in the upgrade -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [nifi] markap14 opened a new pull request #3544: NIFI-6028: Updates to ignore changes between local version of a flow …
markap14 opened a new pull request #3544: NIFI-6028: Updates to ignore changes
between local version of a flow …
URL: https://github.com/apache/nifi/pull/3544
…and a remote version of a flow if the difference is the addition of a new
property that has the default value
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
_Enables X functionality; fixes bug NIFI-._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically `master`)?
- [ ] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi-registry] markap14 opened a new pull request #201: NIFIREG-287: For Flow Difference objects of type PROPERTY_MODIFIED, P…
markap14 opened a new pull request #201: NIFIREG-287: For Flow Difference objects of type PROPERTY_MODIFIED, P… URL: https://github.com/apache/nifi-registry/pull/201 …ROPERTY_ADDED, PROPERTY_REMOVED, include the name of hte property as the fieldName. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (NIFIREG-287) FlowDifference object should include property name when a property is added/removed
Mark Payne created NIFIREG-287: -- Summary: FlowDifference object should include property name when a property is added/removed Key: NIFIREG-287 URL: https://issues.apache.org/jira/browse/NIFIREG-287 Project: NiFi Registry Issue Type: Bug Reporter: Mark Payne Assignee: Mark Payne When a FlowDifference object is created, it does not populate the 'fieldName' of the FlowDifference. This means that consumers of the FlowDifference object are unable to determine which property was added/removed/modified -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [nifi-minifi-cpp] phrocker commented on issue #590: MINIFICPP-621 Nanofi Tailfile example
phrocker commented on issue #590: MINIFICPP-621 Nanofi Tailfile example URL: https://github.com/apache/nifi-minifi-cpp/pull/590#issuecomment-504439214 @bakaid Are you good with merging? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (MINIFICPP-919) Site2Site transaction CRC checksum randomly fails to match on Windows
[
https://issues.apache.org/jira/browse/MINIFICPP-919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16869498#comment-16869498
]
Mr TheSegfault commented on MINIFICPP-919:
--
Despite this being closed I found with unit tests on windows another place
where this occurs. Fixing as a result of that PR to get unit tests working on
windows, but will place this comment here for tracking purposes.
> Site2Site transaction CRC checksum randomly fails to match on Windows
> -
>
> Key: MINIFICPP-919
> URL: https://issues.apache.org/jira/browse/MINIFICPP-919
> Project: Apache NiFi MiNiFi C++
> Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Arpad Boda
>Assignee: Arpad Boda
>Priority: Blocker
> Fix For: 0.7.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {code}
> [2019-04-30 00:33:19.763] [class
> org::apache::nifi::minifi::sitetosite::SiteToSiteClient] [debug] Site2Site
> transaction 39f64f06-6b1a-11e9-ad7b-6f7078e78cba peer confirm transaction
> with CRC 2402037639
> [2019-04-30 00:33:19.763] [class
> org::apache::nifi::minifi::sitetosite::SiteToSiteClient] [debug] Site2Site
> transaction 39f64f06-6b1a-11e9-ad7b-6f7078e78cba CRC not matched -1892929657
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [nifi] aramatev commented on issue #3540: NIFI-6385 Added signal.id penalization
aramatev commented on issue #3540: NIFI-6385 Added signal.id penalization
URL: https://github.com/apache/nifi/pull/3540#issuecomment-504428325
@ijokarumawak thank you for explaining, with the following I am able to get
the performance I desire.
Run Schedule = 0 sec
Wait Penalize Duration = 1 sec
Release Signal Identifier = ${uuid}
From logging I could see (even on my laptop), its able to loop through about
450+ flowfiles in a second, or rather. Before my wait penalty was too short -
at 1ms, so onTrigger() was not able to see the rest of the flowfiles.
Thank you for your work!
@markap14 The commit looks good, when can we expect this in the product?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (NIFI-6390) Avoid assigning Content Claim to FlowFile if claim has no data
Mark Payne created NIFI-6390: Summary: Avoid assigning Content Claim to FlowFile if claim has no data Key: NIFI-6390 URL: https://issues.apache.org/jira/browse/NIFI-6390 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Mark Payne Assignee: Mark Payne When a Processor writes to a FlowFile, it creates a Content Claim to write to, and this Content Claim is assigned a Resource Claim (a file on disk in the case of the FileSystemRepository). If no data is then written to the FlowFile, the content claim remains. This means that the FlowFile is now holding a reference to a Resource Claim that it doesn't really need. Unfortunately, this can potentially keep data in the Content Repository longer than it needs to. Take, for example, the case where a FlowFile is created and 0 bytes are written to it. Then, a Processor writes 10 MB to another FlowFile. If these two FlowFIles hold claims to the same file on disk, then even after the second FlowFile is destroyed, that 10 MB claim must stay around until the 0-byte FlowFile is also destroyed, because it holds the claim. Instead, the framework should detect that no data was written to the FlowFile and destroy the content claim & decrement the claimant count on the Resource Claim. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-6389) Status History panel doesn't retain selected time interval when switched between different metrics
Purushotham Pushpavanthar created NIFI-6389: --- Summary: Status History panel doesn't retain selected time interval when switched between different metrics Key: NIFI-6389 URL: https://issues.apache.org/jira/browse/NIFI-6389 Project: Apache NiFi Issue Type: Improvement Components: Core UI Affects Versions: 1.9.2 Reporter: Purushotham Pushpavanthar The Status History has two charts - one for 24 HR summary and another for details. One can select portion of a chart to view its details in detailed chart. However when specific portion/time interval is selected on one metric and then switch to another metric the selected portion doesn't apply in the newly selected metric. It will be very convenient if the selected portion is retained when there is a switch. I've attached screenshots: In the first image I'm interested in *Bytes Transferred* between 9.20 and 10.00 (This is a batch job which runs around this time daily). Where as in second image, when I switch to *FlowFiles Out* metric, the previous selection vanishes. I had to select the region again which is a pain. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [nifi] ijokarumawak commented on issue #3540: NIFI-6385 Added signal.id penalization
ijokarumawak commented on issue #3540: NIFI-6385 Added signal.id penalization URL: https://github.com/apache/nifi/pull/3540#issuecomment-504343120 @aramatev I've added additional details documentation. Please check to find if that is helpful. @markap14 I've closed the previous PR, and addressed the issue with a different approach in this PR. Would you be able to review this one? Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] kaHaleMaKai opened a new pull request #3543: NIFI-6388 Add dynamic relationships to the ExecuteScript processor.
kaHaleMaKai opened a new pull request #3543: NIFI-6388 Add dynamic
relationships to the ExecuteScript processor.
URL: https://github.com/apache/nifi/pull/3543
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
Add dynamic relationships to the ExecuteScriptProcessor.
The previous relationships `SUCCESS` and `FAILURE` are kept for backwards
compability. The variable of type `Map` `rel` is
introduced. If the processor has a dynamic property with name like
`rel.my_new_relationship`, a relationship of name `my_new_relationship` is
added.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ x ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ x ] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ x ] Has your PR been rebased against the latest commit within the target
branch (typically `master`)?
- [ x ] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ x ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [ x ] Have you written or updated unit tests to verify your changes?
- [ x ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ x ] If applicable, have you updated the `LICENSE` file, including the
main `LICENSE` file under `nifi-assembly`?
- [ x ] If applicable, have you updated the `NOTICE` file, including the
main `NOTICE` file found under `nifi-assembly`?
- [ x ] If adding new Properties, have you added `.displayName` in addition
to .name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ x ] Have you ensured that format looks appropriate for the output in
which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
