[jira] [Updated] (NIFI-5952) RAW Site-to-Site fails with java.nio.channels.IllegalBlockingModeException
[
https://issues.apache.org/jira/browse/NIFI-5952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Kawamura updated NIFI-5952:
Priority: Blocker (was: Critical)
> RAW Site-to-Site fails with java.nio.channels.IllegalBlockingModeException
> --
>
> Key: NIFI-5952
> URL: https://issues.apache.org/jira/browse/NIFI-5952
> Project: Apache NiFi
> Issue Type: Sub-task
> Components: Core Framework
> Environment: jdk-11.0.1
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>Priority: Blocker
> Labels: Java11
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> During the review cycle of NIFI-5820, I found that while HTTP S2S works
> without issue, RAW S2S is failing with following Exception:
> {code:java}
> 2018-12-19 16:19:26,811 ERROR [Site-to-Site Listener] org.apache.nifi.NiFi
> java.nio.channels.IllegalBlockingModeException: null
> at
> java.base/sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:121)
> at
> org.apache.nifi.remote.SocketRemoteSiteListener$1.run(SocketRemoteSiteListener.java:125)
> at java.base/java.lang.Thread.run(Thread.java:834)
> {code}
> Despite of the fact that the RAW has been worked with older Java versions, it
> seems current nio usage at RAW S2S is not correct. And JDK 11 starts
> complaining about it.
> Here are few things I've discovered with current NiFi and nio SocketChannel:
> - NiFi accepts RAW S2S client connection with SocketRemoteSiteListener,
> which uses ServerSocketChannel as non-blocking manner [1]
> - But SocketRemoteSiteListener doesn't use Selector API to accept incoming
> connection and transfer data with the channel. This is the cause of above
> exception.
> - SocketRemoteSiteListener spawns new thread when it accepts connection.
> This is how connections are handled with a non-nio, standard Socket
> programming. If we want to use non-blocking NIO, we need to use channels with
> Selector
> - But using non-blocking IO with current NiFi S2S protocol can only add few
> or none benefit by doing so. [2]
> To make RAW S2S work with Java 11, we need either:
> A. Stop using nio packages.
> B. Implement correct nio usage, meaning use Selector IO and probably we need
> another thread pool.
> I'm going to take the approach A above, because B would take much more
> refactoring.
> [1]
> [https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-site-to-site/src/main/java/org/apache/nifi/remote/SocketRemoteSiteListener.java#L120]
> [2]
> [https://stackoverflow.com/questions/12338204/in-java-nio-is-a-selector-useful-for-a-client-socketchannel]
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [nifi] natural commented on a change in pull request #3672: NIFI-6363 Additional Sensitive Property Providers
natural commented on a change in pull request #3672: NIFI-6363 Additional
Sensitive Property Providers
URL: https://github.com/apache/nifi/pull/3672#discussion_r319322547
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/test/groovy/org/apache/nifi/properties/sensitive/aes/AESSensitivePropertyProviderTest.groovy
##
@@ -381,6 +389,7 @@ class AESSensitivePropertyProviderTest extends
GroovyTestCase {
}
}
+@Ignore // for now bc size is gone
Review comment:
Related to the erroneous change to the `aes/gcm/{suffix}` format, reverted.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] natural commented on a change in pull request #3672: NIFI-6363 Additional Sensitive Property Providers
natural commented on a change in pull request #3672: NIFI-6363 Additional
Sensitive Property Providers
URL: https://github.com/apache/nifi/pull/3672#discussion_r319322208
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/keystore/KeyStoreSensitivePropertyProvider.java
##
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties.sensitive.keystore;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.properties.sensitive.ExternalProperties;
+import
org.apache.nifi.properties.sensitive.SensitivePropertyConfigurationException;
+import
org.apache.nifi.properties.sensitive.SensitivePropertyProtectionException;
+import org.apache.nifi.properties.sensitive.SensitivePropertyProvider;
+import org.apache.nifi.properties.sensitive.StandardExternalPropertyLookup;
+import org.apache.nifi.properties.sensitive.aes.AESSensitivePropertyProvider;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.security.Key;
+import java.security.KeyStore;
+import java.security.KeyStoreException;
+import java.security.NoSuchAlgorithmException;
+import java.security.UnrecoverableKeyException;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Sensitive properties using KeyStore keys with an inner AES SPP.
+ */
+public class KeyStoreSensitivePropertyProvider implements
SensitivePropertyProvider {
Review comment:
Renamed to match intent.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] natural commented on a change in pull request #3672: NIFI-6363 Additional Sensitive Property Providers
natural commented on a change in pull request #3672: NIFI-6363 Additional Sensitive Property Providers URL: https://github.com/apache/nifi/pull/3672#discussion_r319321910 ## File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/keystore/ByteArrayKeyStoreProvider.java ## @@ -0,0 +1,57 @@ +/* Review comment: Moved. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] natural commented on a change in pull request #3672: NIFI-6363 Additional Sensitive Property Providers
natural commented on a change in pull request #3672: NIFI-6363 Additional
Sensitive Property Providers
URL: https://github.com/apache/nifi/pull/3672#discussion_r319321775
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/gcp/kms/GCPKMSSensitivePropertyProvider.java
##
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties.sensitive.gcp.kms;
+
+import com.google.cloud.kms.v1.CryptoKeyName;
+import com.google.cloud.kms.v1.DecryptResponse;
+import com.google.cloud.kms.v1.EncryptResponse;
+import com.google.cloud.kms.v1.KeyManagementServiceClient;
+import com.google.protobuf.ByteString;
+import org.apache.commons.lang3.StringUtils;
+import
org.apache.nifi.properties.sensitive.SensitivePropertyConfigurationException;
+import
org.apache.nifi.properties.sensitive.SensitivePropertyProtectionException;
+import org.apache.nifi.properties.sensitive.SensitivePropertyProvider;
+import org.bouncycastle.util.encoders.Base64;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * This provider uses the GCP SDK to interact with the GCP KMS. Values are
encoded/decoded base64, using the
+ * standard encoders from bouncycastle.
+ */
+public class GCPKMSSensitivePropertyProvider implements
SensitivePropertyProvider {
+private static final Logger logger =
LoggerFactory.getLogger(GCPKMSSensitivePropertyProvider.class);
+private static final String IMPLEMENTATION_NAME = "GCP KMS Sensitive
Property Provider";
+
+private static final String MATERIAL_PREFIX = "gcp";
+private static final String MATERIAL_KEY_TYPE = "kms";
+private static final String MATERIAL_SEPARATOR = "/";
+private static final String IMPLEMENTATION_PREFIX = MATERIAL_PREFIX +
MATERIAL_SEPARATOR + MATERIAL_KEY_TYPE + MATERIAL_SEPARATOR;
+
+private String keyId;
+private String projectId;
+private String locationId;
+private String keyRingId;
+private String cryptoKeyId;
+
+private final String resource;
+private final KeyManagementServiceClient client;
+
+
+public GCPKMSSensitivePropertyProvider(String keyId) {
+if
(StringUtils.isBlank(System.getenv("GOOGLE_APPLICATION_CREDENTIALS"))) {
+throw new SensitivePropertyConfigurationException("Unable to find
Google Application Credentials");
+}
+extractKeyParts(keyId);
+this.resource = CryptoKeyName.format(projectId, locationId, keyRingId,
cryptoKeyId);
+try {
+this.client = KeyManagementServiceClient.create();
+} catch (IOException e) {
+throw new SensitivePropertyConfigurationException("Unable to
create service client", e);
+}
+}
+
+private void extractKeyParts(String keyId) {
+if (StringUtils.isBlank(keyId)) throw new
SensitivePropertyConfigurationException("The key cannot be empty");
+
+this.keyId = keyId;
+
+String[] parts = this.keyId.split(MATERIAL_SEPARATOR, 3);
+if (parts.length != 3 || !parts[0].equals(MATERIAL_PREFIX) ||
!parts[1].equals(MATERIAL_KEY_TYPE))
+throw new SensitivePropertyConfigurationException("Invalid GCP
key");
+
+String path = parts[2];
+Pattern simplePattern =
Pattern.compile("([^/]+)/([^/]+)/([^/]+)/([^/]+)");
+Pattern verbosePattern =
Pattern.compile("projects/([^/]+)/locations/([^/]+)/keyRings/([^/]+)/cryptoKeys/([^/]+)");
+
+Matcher match = null;
+if (verbosePattern.asPredicate().test(path)) {
+match = verbosePattern.matcher(path);
+} else if (simplePattern.asPredicate().test(path)) {
+match = simplePattern.matcher(path);
+}
+
+if (match == null || match.groupCount() != 4 || !match.find())
+throw new SensitivePropertyConfigurationException("Invalid GCP key
pattern");
+
+projectId = match.group(1);
+locationId = match.group(2);
+keyRingId = match.group(3);
+cryptoKeyId =
[GitHub] [nifi] natural commented on a change in pull request #3672: NIFI-6363 Additional Sensitive Property Providers
natural commented on a change in pull request #3672: NIFI-6363 Additional
Sensitive Property Providers
URL: https://github.com/apache/nifi/pull/3672#discussion_r319321714
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/gcp/kms/GCPKMSSensitivePropertyProvider.java
##
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties.sensitive.gcp.kms;
+
+import com.google.cloud.kms.v1.CryptoKeyName;
+import com.google.cloud.kms.v1.DecryptResponse;
+import com.google.cloud.kms.v1.EncryptResponse;
+import com.google.cloud.kms.v1.KeyManagementServiceClient;
+import com.google.protobuf.ByteString;
+import org.apache.commons.lang3.StringUtils;
+import
org.apache.nifi.properties.sensitive.SensitivePropertyConfigurationException;
+import
org.apache.nifi.properties.sensitive.SensitivePropertyProtectionException;
+import org.apache.nifi.properties.sensitive.SensitivePropertyProvider;
+import org.bouncycastle.util.encoders.Base64;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * This provider uses the GCP SDK to interact with the GCP KMS. Values are
encoded/decoded base64, using the
+ * standard encoders from bouncycastle.
+ */
+public class GCPKMSSensitivePropertyProvider implements
SensitivePropertyProvider {
+private static final Logger logger =
LoggerFactory.getLogger(GCPKMSSensitivePropertyProvider.class);
+private static final String IMPLEMENTATION_NAME = "GCP KMS Sensitive
Property Provider";
+
+private static final String MATERIAL_PREFIX = "gcp";
+private static final String MATERIAL_KEY_TYPE = "kms";
+private static final String MATERIAL_SEPARATOR = "/";
+private static final String IMPLEMENTATION_PREFIX = MATERIAL_PREFIX +
MATERIAL_SEPARATOR + MATERIAL_KEY_TYPE + MATERIAL_SEPARATOR;
+
+private String keyId;
+private String projectId;
+private String locationId;
+private String keyRingId;
+private String cryptoKeyId;
+
+private final String resource;
+private final KeyManagementServiceClient client;
+
+
+public GCPKMSSensitivePropertyProvider(String keyId) {
+if
(StringUtils.isBlank(System.getenv("GOOGLE_APPLICATION_CREDENTIALS"))) {
+throw new SensitivePropertyConfigurationException("Unable to find
Google Application Credentials");
+}
+extractKeyParts(keyId);
+this.resource = CryptoKeyName.format(projectId, locationId, keyRingId,
cryptoKeyId);
+try {
+this.client = KeyManagementServiceClient.create();
+} catch (IOException e) {
+throw new SensitivePropertyConfigurationException("Unable to
create service client", e);
+}
+}
+
+private void extractKeyParts(String keyId) {
+if (StringUtils.isBlank(keyId)) throw new
SensitivePropertyConfigurationException("The key cannot be empty");
+
+this.keyId = keyId;
+
+String[] parts = this.keyId.split(MATERIAL_SEPARATOR, 3);
+if (parts.length != 3 || !parts[0].equals(MATERIAL_PREFIX) ||
!parts[1].equals(MATERIAL_KEY_TYPE))
+throw new SensitivePropertyConfigurationException("Invalid GCP
key");
+
+String path = parts[2];
+Pattern simplePattern =
Pattern.compile("([^/]+)/([^/]+)/([^/]+)/([^/]+)");
+Pattern verbosePattern =
Pattern.compile("projects/([^/]+)/locations/([^/]+)/keyRings/([^/]+)/cryptoKeys/([^/]+)");
+
+Matcher match = null;
+if (verbosePattern.asPredicate().test(path)) {
+match = verbosePattern.matcher(path);
+} else if (simplePattern.asPredicate().test(path)) {
+match = simplePattern.matcher(path);
+}
+
+if (match == null || match.groupCount() != 4 || !match.find())
+throw new SensitivePropertyConfigurationException("Invalid GCP key
pattern");
+
+projectId = match.group(1);
+locationId = match.group(2);
+keyRingId = match.group(3);
+cryptoKeyId =
[GitHub] [nifi] natural commented on a change in pull request #3672: NIFI-6363 Additional Sensitive Property Providers
natural commented on a change in pull request #3672: NIFI-6363 Additional
Sensitive Property Providers
URL: https://github.com/apache/nifi/pull/3672#discussion_r319319279
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/gcp/kms/GCPKMSSensitivePropertyProvider.java
##
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties.sensitive.gcp.kms;
+
+import com.google.cloud.kms.v1.CryptoKeyName;
+import com.google.cloud.kms.v1.DecryptResponse;
+import com.google.cloud.kms.v1.EncryptResponse;
+import com.google.cloud.kms.v1.KeyManagementServiceClient;
+import com.google.protobuf.ByteString;
+import org.apache.commons.lang3.StringUtils;
+import
org.apache.nifi.properties.sensitive.SensitivePropertyConfigurationException;
+import
org.apache.nifi.properties.sensitive.SensitivePropertyProtectionException;
+import org.apache.nifi.properties.sensitive.SensitivePropertyProvider;
+import org.bouncycastle.util.encoders.Base64;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * This provider uses the GCP SDK to interact with the GCP KMS. Values are
encoded/decoded base64, using the
+ * standard encoders from bouncycastle.
+ */
+public class GCPKMSSensitivePropertyProvider implements
SensitivePropertyProvider {
+private static final Logger logger =
LoggerFactory.getLogger(GCPKMSSensitivePropertyProvider.class);
+private static final String IMPLEMENTATION_NAME = "GCP KMS Sensitive
Property Provider";
+
+private static final String MATERIAL_PREFIX = "gcp";
+private static final String MATERIAL_KEY_TYPE = "kms";
+private static final String MATERIAL_SEPARATOR = "/";
+private static final String IMPLEMENTATION_PREFIX = MATERIAL_PREFIX +
MATERIAL_SEPARATOR + MATERIAL_KEY_TYPE + MATERIAL_SEPARATOR;
+
+private String keyId;
+private String projectId;
+private String locationId;
+private String keyRingId;
+private String cryptoKeyId;
+
+private final String resource;
+private final KeyManagementServiceClient client;
+
+
+public GCPKMSSensitivePropertyProvider(String keyId) {
+if
(StringUtils.isBlank(System.getenv("GOOGLE_APPLICATION_CREDENTIALS"))) {
+throw new SensitivePropertyConfigurationException("Unable to find
Google Application Credentials");
+}
+extractKeyParts(keyId);
+this.resource = CryptoKeyName.format(projectId, locationId, keyRingId,
cryptoKeyId);
+try {
+this.client = KeyManagementServiceClient.create();
+} catch (IOException e) {
+throw new SensitivePropertyConfigurationException("Unable to
create service client", e);
+}
+}
+
+private void extractKeyParts(String keyId) {
+if (StringUtils.isBlank(keyId)) throw new
SensitivePropertyConfigurationException("The key cannot be empty");
+
+this.keyId = keyId;
+
+String[] parts = this.keyId.split(MATERIAL_SEPARATOR, 3);
+if (parts.length != 3 || !parts[0].equals(MATERIAL_PREFIX) ||
!parts[1].equals(MATERIAL_KEY_TYPE))
+throw new SensitivePropertyConfigurationException("Invalid GCP
key");
+
+String path = parts[2];
+Pattern simplePattern =
Pattern.compile("([^/]+)/([^/]+)/([^/]+)/([^/]+)");
+Pattern verbosePattern =
Pattern.compile("projects/([^/]+)/locations/([^/]+)/keyRings/([^/]+)/cryptoKeys/([^/]+)");
+
+Matcher match = null;
+if (verbosePattern.asPredicate().test(path)) {
+match = verbosePattern.matcher(path);
+} else if (simplePattern.asPredicate().test(path)) {
+match = simplePattern.matcher(path);
+}
+
+if (match == null || match.groupCount() != 4 || !match.find())
+throw new SensitivePropertyConfigurationException("Invalid GCP key
pattern");
+
+projectId = match.group(1);
+locationId = match.group(2);
+keyRingId = match.group(3);
+cryptoKeyId =
[GitHub] [nifi] natural commented on a change in pull request #3672: NIFI-6363 Additional Sensitive Property Providers
natural commented on a change in pull request #3672: NIFI-6363 Additional
Sensitive Property Providers
URL: https://github.com/apache/nifi/pull/3672#discussion_r319319188
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/gcp/kms/GCPKMSSensitivePropertyProvider.java
##
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties.sensitive.gcp.kms;
+
+import com.google.cloud.kms.v1.CryptoKeyName;
+import com.google.cloud.kms.v1.DecryptResponse;
+import com.google.cloud.kms.v1.EncryptResponse;
+import com.google.cloud.kms.v1.KeyManagementServiceClient;
+import com.google.protobuf.ByteString;
+import org.apache.commons.lang3.StringUtils;
+import
org.apache.nifi.properties.sensitive.SensitivePropertyConfigurationException;
+import
org.apache.nifi.properties.sensitive.SensitivePropertyProtectionException;
+import org.apache.nifi.properties.sensitive.SensitivePropertyProvider;
+import org.bouncycastle.util.encoders.Base64;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * This provider uses the GCP SDK to interact with the GCP KMS. Values are
encoded/decoded base64, using the
+ * standard encoders from bouncycastle.
+ */
+public class GCPKMSSensitivePropertyProvider implements
SensitivePropertyProvider {
+private static final Logger logger =
LoggerFactory.getLogger(GCPKMSSensitivePropertyProvider.class);
+private static final String IMPLEMENTATION_NAME = "GCP KMS Sensitive
Property Provider";
+
+private static final String MATERIAL_PREFIX = "gcp";
+private static final String MATERIAL_KEY_TYPE = "kms";
+private static final String MATERIAL_SEPARATOR = "/";
+private static final String IMPLEMENTATION_PREFIX = MATERIAL_PREFIX +
MATERIAL_SEPARATOR + MATERIAL_KEY_TYPE + MATERIAL_SEPARATOR;
+
+private String keyId;
+private String projectId;
+private String locationId;
+private String keyRingId;
+private String cryptoKeyId;
+
+private final String resource;
+private final KeyManagementServiceClient client;
+
+
+public GCPKMSSensitivePropertyProvider(String keyId) {
+if
(StringUtils.isBlank(System.getenv("GOOGLE_APPLICATION_CREDENTIALS"))) {
+throw new SensitivePropertyConfigurationException("Unable to find
Google Application Credentials");
+}
+extractKeyParts(keyId);
+this.resource = CryptoKeyName.format(projectId, locationId, keyRingId,
cryptoKeyId);
+try {
+this.client = KeyManagementServiceClient.create();
+} catch (IOException e) {
+throw new SensitivePropertyConfigurationException("Unable to
create service client", e);
+}
+}
+
+private void extractKeyParts(String keyId) {
Review comment:
Renamed to `setKeyParts` to better reflect intention + added javadocs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] natural commented on a change in pull request #3672: NIFI-6363 Additional Sensitive Property Providers
natural commented on a change in pull request #3672: NIFI-6363 Additional
Sensitive Property Providers
URL: https://github.com/apache/nifi/pull/3672#discussion_r319319245
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/gcp/kms/GCPKMSSensitivePropertyProvider.java
##
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties.sensitive.gcp.kms;
+
+import com.google.cloud.kms.v1.CryptoKeyName;
+import com.google.cloud.kms.v1.DecryptResponse;
+import com.google.cloud.kms.v1.EncryptResponse;
+import com.google.cloud.kms.v1.KeyManagementServiceClient;
+import com.google.protobuf.ByteString;
+import org.apache.commons.lang3.StringUtils;
+import
org.apache.nifi.properties.sensitive.SensitivePropertyConfigurationException;
+import
org.apache.nifi.properties.sensitive.SensitivePropertyProtectionException;
+import org.apache.nifi.properties.sensitive.SensitivePropertyProvider;
+import org.bouncycastle.util.encoders.Base64;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * This provider uses the GCP SDK to interact with the GCP KMS. Values are
encoded/decoded base64, using the
+ * standard encoders from bouncycastle.
+ */
+public class GCPKMSSensitivePropertyProvider implements
SensitivePropertyProvider {
+private static final Logger logger =
LoggerFactory.getLogger(GCPKMSSensitivePropertyProvider.class);
+private static final String IMPLEMENTATION_NAME = "GCP KMS Sensitive
Property Provider";
+
+private static final String MATERIAL_PREFIX = "gcp";
+private static final String MATERIAL_KEY_TYPE = "kms";
+private static final String MATERIAL_SEPARATOR = "/";
+private static final String IMPLEMENTATION_PREFIX = MATERIAL_PREFIX +
MATERIAL_SEPARATOR + MATERIAL_KEY_TYPE + MATERIAL_SEPARATOR;
+
+private String keyId;
+private String projectId;
+private String locationId;
+private String keyRingId;
+private String cryptoKeyId;
+
+private final String resource;
+private final KeyManagementServiceClient client;
+
+
+public GCPKMSSensitivePropertyProvider(String keyId) {
+if
(StringUtils.isBlank(System.getenv("GOOGLE_APPLICATION_CREDENTIALS"))) {
+throw new SensitivePropertyConfigurationException("Unable to find
Google Application Credentials");
+}
+extractKeyParts(keyId);
+this.resource = CryptoKeyName.format(projectId, locationId, keyRingId,
cryptoKeyId);
+try {
+this.client = KeyManagementServiceClient.create();
+} catch (IOException e) {
+throw new SensitivePropertyConfigurationException("Unable to
create service client", e);
+}
+}
+
+private void extractKeyParts(String keyId) {
+if (StringUtils.isBlank(keyId)) throw new
SensitivePropertyConfigurationException("The key cannot be empty");
+
+this.keyId = keyId;
+
+String[] parts = this.keyId.split(MATERIAL_SEPARATOR, 3);
+if (parts.length != 3 || !parts[0].equals(MATERIAL_PREFIX) ||
!parts[1].equals(MATERIAL_KEY_TYPE))
+throw new SensitivePropertyConfigurationException("Invalid GCP
key");
+
+String path = parts[2];
+Pattern simplePattern =
Pattern.compile("([^/]+)/([^/]+)/([^/]+)/([^/]+)");
Review comment:
Migrated, thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] natural commented on a change in pull request #3672: NIFI-6363 Additional Sensitive Property Providers
natural commented on a change in pull request #3672: NIFI-6363 Additional
Sensitive Property Providers
URL: https://github.com/apache/nifi/pull/3672#discussion_r319318908
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/gcp/kms/GCPKMSSensitivePropertyProvider.java
##
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties.sensitive.gcp.kms;
+
+import com.google.cloud.kms.v1.CryptoKeyName;
+import com.google.cloud.kms.v1.DecryptResponse;
+import com.google.cloud.kms.v1.EncryptResponse;
+import com.google.cloud.kms.v1.KeyManagementServiceClient;
+import com.google.protobuf.ByteString;
+import org.apache.commons.lang3.StringUtils;
+import
org.apache.nifi.properties.sensitive.SensitivePropertyConfigurationException;
+import
org.apache.nifi.properties.sensitive.SensitivePropertyProtectionException;
+import org.apache.nifi.properties.sensitive.SensitivePropertyProvider;
+import org.bouncycastle.util.encoders.Base64;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * This provider uses the GCP SDK to interact with the GCP KMS. Values are
encoded/decoded base64, using the
+ * standard encoders from bouncycastle.
+ */
+public class GCPKMSSensitivePropertyProvider implements
SensitivePropertyProvider {
+private static final Logger logger =
LoggerFactory.getLogger(GCPKMSSensitivePropertyProvider.class);
+private static final String IMPLEMENTATION_NAME = "GCP KMS Sensitive
Property Provider";
+
+private static final String MATERIAL_PREFIX = "gcp";
+private static final String MATERIAL_KEY_TYPE = "kms";
+private static final String MATERIAL_SEPARATOR = "/";
+private static final String IMPLEMENTATION_PREFIX = MATERIAL_PREFIX +
MATERIAL_SEPARATOR + MATERIAL_KEY_TYPE + MATERIAL_SEPARATOR;
+
+private String keyId;
+private String projectId;
+private String locationId;
+private String keyRingId;
+private String cryptoKeyId;
+
+private final String resource;
+private final KeyManagementServiceClient client;
+
+
+public GCPKMSSensitivePropertyProvider(String keyId) {
+if
(StringUtils.isBlank(System.getenv("GOOGLE_APPLICATION_CREDENTIALS"))) {
Review comment:
Thanks for the feedback. I'll double back on this and change the auth if I
can.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] natural commented on a change in pull request #3672: NIFI-6363 Additional Sensitive Property Providers
natural commented on a change in pull request #3672: NIFI-6363 Additional
Sensitive Property Providers
URL: https://github.com/apache/nifi/pull/3672#discussion_r319318113
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/azure/keyvault/AzureKeyVaultSensitivePropertyProvider.java
##
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties.sensitive.azure.keyvault;
+
+import com.microsoft.azure.credentials.ApplicationTokenCredentials;
+import com.microsoft.azure.keyvault.KeyVaultClient;
+import com.microsoft.azure.keyvault.models.KeyOperationResult;
+import com.microsoft.azure.keyvault.models.KeyVaultErrorException;
+import com.microsoft.azure.keyvault.webkey.JsonWebKeyEncryptionAlgorithm;
+import com.microsoft.azure.management.Azure;
+import com.microsoft.rest.LogLevel;
+import org.apache.commons.lang3.StringUtils;
+import
org.apache.nifi.properties.sensitive.SensitivePropertyConfigurationException;
+import
org.apache.nifi.properties.sensitive.SensitivePropertyProtectionException;
+import org.apache.nifi.properties.sensitive.SensitivePropertyProvider;
+import org.bouncycastle.util.encoders.Base64;
+import org.bouncycastle.util.encoders.DecoderException;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+
+
+// Azure with Java:
+// https://docs.microsoft.com/en-us/azure/java/?view=azure-java-stable
+
+// Azure auth with Java:
+// https://github.com/Azure/azure-libraries-for-java/blob/master/AUTH.md
+
+// Azure Key Vault with Java:
+//
https://docs.microsoft.com/en-us/java/api/com.microsoft.azure.keyvault?view=azure-java-stable
+
+
+public class AzureKeyVaultSensitivePropertyProvider implements
SensitivePropertyProvider {
+private static final String IMPLEMENTATION_NAME = "Azure Key Vault
Sensitive Property Provider";
+private static final String MATERIAL_PREFIX = "azure";
+private static final String MATERIAL_KEY_TYPE = "vault";
+private static final JsonWebKeyEncryptionAlgorithm algo =
JsonWebKeyEncryptionAlgorithm.RSA_OAEP;
+
+private final KeyVaultClient client;
+private final String vaultId;
+private final String keyId;
+
+public AzureKeyVaultSensitivePropertyProvider(String material) {
+final File credFile = new File(System.getenv("AZURE_AUTH_LOCATION"));
+try {
+ApplicationTokenCredentials.fromFile(credFile).clientId();
+} catch (IOException e) {
+throw new SensitivePropertyConfigurationException(e);
+}
+
+String prefix = MATERIAL_PREFIX + "/" + MATERIAL_KEY_TYPE + "/";
Review comment:
Fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] natural commented on a change in pull request #3672: NIFI-6363 Additional Sensitive Property Providers
natural commented on a change in pull request #3672: NIFI-6363 Additional
Sensitive Property Providers
URL: https://github.com/apache/nifi/pull/3672#discussion_r319317965
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/aes/AESSensitivePropertyProvider.java
##
@@ -138,7 +146,7 @@ public String getName() {
*/
@Override
public String getIdentifierKey() {
-return IMPLEMENTATION_KEY +
getKeySize(Hex.toHexString(key.getEncoded()));
+return IMPLEMENTATION_KEY + Hex.toHexString(key.getEncoded()); //
getKeySize();
Review comment:
Oof, yeah, that wasn't supposed to end up this way. I've reverted all of
the bits related to this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] natural commented on a change in pull request #3672: NIFI-6363 Additional Sensitive Property Providers
natural commented on a change in pull request #3672: NIFI-6363 Additional Sensitive Property Providers URL: https://github.com/apache/nifi/pull/3672#discussion_r319317666 ## File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/StandardExternalPropertyLookup.java ## @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.properties.sensitive; + +import org.apache.commons.lang3.StringUtils; + +import java.io.File; +import java.io.FileInputStream; +import java.io.IOException; +import java.util.HashMap; +import java.util.Map; +import java.util.Properties; + +/** + * Slurps property values from the usual places in the usual order. Review comment: Good catch, updated comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] natural commented on a change in pull request #3672: NIFI-6363 Additional Sensitive Property Providers
natural commented on a change in pull request #3672: NIFI-6363 Additional
Sensitive Property Providers
URL: https://github.com/apache/nifi/pull/3672#discussion_r319317569
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/SensitivePropertyConfigurationException.java
##
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties.sensitive;
+
+public class SensitivePropertyConfigurationException extends RuntimeException {
Review comment:
Thanks for the suggestion. Added exception hierarchy and reviewed for
simplification opportunities.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] natural commented on a change in pull request #3672: NIFI-6363 Additional Sensitive Property Providers
natural commented on a change in pull request #3672: NIFI-6363 Additional
Sensitive Property Providers
URL: https://github.com/apache/nifi/pull/3672#discussion_r319312422
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/ProtectedNiFiProperties.java
##
@@ -72,10 +75,19 @@ public ProtectedNiFiProperties(NiFiProperties props) {
*
* @param rawProps the Properties to contain
*/
-public ProtectedNiFiProperties(Properties rawProps) {
-this(new StandardNiFiProperties(rawProps));
+public ProtectedNiFiProperties(Properties rawProps,
SensitivePropertyProvider sensitivePropertyProvider) {
+this(new StandardNiFiProperties(rawProps), sensitivePropertyProvider);
}
+public ProtectedNiFiProperties(NiFiProperties props, String keyOrKeyId) {
Review comment:
Added.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] natural commented on a change in pull request #3672: NIFI-6363 Additional Sensitive Property Providers
natural commented on a change in pull request #3672: NIFI-6363 Additional
Sensitive Property Providers
URL: https://github.com/apache/nifi/pull/3672#discussion_r319312033
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/ExternalProperties.java
##
@@ -14,10 +14,21 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.nifi.properties;
+package org.apache.nifi.properties.sensitive;
-public interface SensitivePropertyProviderFactory {
+/**
+ * ExternalProperties is an interface for reading external values by name.
+ *
+ */
+public interface ExternalProperties {
-SensitivePropertyProvider getProvider();
+/**
+ * Read an external property by name.
+ *
+ * @param name the name or key of the external property
+ * @return external property value, as a string
+ */
+String get(String name);
+String get(String name, String missing);
Review comment:
Added, thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] natural commented on a change in pull request #3672: NIFI-6363 Additional Sensitive Property Providers
natural commented on a change in pull request #3672: NIFI-6363 Additional
Sensitive Property Providers
URL: https://github.com/apache/nifi/pull/3672#discussion_r319311223
##
File path:
nifi-commons/nifi-security-utils/src/test/java/org/apache/nifi/security/util/CipherUtilsTest.java
##
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.security.util;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+import javax.crypto.Cipher;
+import javax.crypto.NoSuchPaddingException;
+import java.security.NoSuchAlgorithmException;
+import java.security.NoSuchProviderException;
+
+
+public class CipherUtilsTest {
+
+// This test shows we get an IV of the correct length when we ask for one.
+@Test
+public void showIvLengthIsCorrect() {
+byte[] iv = CipherUtils.generateIV();
+Assert.assertNotNull(iv);
+Assert.assertEquals(CipherUtils.IV_LENGTH, iv.length);
+}
+
+// This test shows we get new IVs each time we ask for one.
+@Test
+public void showIvsAreDifferentEachTime() {
+// Create a bunch of IVs
+int count = 4;
+byte [][] ivs = new byte[count][];
+for (int i = 0; i < count; i++) {
+ivs[i] = CipherUtils.generateIV();
+}
+
+// Compare each IV to every other IV
+for (int i = 0; i < count; i++) {
+byte[] iv = ivs[i];
+
+for (int j = 0; j < count; j++) {
Review comment:
Great example, note added.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] natural commented on a change in pull request #3672: NIFI-6363 Additional Sensitive Property Providers
natural commented on a change in pull request #3672: NIFI-6363 Additional
Sensitive Property Providers
URL: https://github.com/apache/nifi/pull/3672#discussion_r319311044
##
File path:
nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/security/util/CipherUtils.java
##
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.security.util;
+
+import org.bouncycastle.jce.provider.BouncyCastleProvider;
+import org.bouncycastle.util.encoders.Hex;
+
+import javax.crypto.Cipher;
+import javax.crypto.NoSuchPaddingException;
+import java.security.NoSuchAlgorithmException;
+import java.security.NoSuchProviderException;
+import java.security.SecureRandom;
+import java.security.Security;
+
+
+/**
+ * Common functionality for ciphers: pre-initialized ciphers, IVs, random
value generators, etc.
+ */
+public class CipherUtils {
+final static SecureRandom random = new SecureRandom();
+public final static int IV_LENGTH = 12;
Review comment:
Good catch, added.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] apiri opened a new pull request #3682: NIFI-6604: Removing large NARs from generated assemblies to reduce size.
apiri opened a new pull request #3682: NIFI-6604: Removing large NARs from
generated assemblies to reduce size.
URL: https://github.com/apache/nifi/pull/3682
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
_Enables X functionality; fixes bug NIFI-._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically `master`)?
- [ ] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on both JDK 8 and
JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (NIFI-6604) Reduce the size of the generated assemblies
Aldrin Piri created NIFI-6604: - Summary: Reduce the size of the generated assemblies Key: NIFI-6604 URL: https://issues.apache.org/jira/browse/NIFI-6604 Project: Apache NiFi Issue Type: Task Components: Tools and Build Reporter: Aldrin Piri Assignee: Aldrin Piri Fix For: 1.10.0 As per the discussion on https://lists.apache.org/thread.html/d45f33805dcb8096914a06853bc3b8eca1acf50c2ae7b68e75eabfe8@ we must reduce the assembly size such that we can publish the convenience binaries as part of the release. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6546) Add NiFi Expression to update JSON value
[ https://issues.apache.org/jira/browse/NIFI-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919001#comment-16919001 ] ASF subversion and git services commented on NIFI-6546: --- Commit 5df6b0edbb8de259c871e7843c2057b6938af20f in nifi's branch refs/heads/master from mans2singh [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=5df6b0e ] NIFI-6546 - Add JsonPath set value support NIFI-6546 - Addressed review comments (double semi-colon, javadoc, error message, null init) NIFI-6546 - Refactored test and added log error for JsonPath exception based on review NIFI-6546 - Refactored tests based on review comments NIFI-6546 - Removed redundant phone check and added constant for empty path NIFI-6546 - Added brackets based on review comments This closes #3646 Signed-off-by: Mike Thomsen > Add NiFi Expression to update JSON value > > > Key: NIFI-6546 > URL: https://issues.apache.org/jira/browse/NIFI-6546 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.9.1 >Reporter: Mans Singh >Assignee: Mans Singh >Priority: Minor > Labels: expression-language, json, set > Fix For: 1.9.2 > > Original Estimate: 8h > Time Spent: 5h > Remaining Estimate: 3h > > Add expression language JsonPath support to set values. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6546) Add NiFi Expression to update JSON value
[ https://issues.apache.org/jira/browse/NIFI-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918999#comment-16918999 ] ASF subversion and git services commented on NIFI-6546: --- Commit 5df6b0edbb8de259c871e7843c2057b6938af20f in nifi's branch refs/heads/master from mans2singh [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=5df6b0e ] NIFI-6546 - Add JsonPath set value support NIFI-6546 - Addressed review comments (double semi-colon, javadoc, error message, null init) NIFI-6546 - Refactored test and added log error for JsonPath exception based on review NIFI-6546 - Refactored tests based on review comments NIFI-6546 - Removed redundant phone check and added constant for empty path NIFI-6546 - Added brackets based on review comments This closes #3646 Signed-off-by: Mike Thomsen > Add NiFi Expression to update JSON value > > > Key: NIFI-6546 > URL: https://issues.apache.org/jira/browse/NIFI-6546 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.9.1 >Reporter: Mans Singh >Assignee: Mans Singh >Priority: Minor > Labels: expression-language, json, set > Fix For: 1.9.2 > > Original Estimate: 8h > Time Spent: 5h > Remaining Estimate: 3h > > Add expression language JsonPath support to set values. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6546) Add NiFi Expression to update JSON value
[ https://issues.apache.org/jira/browse/NIFI-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919000#comment-16919000 ] ASF subversion and git services commented on NIFI-6546: --- Commit 5df6b0edbb8de259c871e7843c2057b6938af20f in nifi's branch refs/heads/master from mans2singh [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=5df6b0e ] NIFI-6546 - Add JsonPath set value support NIFI-6546 - Addressed review comments (double semi-colon, javadoc, error message, null init) NIFI-6546 - Refactored test and added log error for JsonPath exception based on review NIFI-6546 - Refactored tests based on review comments NIFI-6546 - Removed redundant phone check and added constant for empty path NIFI-6546 - Added brackets based on review comments This closes #3646 Signed-off-by: Mike Thomsen > Add NiFi Expression to update JSON value > > > Key: NIFI-6546 > URL: https://issues.apache.org/jira/browse/NIFI-6546 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.9.1 >Reporter: Mans Singh >Assignee: Mans Singh >Priority: Minor > Labels: expression-language, json, set > Fix For: 1.9.2 > > Original Estimate: 8h > Time Spent: 5h > Remaining Estimate: 3h > > Add expression language JsonPath support to set values. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6546) Add NiFi Expression to update JSON value
[ https://issues.apache.org/jira/browse/NIFI-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918996#comment-16918996 ] ASF subversion and git services commented on NIFI-6546: --- Commit 5df6b0edbb8de259c871e7843c2057b6938af20f in nifi's branch refs/heads/master from mans2singh [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=5df6b0e ] NIFI-6546 - Add JsonPath set value support NIFI-6546 - Addressed review comments (double semi-colon, javadoc, error message, null init) NIFI-6546 - Refactored test and added log error for JsonPath exception based on review NIFI-6546 - Refactored tests based on review comments NIFI-6546 - Removed redundant phone check and added constant for empty path NIFI-6546 - Added brackets based on review comments This closes #3646 Signed-off-by: Mike Thomsen > Add NiFi Expression to update JSON value > > > Key: NIFI-6546 > URL: https://issues.apache.org/jira/browse/NIFI-6546 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.9.1 >Reporter: Mans Singh >Assignee: Mans Singh >Priority: Minor > Labels: expression-language, json, set > Fix For: 1.9.2 > > Original Estimate: 8h > Time Spent: 4h 50m > Remaining Estimate: 3h 10m > > Add expression language JsonPath support to set values. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6546) Add NiFi Expression to update JSON value
[ https://issues.apache.org/jira/browse/NIFI-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918998#comment-16918998 ] ASF subversion and git services commented on NIFI-6546: --- Commit 5df6b0edbb8de259c871e7843c2057b6938af20f in nifi's branch refs/heads/master from mans2singh [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=5df6b0e ] NIFI-6546 - Add JsonPath set value support NIFI-6546 - Addressed review comments (double semi-colon, javadoc, error message, null init) NIFI-6546 - Refactored test and added log error for JsonPath exception based on review NIFI-6546 - Refactored tests based on review comments NIFI-6546 - Removed redundant phone check and added constant for empty path NIFI-6546 - Added brackets based on review comments This closes #3646 Signed-off-by: Mike Thomsen > Add NiFi Expression to update JSON value > > > Key: NIFI-6546 > URL: https://issues.apache.org/jira/browse/NIFI-6546 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.9.1 >Reporter: Mans Singh >Assignee: Mans Singh >Priority: Minor > Labels: expression-language, json, set > Fix For: 1.9.2 > > Original Estimate: 8h > Time Spent: 4h 50m > Remaining Estimate: 3h 10m > > Add expression language JsonPath support to set values. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Resolved] (NIFI-6546) Add NiFi Expression to update JSON value
[ https://issues.apache.org/jira/browse/NIFI-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Thomsen resolved NIFI-6546. Fix Version/s: (was: 1.9.2) 1.10.0 Resolution: Fixed > Add NiFi Expression to update JSON value > > > Key: NIFI-6546 > URL: https://issues.apache.org/jira/browse/NIFI-6546 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.9.1 >Reporter: Mans Singh >Assignee: Mans Singh >Priority: Minor > Labels: expression-language, json, set > Fix For: 1.10.0 > > Original Estimate: 8h > Time Spent: 5h > Remaining Estimate: 3h > > Add expression language JsonPath support to set values. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] asfgit closed pull request #3646: NIFI-6546 - Add JsonPath set value support
asfgit closed pull request #3646: NIFI-6546 - Add JsonPath set value support URL: https://github.com/apache/nifi/pull/3646 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (NIFI-6546) Add NiFi Expression to update JSON value
[ https://issues.apache.org/jira/browse/NIFI-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918997#comment-16918997 ] ASF subversion and git services commented on NIFI-6546: --- Commit 5df6b0edbb8de259c871e7843c2057b6938af20f in nifi's branch refs/heads/master from mans2singh [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=5df6b0e ] NIFI-6546 - Add JsonPath set value support NIFI-6546 - Addressed review comments (double semi-colon, javadoc, error message, null init) NIFI-6546 - Refactored test and added log error for JsonPath exception based on review NIFI-6546 - Refactored tests based on review comments NIFI-6546 - Removed redundant phone check and added constant for empty path NIFI-6546 - Added brackets based on review comments This closes #3646 Signed-off-by: Mike Thomsen > Add NiFi Expression to update JSON value > > > Key: NIFI-6546 > URL: https://issues.apache.org/jira/browse/NIFI-6546 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.9.1 >Reporter: Mans Singh >Assignee: Mans Singh >Priority: Minor > Labels: expression-language, json, set > Fix For: 1.9.2 > > Original Estimate: 8h > Time Spent: 4h 50m > Remaining Estimate: 3h 10m > > Add expression language JsonPath support to set values. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] ottobackwards commented on a change in pull request #3671: NIFI-6586 - Code comments and doc updates
ottobackwards commented on a change in pull request #3671: NIFI-6586 - Code comments and doc updates URL: https://github.com/apache/nifi/pull/3671#discussion_r319273925 ## File path: nifi-docs/src/main/asciidoc/user-guide.adoc ## @@ -1685,6 +1685,8 @@ The FlowFiles enqueued in a Connection can also be deleted when necessary. The r via `Empty queue` in the Connection's context menu. This action can also be performed if the source and destination are actively running. +Hovering over the queue will also reveal predicted statistics on when the queue may encounter back pressure, either due to the object count or content size meeting the current threshold +settings. Predictions will only be available when NiFi has enough data in it's internal repository and if it's model is accurate enough to broadcast a prediction. Review comment: What happens if they aren't available? Will it tell you? Can you tell the difference between not enough data and not configured? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] marcelojscosta commented on issue #2231: NIFI-4521 MS SQL CDC Processor
marcelojscosta commented on issue #2231: NIFI-4521 MS SQL CDC Processor URL: https://github.com/apache/nifi/pull/2231#issuecomment-526354816 Hi folks, I will just try to test this branch in my environment too. I will inform news at the next days This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] bbende commented on issue #3614: NIFI-6371: Fixed bug that caused root-level Controller Services to st…
bbende commented on issue #3614: NIFI-6371: Fixed bug that caused root-level Controller Services to st… URL: https://github.com/apache/nifi/pull/3614#issuecomment-526348138 Looks good, thanks! Merged to master This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (NIFI-6371) Validation still occurs for deleted 'Controller Level' Controller Services
[ https://issues.apache.org/jira/browse/NIFI-6371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Bende updated NIFI-6371: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Validation still occurs for deleted 'Controller Level' Controller Services > --- > > Key: NIFI-6371 > URL: https://issues.apache.org/jira/browse/NIFI-6371 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Nathan Gough >Assignee: Mark Payne >Priority: Major > Labels: controller_services, validator > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > > When verifying an XXE validator PR > ([https://github.com/alopresto/nifi/commits/pr3507]) for the > XmlConfigurationLookupService, Andy found that the validator on a > controller-level scoped controller service will still be executed/emit logs > even after it is deleted. This does not occur on the process-group scoped > controller services. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] asfgit closed pull request #3614: NIFI-6371: Fixed bug that caused root-level Controller Services to st…
asfgit closed pull request #3614: NIFI-6371: Fixed bug that caused root-level Controller Services to st… URL: https://github.com/apache/nifi/pull/3614 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (NIFI-6371) Validation still occurs for deleted 'Controller Level' Controller Services
[ https://issues.apache.org/jira/browse/NIFI-6371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918928#comment-16918928 ] ASF subversion and git services commented on NIFI-6371: --- Commit 254a84d74da1cd5299e67605e8e20e6e4f839457 in nifi's branch refs/heads/master from Mark Payne [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=254a84d ] NIFI-6371: Fixed bug that caused root-level Controller Services to stay cached and therefore continually be validated, even after being deleted. Also cleaned up code that looked for specific text in Exceptions instead of of catching specific Exceptions This closes #3614. Signed-off-by: Bryan Bende > Validation still occurs for deleted 'Controller Level' Controller Services > --- > > Key: NIFI-6371 > URL: https://issues.apache.org/jira/browse/NIFI-6371 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Nathan Gough >Assignee: Mark Payne >Priority: Major > Labels: controller_services, validator > Fix For: 1.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > > When verifying an XXE validator PR > ([https://github.com/alopresto/nifi/commits/pr3507]) for the > XmlConfigurationLookupService, Andy found that the validator on a > controller-level scoped controller service will still be executed/emit logs > even after it is deleted. This does not occur on the process-group scoped > controller services. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (NIFI-6603) Variables: There is no way to cancel creating a new variable unless a variable name is provided.
Andrew Lim created NIFI-6603: Summary: Variables: There is no way to cancel creating a new variable unless a variable name is provided. Key: NIFI-6603 URL: https://issues.apache.org/jira/browse/NIFI-6603 Project: Apache NiFi Issue Type: Improvement Components: Core UI Reporter: Andrew Lim Open the variables window and select "+" . In the New Variable window do not enter a value for the Variable Name field and select "OK". A Configuration Error dialog is shown, select "OK". There is no way for the user to stop variable creation without entering a name and then selecting Cancel from the Variables window. We should add a "Cancel" button to the "New Variable" window. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (NIFI-6602) Parameters: "Apply" button should be greyed out if no changes have been
Andrew Lim created NIFI-6602: Summary: Parameters: "Apply" button should be greyed out if no changes have been Key: NIFI-6602 URL: https://issues.apache.org/jira/browse/NIFI-6602 Project: Apache NiFi Issue Type: Improvement Components: Core UI Reporter: Andrew Lim Open the Parameters Context window. Whether on the Settings tab or Parameters tab, the "Apply" button is available for selection immediately even when no edits have been made to apply. If selected, the window is closed which is what the Cancel button is for. Noticed while testing the PR branch for NIFI-6282. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (NIFI-6601) Variables: "Apply" button should be greyed out if no changes have been made
Andrew Lim created NIFI-6601: Summary: Variables: "Apply" button should be greyed out if no changes have been made Key: NIFI-6601 URL: https://issues.apache.org/jira/browse/NIFI-6601 Project: Apache NiFi Issue Type: Improvement Components: Core UI Reporter: Andrew Lim After opening the Variables window, the "Apply" button is available for selection immediately even when no edits have been made to apply. If selected, the window is closed which is what the Cancel button is for. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] bbende commented on a change in pull request #3679: NIFI-6089 Add Parquet record reader and writer
bbende commented on a change in pull request #3679: NIFI-6089 Add Parquet
record reader and writer
URL: https://github.com/apache/nifi/pull/3679#discussion_r319228444
##
File path:
nifi-nar-bundles/nifi-parquet-bundle/nifi-parquet-processors/src/test/java/org/apache/nifi/parquet/TestParquetRecordSetWriter.java
##
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.parquet;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.commons.io.IOUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.schema.access.SchemaAccessUtils;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.record.MapRecord;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.util.MockComponentLog;
+import org.apache.nifi.util.MockConfigurationContext;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.util.HadoopInputFile;
+import org.apache.parquet.io.InputFile;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+
+public class TestParquetRecordSetWriter {
Review comment:
Added a test for one of the branches, but wasn't straight forward for the
other. In practice I think it will always hit the code path with
compiledAvroSchemaCache while running the real application.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] bbende commented on issue #3679: NIFI-6089 Add Parquet record reader and writer
bbende commented on issue #3679: NIFI-6089 Add Parquet record reader and writer URL: https://github.com/apache/nifi/pull/3679#issuecomment-526321308 Added new commit to address feedback. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (NIFI-6282) UI - create parameter context
[ https://issues.apache.org/jira/browse/NIFI-6282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman updated NIFI-6282: -- Fix Version/s: 1.10.0 Resolution: Fixed Status: Resolved (was: Patch Available) > UI - create parameter context > - > > Key: NIFI-6282 > URL: https://issues.apache.org/jira/browse/NIFI-6282 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Scott Aslan >Assignee: Scott Aslan >Priority: Major > Fix For: 1.10.0 > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] mcgilman closed pull request #3606: [WIP] Nifi 6282
mcgilman closed pull request #3606: [WIP] Nifi 6282 URL: https://github.com/apache/nifi/pull/3606 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (NIFI-6282) UI - create parameter context
[ https://issues.apache.org/jira/browse/NIFI-6282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918884#comment-16918884 ] ASF subversion and git services commented on NIFI-6282: --- Commit f678c75d7004656918d4249db49350515cc2 in nifi's branch refs/heads/master from Scott Aslan [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f678c75 ] [NIFI-6282] manage parameters and parameter contexts [NIFI-6282] when creating a parameter context inline during PG configuration set the newly created parameter context as the selected option [NIFI-6282] If a request to update a parameter context fails, then update the button model to give the user the ability to Apply or Cancel again. [NIFI-6282] address pr review comments [NIFI-6282] remove es6 let [NIFI-6282] update marshall parameters logic and add comments [NIFI-6282] deterministic parameter usage listing, update CS status on PG PC change, expand all twisties by default, remove es6 const [NIFI-6282] update regex and [NIFI-6282] update parameter loading/serialization/marshalling [NIFI-6282] use referencingComponents instead of affectedComponents [NIFI-6282] activate Apply button for sensitive parameter set empty string change [NIFI-6282] fix bug with PG parameters context menu enable [NIFI-6282] only allow delete and recreate of a parameter with equivalent sensitivity [NIFI-6282] display referencing components during parameter management as well as during the parameter context update [NIFI-6282] display no value set in parameter table when parameter value is null [NIFI-6282] - Add ellipsis to referencing component names. - Addressing issues canceling update requests. - Addressing issues with incorrect service scope. - Addressing issue showing the affected parameters. > UI - create parameter context > - > > Key: NIFI-6282 > URL: https://issues.apache.org/jira/browse/NIFI-6282 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Scott Aslan >Assignee: Scott Aslan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6282) UI - create parameter context
[ https://issues.apache.org/jira/browse/NIFI-6282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918880#comment-16918880 ] ASF subversion and git services commented on NIFI-6282: --- Commit f678c75d7004656918d4249db49350515cc2 in nifi's branch refs/heads/master from Scott Aslan [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f678c75 ] [NIFI-6282] manage parameters and parameter contexts [NIFI-6282] when creating a parameter context inline during PG configuration set the newly created parameter context as the selected option [NIFI-6282] If a request to update a parameter context fails, then update the button model to give the user the ability to Apply or Cancel again. [NIFI-6282] address pr review comments [NIFI-6282] remove es6 let [NIFI-6282] update marshall parameters logic and add comments [NIFI-6282] deterministic parameter usage listing, update CS status on PG PC change, expand all twisties by default, remove es6 const [NIFI-6282] update regex and [NIFI-6282] update parameter loading/serialization/marshalling [NIFI-6282] use referencingComponents instead of affectedComponents [NIFI-6282] activate Apply button for sensitive parameter set empty string change [NIFI-6282] fix bug with PG parameters context menu enable [NIFI-6282] only allow delete and recreate of a parameter with equivalent sensitivity [NIFI-6282] display referencing components during parameter management as well as during the parameter context update [NIFI-6282] display no value set in parameter table when parameter value is null [NIFI-6282] - Add ellipsis to referencing component names. - Addressing issues canceling update requests. - Addressing issues with incorrect service scope. - Addressing issue showing the affected parameters. > UI - create parameter context > - > > Key: NIFI-6282 > URL: https://issues.apache.org/jira/browse/NIFI-6282 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Scott Aslan >Assignee: Scott Aslan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6282) UI - create parameter context
[ https://issues.apache.org/jira/browse/NIFI-6282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918871#comment-16918871 ] ASF subversion and git services commented on NIFI-6282: --- Commit f678c75d7004656918d4249db49350515cc2 in nifi's branch refs/heads/master from Scott Aslan [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f678c75 ] [NIFI-6282] manage parameters and parameter contexts [NIFI-6282] when creating a parameter context inline during PG configuration set the newly created parameter context as the selected option [NIFI-6282] If a request to update a parameter context fails, then update the button model to give the user the ability to Apply or Cancel again. [NIFI-6282] address pr review comments [NIFI-6282] remove es6 let [NIFI-6282] update marshall parameters logic and add comments [NIFI-6282] deterministic parameter usage listing, update CS status on PG PC change, expand all twisties by default, remove es6 const [NIFI-6282] update regex and [NIFI-6282] update parameter loading/serialization/marshalling [NIFI-6282] use referencingComponents instead of affectedComponents [NIFI-6282] activate Apply button for sensitive parameter set empty string change [NIFI-6282] fix bug with PG parameters context menu enable [NIFI-6282] only allow delete and recreate of a parameter with equivalent sensitivity [NIFI-6282] display referencing components during parameter management as well as during the parameter context update [NIFI-6282] display no value set in parameter table when parameter value is null [NIFI-6282] - Add ellipsis to referencing component names. - Addressing issues canceling update requests. - Addressing issues with incorrect service scope. - Addressing issue showing the affected parameters. > UI - create parameter context > - > > Key: NIFI-6282 > URL: https://issues.apache.org/jira/browse/NIFI-6282 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Scott Aslan >Assignee: Scott Aslan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6282) UI - create parameter context
[ https://issues.apache.org/jira/browse/NIFI-6282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918887#comment-16918887 ] ASF subversion and git services commented on NIFI-6282: --- Commit f678c75d7004656918d4249db49350515cc2 in nifi's branch refs/heads/master from Scott Aslan [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f678c75 ] [NIFI-6282] manage parameters and parameter contexts [NIFI-6282] when creating a parameter context inline during PG configuration set the newly created parameter context as the selected option [NIFI-6282] If a request to update a parameter context fails, then update the button model to give the user the ability to Apply or Cancel again. [NIFI-6282] address pr review comments [NIFI-6282] remove es6 let [NIFI-6282] update marshall parameters logic and add comments [NIFI-6282] deterministic parameter usage listing, update CS status on PG PC change, expand all twisties by default, remove es6 const [NIFI-6282] update regex and [NIFI-6282] update parameter loading/serialization/marshalling [NIFI-6282] use referencingComponents instead of affectedComponents [NIFI-6282] activate Apply button for sensitive parameter set empty string change [NIFI-6282] fix bug with PG parameters context menu enable [NIFI-6282] only allow delete and recreate of a parameter with equivalent sensitivity [NIFI-6282] display referencing components during parameter management as well as during the parameter context update [NIFI-6282] display no value set in parameter table when parameter value is null [NIFI-6282] - Add ellipsis to referencing component names. - Addressing issues canceling update requests. - Addressing issues with incorrect service scope. - Addressing issue showing the affected parameters. > UI - create parameter context > - > > Key: NIFI-6282 > URL: https://issues.apache.org/jira/browse/NIFI-6282 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Scott Aslan >Assignee: Scott Aslan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6282) UI - create parameter context
[ https://issues.apache.org/jira/browse/NIFI-6282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918876#comment-16918876 ] ASF subversion and git services commented on NIFI-6282: --- Commit f678c75d7004656918d4249db49350515cc2 in nifi's branch refs/heads/master from Scott Aslan [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f678c75 ] [NIFI-6282] manage parameters and parameter contexts [NIFI-6282] when creating a parameter context inline during PG configuration set the newly created parameter context as the selected option [NIFI-6282] If a request to update a parameter context fails, then update the button model to give the user the ability to Apply or Cancel again. [NIFI-6282] address pr review comments [NIFI-6282] remove es6 let [NIFI-6282] update marshall parameters logic and add comments [NIFI-6282] deterministic parameter usage listing, update CS status on PG PC change, expand all twisties by default, remove es6 const [NIFI-6282] update regex and [NIFI-6282] update parameter loading/serialization/marshalling [NIFI-6282] use referencingComponents instead of affectedComponents [NIFI-6282] activate Apply button for sensitive parameter set empty string change [NIFI-6282] fix bug with PG parameters context menu enable [NIFI-6282] only allow delete and recreate of a parameter with equivalent sensitivity [NIFI-6282] display referencing components during parameter management as well as during the parameter context update [NIFI-6282] display no value set in parameter table when parameter value is null [NIFI-6282] - Add ellipsis to referencing component names. - Addressing issues canceling update requests. - Addressing issues with incorrect service scope. - Addressing issue showing the affected parameters. > UI - create parameter context > - > > Key: NIFI-6282 > URL: https://issues.apache.org/jira/browse/NIFI-6282 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Scott Aslan >Assignee: Scott Aslan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6282) UI - create parameter context
[ https://issues.apache.org/jira/browse/NIFI-6282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918885#comment-16918885 ] ASF subversion and git services commented on NIFI-6282: --- Commit f678c75d7004656918d4249db49350515cc2 in nifi's branch refs/heads/master from Scott Aslan [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f678c75 ] [NIFI-6282] manage parameters and parameter contexts [NIFI-6282] when creating a parameter context inline during PG configuration set the newly created parameter context as the selected option [NIFI-6282] If a request to update a parameter context fails, then update the button model to give the user the ability to Apply or Cancel again. [NIFI-6282] address pr review comments [NIFI-6282] remove es6 let [NIFI-6282] update marshall parameters logic and add comments [NIFI-6282] deterministic parameter usage listing, update CS status on PG PC change, expand all twisties by default, remove es6 const [NIFI-6282] update regex and [NIFI-6282] update parameter loading/serialization/marshalling [NIFI-6282] use referencingComponents instead of affectedComponents [NIFI-6282] activate Apply button for sensitive parameter set empty string change [NIFI-6282] fix bug with PG parameters context menu enable [NIFI-6282] only allow delete and recreate of a parameter with equivalent sensitivity [NIFI-6282] display referencing components during parameter management as well as during the parameter context update [NIFI-6282] display no value set in parameter table when parameter value is null [NIFI-6282] - Add ellipsis to referencing component names. - Addressing issues canceling update requests. - Addressing issues with incorrect service scope. - Addressing issue showing the affected parameters. > UI - create parameter context > - > > Key: NIFI-6282 > URL: https://issues.apache.org/jira/browse/NIFI-6282 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Scott Aslan >Assignee: Scott Aslan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6282) UI - create parameter context
[ https://issues.apache.org/jira/browse/NIFI-6282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918883#comment-16918883 ] ASF subversion and git services commented on NIFI-6282: --- Commit f678c75d7004656918d4249db49350515cc2 in nifi's branch refs/heads/master from Scott Aslan [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f678c75 ] [NIFI-6282] manage parameters and parameter contexts [NIFI-6282] when creating a parameter context inline during PG configuration set the newly created parameter context as the selected option [NIFI-6282] If a request to update a parameter context fails, then update the button model to give the user the ability to Apply or Cancel again. [NIFI-6282] address pr review comments [NIFI-6282] remove es6 let [NIFI-6282] update marshall parameters logic and add comments [NIFI-6282] deterministic parameter usage listing, update CS status on PG PC change, expand all twisties by default, remove es6 const [NIFI-6282] update regex and [NIFI-6282] update parameter loading/serialization/marshalling [NIFI-6282] use referencingComponents instead of affectedComponents [NIFI-6282] activate Apply button for sensitive parameter set empty string change [NIFI-6282] fix bug with PG parameters context menu enable [NIFI-6282] only allow delete and recreate of a parameter with equivalent sensitivity [NIFI-6282] display referencing components during parameter management as well as during the parameter context update [NIFI-6282] display no value set in parameter table when parameter value is null [NIFI-6282] - Add ellipsis to referencing component names. - Addressing issues canceling update requests. - Addressing issues with incorrect service scope. - Addressing issue showing the affected parameters. > UI - create parameter context > - > > Key: NIFI-6282 > URL: https://issues.apache.org/jira/browse/NIFI-6282 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Scott Aslan >Assignee: Scott Aslan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6282) UI - create parameter context
[ https://issues.apache.org/jira/browse/NIFI-6282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918877#comment-16918877 ] ASF subversion and git services commented on NIFI-6282: --- Commit f678c75d7004656918d4249db49350515cc2 in nifi's branch refs/heads/master from Scott Aslan [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f678c75 ] [NIFI-6282] manage parameters and parameter contexts [NIFI-6282] when creating a parameter context inline during PG configuration set the newly created parameter context as the selected option [NIFI-6282] If a request to update a parameter context fails, then update the button model to give the user the ability to Apply or Cancel again. [NIFI-6282] address pr review comments [NIFI-6282] remove es6 let [NIFI-6282] update marshall parameters logic and add comments [NIFI-6282] deterministic parameter usage listing, update CS status on PG PC change, expand all twisties by default, remove es6 const [NIFI-6282] update regex and [NIFI-6282] update parameter loading/serialization/marshalling [NIFI-6282] use referencingComponents instead of affectedComponents [NIFI-6282] activate Apply button for sensitive parameter set empty string change [NIFI-6282] fix bug with PG parameters context menu enable [NIFI-6282] only allow delete and recreate of a parameter with equivalent sensitivity [NIFI-6282] display referencing components during parameter management as well as during the parameter context update [NIFI-6282] display no value set in parameter table when parameter value is null [NIFI-6282] - Add ellipsis to referencing component names. - Addressing issues canceling update requests. - Addressing issues with incorrect service scope. - Addressing issue showing the affected parameters. > UI - create parameter context > - > > Key: NIFI-6282 > URL: https://issues.apache.org/jira/browse/NIFI-6282 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Scott Aslan >Assignee: Scott Aslan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6282) UI - create parameter context
[ https://issues.apache.org/jira/browse/NIFI-6282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918874#comment-16918874 ] ASF subversion and git services commented on NIFI-6282: --- Commit f678c75d7004656918d4249db49350515cc2 in nifi's branch refs/heads/master from Scott Aslan [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f678c75 ] [NIFI-6282] manage parameters and parameter contexts [NIFI-6282] when creating a parameter context inline during PG configuration set the newly created parameter context as the selected option [NIFI-6282] If a request to update a parameter context fails, then update the button model to give the user the ability to Apply or Cancel again. [NIFI-6282] address pr review comments [NIFI-6282] remove es6 let [NIFI-6282] update marshall parameters logic and add comments [NIFI-6282] deterministic parameter usage listing, update CS status on PG PC change, expand all twisties by default, remove es6 const [NIFI-6282] update regex and [NIFI-6282] update parameter loading/serialization/marshalling [NIFI-6282] use referencingComponents instead of affectedComponents [NIFI-6282] activate Apply button for sensitive parameter set empty string change [NIFI-6282] fix bug with PG parameters context menu enable [NIFI-6282] only allow delete and recreate of a parameter with equivalent sensitivity [NIFI-6282] display referencing components during parameter management as well as during the parameter context update [NIFI-6282] display no value set in parameter table when parameter value is null [NIFI-6282] - Add ellipsis to referencing component names. - Addressing issues canceling update requests. - Addressing issues with incorrect service scope. - Addressing issue showing the affected parameters. > UI - create parameter context > - > > Key: NIFI-6282 > URL: https://issues.apache.org/jira/browse/NIFI-6282 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Scott Aslan >Assignee: Scott Aslan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] mcgilman commented on issue #3606: [WIP] Nifi 6282
mcgilman commented on issue #3606: [WIP] Nifi 6282 URL: https://github.com/apache/nifi/pull/3606#issuecomment-526314519 @scottyaslan Thanks for the PR! Lot's good stuff here. This has been merged to master. One issue I found that we should address in the future... If the user does not have read access to the Process Group and they attempt to update a parameter context, they can access the Parameters context menu item if we are in the group in question. However, they cannot access the Parameters context menu item if they are in the parent group and select the group in question on the canvas. We likely need to update the back end to include the parameter context id in the process group entity. This is something that can be tracked in a follow on effort. @rfellows Thanks for the review! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (NIFI-6282) UI - create parameter context
[ https://issues.apache.org/jira/browse/NIFI-6282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918886#comment-16918886 ] ASF subversion and git services commented on NIFI-6282: --- Commit f678c75d7004656918d4249db49350515cc2 in nifi's branch refs/heads/master from Scott Aslan [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f678c75 ] [NIFI-6282] manage parameters and parameter contexts [NIFI-6282] when creating a parameter context inline during PG configuration set the newly created parameter context as the selected option [NIFI-6282] If a request to update a parameter context fails, then update the button model to give the user the ability to Apply or Cancel again. [NIFI-6282] address pr review comments [NIFI-6282] remove es6 let [NIFI-6282] update marshall parameters logic and add comments [NIFI-6282] deterministic parameter usage listing, update CS status on PG PC change, expand all twisties by default, remove es6 const [NIFI-6282] update regex and [NIFI-6282] update parameter loading/serialization/marshalling [NIFI-6282] use referencingComponents instead of affectedComponents [NIFI-6282] activate Apply button for sensitive parameter set empty string change [NIFI-6282] fix bug with PG parameters context menu enable [NIFI-6282] only allow delete and recreate of a parameter with equivalent sensitivity [NIFI-6282] display referencing components during parameter management as well as during the parameter context update [NIFI-6282] display no value set in parameter table when parameter value is null [NIFI-6282] - Add ellipsis to referencing component names. - Addressing issues canceling update requests. - Addressing issues with incorrect service scope. - Addressing issue showing the affected parameters. > UI - create parameter context > - > > Key: NIFI-6282 > URL: https://issues.apache.org/jira/browse/NIFI-6282 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Scott Aslan >Assignee: Scott Aslan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6282) UI - create parameter context
[ https://issues.apache.org/jira/browse/NIFI-6282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918878#comment-16918878 ] ASF subversion and git services commented on NIFI-6282: --- Commit f678c75d7004656918d4249db49350515cc2 in nifi's branch refs/heads/master from Scott Aslan [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f678c75 ] [NIFI-6282] manage parameters and parameter contexts [NIFI-6282] when creating a parameter context inline during PG configuration set the newly created parameter context as the selected option [NIFI-6282] If a request to update a parameter context fails, then update the button model to give the user the ability to Apply or Cancel again. [NIFI-6282] address pr review comments [NIFI-6282] remove es6 let [NIFI-6282] update marshall parameters logic and add comments [NIFI-6282] deterministic parameter usage listing, update CS status on PG PC change, expand all twisties by default, remove es6 const [NIFI-6282] update regex and [NIFI-6282] update parameter loading/serialization/marshalling [NIFI-6282] use referencingComponents instead of affectedComponents [NIFI-6282] activate Apply button for sensitive parameter set empty string change [NIFI-6282] fix bug with PG parameters context menu enable [NIFI-6282] only allow delete and recreate of a parameter with equivalent sensitivity [NIFI-6282] display referencing components during parameter management as well as during the parameter context update [NIFI-6282] display no value set in parameter table when parameter value is null [NIFI-6282] - Add ellipsis to referencing component names. - Addressing issues canceling update requests. - Addressing issues with incorrect service scope. - Addressing issue showing the affected parameters. > UI - create parameter context > - > > Key: NIFI-6282 > URL: https://issues.apache.org/jira/browse/NIFI-6282 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Scott Aslan >Assignee: Scott Aslan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6282) UI - create parameter context
[ https://issues.apache.org/jira/browse/NIFI-6282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918881#comment-16918881 ] ASF subversion and git services commented on NIFI-6282: --- Commit f678c75d7004656918d4249db49350515cc2 in nifi's branch refs/heads/master from Scott Aslan [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f678c75 ] [NIFI-6282] manage parameters and parameter contexts [NIFI-6282] when creating a parameter context inline during PG configuration set the newly created parameter context as the selected option [NIFI-6282] If a request to update a parameter context fails, then update the button model to give the user the ability to Apply or Cancel again. [NIFI-6282] address pr review comments [NIFI-6282] remove es6 let [NIFI-6282] update marshall parameters logic and add comments [NIFI-6282] deterministic parameter usage listing, update CS status on PG PC change, expand all twisties by default, remove es6 const [NIFI-6282] update regex and [NIFI-6282] update parameter loading/serialization/marshalling [NIFI-6282] use referencingComponents instead of affectedComponents [NIFI-6282] activate Apply button for sensitive parameter set empty string change [NIFI-6282] fix bug with PG parameters context menu enable [NIFI-6282] only allow delete and recreate of a parameter with equivalent sensitivity [NIFI-6282] display referencing components during parameter management as well as during the parameter context update [NIFI-6282] display no value set in parameter table when parameter value is null [NIFI-6282] - Add ellipsis to referencing component names. - Addressing issues canceling update requests. - Addressing issues with incorrect service scope. - Addressing issue showing the affected parameters. > UI - create parameter context > - > > Key: NIFI-6282 > URL: https://issues.apache.org/jira/browse/NIFI-6282 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Scott Aslan >Assignee: Scott Aslan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6282) UI - create parameter context
[ https://issues.apache.org/jira/browse/NIFI-6282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918879#comment-16918879 ] ASF subversion and git services commented on NIFI-6282: --- Commit f678c75d7004656918d4249db49350515cc2 in nifi's branch refs/heads/master from Scott Aslan [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f678c75 ] [NIFI-6282] manage parameters and parameter contexts [NIFI-6282] when creating a parameter context inline during PG configuration set the newly created parameter context as the selected option [NIFI-6282] If a request to update a parameter context fails, then update the button model to give the user the ability to Apply or Cancel again. [NIFI-6282] address pr review comments [NIFI-6282] remove es6 let [NIFI-6282] update marshall parameters logic and add comments [NIFI-6282] deterministic parameter usage listing, update CS status on PG PC change, expand all twisties by default, remove es6 const [NIFI-6282] update regex and [NIFI-6282] update parameter loading/serialization/marshalling [NIFI-6282] use referencingComponents instead of affectedComponents [NIFI-6282] activate Apply button for sensitive parameter set empty string change [NIFI-6282] fix bug with PG parameters context menu enable [NIFI-6282] only allow delete and recreate of a parameter with equivalent sensitivity [NIFI-6282] display referencing components during parameter management as well as during the parameter context update [NIFI-6282] display no value set in parameter table when parameter value is null [NIFI-6282] - Add ellipsis to referencing component names. - Addressing issues canceling update requests. - Addressing issues with incorrect service scope. - Addressing issue showing the affected parameters. > UI - create parameter context > - > > Key: NIFI-6282 > URL: https://issues.apache.org/jira/browse/NIFI-6282 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Scott Aslan >Assignee: Scott Aslan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6282) UI - create parameter context
[ https://issues.apache.org/jira/browse/NIFI-6282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918875#comment-16918875 ] ASF subversion and git services commented on NIFI-6282: --- Commit f678c75d7004656918d4249db49350515cc2 in nifi's branch refs/heads/master from Scott Aslan [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f678c75 ] [NIFI-6282] manage parameters and parameter contexts [NIFI-6282] when creating a parameter context inline during PG configuration set the newly created parameter context as the selected option [NIFI-6282] If a request to update a parameter context fails, then update the button model to give the user the ability to Apply or Cancel again. [NIFI-6282] address pr review comments [NIFI-6282] remove es6 let [NIFI-6282] update marshall parameters logic and add comments [NIFI-6282] deterministic parameter usage listing, update CS status on PG PC change, expand all twisties by default, remove es6 const [NIFI-6282] update regex and [NIFI-6282] update parameter loading/serialization/marshalling [NIFI-6282] use referencingComponents instead of affectedComponents [NIFI-6282] activate Apply button for sensitive parameter set empty string change [NIFI-6282] fix bug with PG parameters context menu enable [NIFI-6282] only allow delete and recreate of a parameter with equivalent sensitivity [NIFI-6282] display referencing components during parameter management as well as during the parameter context update [NIFI-6282] display no value set in parameter table when parameter value is null [NIFI-6282] - Add ellipsis to referencing component names. - Addressing issues canceling update requests. - Addressing issues with incorrect service scope. - Addressing issue showing the affected parameters. > UI - create parameter context > - > > Key: NIFI-6282 > URL: https://issues.apache.org/jira/browse/NIFI-6282 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Scott Aslan >Assignee: Scott Aslan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6282) UI - create parameter context
[ https://issues.apache.org/jira/browse/NIFI-6282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918872#comment-16918872 ] ASF subversion and git services commented on NIFI-6282: --- Commit f678c75d7004656918d4249db49350515cc2 in nifi's branch refs/heads/master from Scott Aslan [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f678c75 ] [NIFI-6282] manage parameters and parameter contexts [NIFI-6282] when creating a parameter context inline during PG configuration set the newly created parameter context as the selected option [NIFI-6282] If a request to update a parameter context fails, then update the button model to give the user the ability to Apply or Cancel again. [NIFI-6282] address pr review comments [NIFI-6282] remove es6 let [NIFI-6282] update marshall parameters logic and add comments [NIFI-6282] deterministic parameter usage listing, update CS status on PG PC change, expand all twisties by default, remove es6 const [NIFI-6282] update regex and [NIFI-6282] update parameter loading/serialization/marshalling [NIFI-6282] use referencingComponents instead of affectedComponents [NIFI-6282] activate Apply button for sensitive parameter set empty string change [NIFI-6282] fix bug with PG parameters context menu enable [NIFI-6282] only allow delete and recreate of a parameter with equivalent sensitivity [NIFI-6282] display referencing components during parameter management as well as during the parameter context update [NIFI-6282] display no value set in parameter table when parameter value is null [NIFI-6282] - Add ellipsis to referencing component names. - Addressing issues canceling update requests. - Addressing issues with incorrect service scope. - Addressing issue showing the affected parameters. > UI - create parameter context > - > > Key: NIFI-6282 > URL: https://issues.apache.org/jira/browse/NIFI-6282 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Scott Aslan >Assignee: Scott Aslan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6282) UI - create parameter context
[ https://issues.apache.org/jira/browse/NIFI-6282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918882#comment-16918882 ] ASF subversion and git services commented on NIFI-6282: --- Commit f678c75d7004656918d4249db49350515cc2 in nifi's branch refs/heads/master from Scott Aslan [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f678c75 ] [NIFI-6282] manage parameters and parameter contexts [NIFI-6282] when creating a parameter context inline during PG configuration set the newly created parameter context as the selected option [NIFI-6282] If a request to update a parameter context fails, then update the button model to give the user the ability to Apply or Cancel again. [NIFI-6282] address pr review comments [NIFI-6282] remove es6 let [NIFI-6282] update marshall parameters logic and add comments [NIFI-6282] deterministic parameter usage listing, update CS status on PG PC change, expand all twisties by default, remove es6 const [NIFI-6282] update regex and [NIFI-6282] update parameter loading/serialization/marshalling [NIFI-6282] use referencingComponents instead of affectedComponents [NIFI-6282] activate Apply button for sensitive parameter set empty string change [NIFI-6282] fix bug with PG parameters context menu enable [NIFI-6282] only allow delete and recreate of a parameter with equivalent sensitivity [NIFI-6282] display referencing components during parameter management as well as during the parameter context update [NIFI-6282] display no value set in parameter table when parameter value is null [NIFI-6282] - Add ellipsis to referencing component names. - Addressing issues canceling update requests. - Addressing issues with incorrect service scope. - Addressing issue showing the affected parameters. > UI - create parameter context > - > > Key: NIFI-6282 > URL: https://issues.apache.org/jira/browse/NIFI-6282 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Scott Aslan >Assignee: Scott Aslan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi-minifi-cpp] phrocker commented on a change in pull request #630: MINIFICPP-1006: minor updates for jni building using win build & correct artifact naming
phrocker commented on a change in pull request #630: MINIFICPP-1006: minor updates for jni building using win build & correct artifact naming URL: https://github.com/apache/nifi-minifi-cpp/pull/630#discussion_r319214222 ## File path: msi/WixWin.wsi ## @@ -297,34 +316,58 @@ ErrorControl="ignore" Start="auto" Interactive="no" /> - + + + + + +
[GitHub] [nifi-minifi-cpp] phrocker commented on a change in pull request #630: MINIFICPP-1006: minor updates for jni building using win build & correct artifact naming
phrocker commented on a change in pull request #630: MINIFICPP-1006: minor updates for jni building using win build & correct artifact naming URL: https://github.com/apache/nifi-minifi-cpp/pull/630#discussion_r319214222 ## File path: msi/WixWin.wsi ## @@ -297,34 +316,58 @@ ErrorControl="ignore" Start="auto" Interactive="no" /> - + + + + + +
[GitHub] [nifi] dbkegley commented on a change in pull request #3651: NIFI-6539 Nifi stateless init from flow xml
dbkegley commented on a change in pull request #3651: NIFI-6539 Nifi stateless
init from flow xml
URL: https://github.com/apache/nifi/pull/3651#discussion_r319208438
##
File path:
nifi-stateless/nifi-stateless-core/src/main/java/org/apache/nifi/stateless/core/StatelessFlow.java
##
@@ -175,28 +194,32 @@ public StatelessFlow(final VersionedProcessGroup flow,
final ExtensionManager ex
componentMap.put(source.getId(), sourceComponent);
}
break;
+
case REMOTE_INPUT_PORT:
-throw new IllegalArgumentException("Unsupported source
type: " + source.getType());
+throw new IllegalArgumentException("Unsupported source
type: " + source.getId());
Review comment:
Got it. Thanks for the quick response!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] bbende commented on a change in pull request #3679: NIFI-6089 Add Parquet record reader and writer
bbende commented on a change in pull request #3679: NIFI-6089 Add Parquet
record reader and writer
URL: https://github.com/apache/nifi/pull/3679#discussion_r319207274
##
File path:
nifi-nar-bundles/nifi-parquet-bundle/nifi-parquet-processors/src/main/java/org/apache/nifi/parquet/stream/NifiSeekableInputStream.java
##
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.parquet.stream;
+
+import org.apache.nifi.stream.io.ByteCountingInputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.parquet.io.DelegatingSeekableInputStream;
+
+import java.io.IOException;
+
+public class NifiSeekableInputStream extends DelegatingSeekableInputStream {
+
+private final ByteCountingInputStream input;
+
+public NifiSeekableInputStream(final ByteCountingInputStream input) {
+super(input);
+this.input = input;
+this.input.mark(Integer.MAX_VALUE);
+}
+
+@Override
+public long getPos() throws IOException {
+return input.getBytesConsumed();
+}
+
+@Override
+public void seek(long newPos) throws IOException {
+final long currentPos = getPos();
+if (newPos == currentPos) {
+return;
+}
+
+if (newPos > currentPos) {
+// seeking forward so skip ahead the difference from current pos
to new pos
+StreamUtils.skip(input, newPos - getPos());
+} else {
+// seeking backwards so first reset back to beginning of the
stream then seek
+input.reset();
+input.mark(Integer.MAX_VALUE);
+StreamUtils.skip(input, newPos - getPos());
Review comment:
Will update
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] bbende commented on a change in pull request #3679: NIFI-6089 Add Parquet record reader and writer
bbende commented on a change in pull request #3679: NIFI-6089 Add Parquet
record reader and writer
URL: https://github.com/apache/nifi/pull/3679#discussion_r319207591
##
File path:
nifi-nar-bundles/nifi-parquet-bundle/nifi-parquet-processors/src/main/java/org/apache/nifi/parquet/stream/NifiSeekableInputStream.java
##
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.parquet.stream;
+
+import org.apache.nifi.stream.io.ByteCountingInputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.parquet.io.DelegatingSeekableInputStream;
+
+import java.io.IOException;
+
+public class NifiSeekableInputStream extends DelegatingSeekableInputStream {
Review comment:
We could, but seek will be used during the main unit test for the reader and
would fail if seek didn't work
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] bbende commented on a change in pull request #3679: NIFI-6089 Add Parquet record reader and writer
bbende commented on a change in pull request #3679: NIFI-6089 Add Parquet
record reader and writer
URL: https://github.com/apache/nifi/pull/3679#discussion_r319207206
##
File path:
nifi-nar-bundles/nifi-parquet-bundle/nifi-parquet-processors/src/main/java/org/apache/nifi/parquet/utils/ParquetUtils.java
##
@@ -96,18 +97,40 @@
.allowableValues(org.apache.parquet.column.ParquetProperties.WriterVersion.values())
.build();
-public static List COMPRESSION_TYPES =
getCompressionTypes();
+public static final PropertyDescriptor AVRO_READ_COMPATIBILITY = new
PropertyDescriptor.Builder()
+.name("avro-read-compatibility")
+.displayName("Avro Read Compatibility")
+.description("Specifies the value for 'parquet.avro.compatible' in
the underlying Parquet library")
+.allowableValues("true", "false")
+.defaultValue("true")
+.required(true)
+.build();
-public static final PropertyDescriptor COMPRESSION_TYPE = new
PropertyDescriptor.Builder()
-.name("compression-type")
-.displayName("Compression Type")
-.description("The type of compression for the file being written.")
-.allowableValues(COMPRESSION_TYPES.toArray(new AllowableValue[0]))
-.defaultValue(COMPRESSION_TYPES.get(0).getValue())
+public static final String PARQUET_AVRO_ADD_LIST_ELEMENT_RECORDS =
"parquet.avro.add-list-element-records";
+
+public static final PropertyDescriptor AVRO_ADD_LIST_ELEMENT_RECORDS = new
PropertyDescriptor.Builder()
+.name("avro-add-list-element-records")
+.displayName("Avro Add List Element Records")
+.description("Specifies the value for
'parquet.avro.add-list-element-records' in the underlying Parquet library")
+.allowableValues("true", "false")
+.defaultValue("true")
+.required(true)
+.build();
+
+public static final String PARQUET_AVRO_WRITE_OLD_LIST_STRUCTURE =
"parquet.avro.write-old-list-structure";
+
+public static final PropertyDescriptor AVRO_WRITE_OLD_LIST_STRUCTURE = new
PropertyDescriptor.Builder()
+.name("avro-write-old-list-structure")
+.displayName("Avro Write Old List Structure")
+.description("Specifies the value for
'parquet.avro.write-old-list-structure' in the underlying Parquet library")
+.allowableValues("true", "false")
+.defaultValue("true")
.required(true)
.build();
-public static List getCompressionTypes() {
+public static List COMPRESSION_TYPES =
getCompressionTypes();
Review comment:
Will update
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] bbende commented on a change in pull request #3679: NIFI-6089 Add Parquet record reader and writer
bbende commented on a change in pull request #3679: NIFI-6089 Add Parquet
record reader and writer
URL: https://github.com/apache/nifi/pull/3679#discussion_r319207154
##
File path:
nifi-nar-bundles/nifi-parquet-bundle/nifi-parquet-processors/src/main/java/org/apache/nifi/parquet/record/ParquetRecordReader.java
##
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.parquet.record;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.nifi.avro.AvroTypeUtil;
+import org.apache.nifi.parquet.stream.NifiParquetInputFile;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.record.MapRecord;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.io.InputFile;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Map;
+
+public class ParquetRecordReader implements RecordReader {
+
+private GenericRecord lastParquetRecord;
+private RecordSchema recordSchema;
+
+private final InputStream inputStream;
+private final InputFile inputFile;
+private final ParquetReader parquetReader;
+
+public ParquetRecordReader(final InputStream inputStream, final long
inputLength, final Configuration configuration) throws IOException {
+if (inputLength < 0) {
+throw new IllegalArgumentException("Invalid input length of '" +
inputLength + "'. This record reader requires knowing " +
+"the length of the InputStream and cannot be used in some
cases where the length may not be known.");
+}
+
+this.inputStream = inputStream;
+
+inputFile = new NifiParquetInputFile(inputStream, inputLength);
+parquetReader =
AvroParquetReader.builder(inputFile).withConf(configuration).build();
+
+// Read the first record so that we can extract the schema
+lastParquetRecord = parquetReader.read();
+
+// TODO what if lastParquetRecord is null ?
Review comment:
Will update
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] bbende commented on a change in pull request #3679: NIFI-6089 Add Parquet record reader and writer
bbende commented on a change in pull request #3679: NIFI-6089 Add Parquet
record reader and writer
URL: https://github.com/apache/nifi/pull/3679#discussion_r319207098
##
File path:
nifi-nar-bundles/nifi-parquet-bundle/nifi-parquet-processors/src/main/java/org/apache/nifi/parquet/ParquetReader.java
##
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.parquet;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.parquet.record.ParquetRecordReader;
+import org.apache.nifi.parquet.utils.ParquetConfig;
+import org.apache.nifi.parquet.utils.ParquetUtils;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.nifi.parquet.utils.ParquetUtils.applyCommonConfig;
+import static org.apache.nifi.parquet.utils.ParquetUtils.createParquetConfig;
+
+@Tags({"parquet", "parse", "record", "row", "reader"})
+@CapabilityDescription("Parses Parquet data and returns each Parquet record as
an separate Record object. " +
Review comment:
Will update
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (NIFI-6510) Predictive Analytics for NiFi Metrics
[
https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yolanda M. Davis updated NIFI-6510:
---
Assignee: Yolanda M. Davis
Status: Patch Available (was: Open)
> Predictive Analytics for NiFi Metrics
> -
>
> Key: NIFI-6510
> URL: https://issues.apache.org/jira/browse/NIFI-6510
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Andrew Christianson
>Assignee: Yolanda M. Davis
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> From Yolanda's email to the list:
>
> {noformat}
> Currently NiFi has lots of metrics available for areas including jvm and flow
> component usage (via component status) as well as provenance data which NiFi
> makes available either through the UI or reporting tasks (for consumption by
> other systems). Past discussions in the community cite users shipping this
> data to applications such as Prometheus, ELK stacks, or Ambari metrics for
> further analysis in order to capture/review performance issues, detect
> anomalies, and send alerts or notifications. These systems are efficient in
> capturing and helping to analyze these metrics however it requires
> customization work and knowledge of NiFi operations to provide meaningful
> analytics within a flow context.
> In speaking with Matt Burgess and Andy Christianson on this topic we feel
> that there is an opportunity to introduce an analytics framework that could
> provide users reasonable predictions on key performance indicators for flows,
> such as back pressure and flow rate, to help administrators improve
> operational management of NiFi clusters. This framework could offer several
> key features:
> - Provide a flexible internal analytics engine and model api which supports
> the addition of or enhancement to onboard models
> - Support integration of remote or cloud based ML models
> - Support both traditional and online (incremental) learning methods
> - Provide support for model caching (perhaps later inclusion into a model
> repository or registry)
> - UI enhancements to display prediction information either in existing
> summary data, new data visualizations, or directly within the flow/canvas
> (where applicable)
> For an initial target we thought that back pressure prediction would be a
> good starting point for this initiative, given that back pressure detection
> is a key indicator of flow performance and many of the metrics currently
> available would provide enough data points to create a reasonable performing
> model. We have some ideas on how this could be achieved however we wanted to
> discuss this more with the community to get thoughts about tackling this
> work, especially if there are specific use cases or other factors that should
> be considered.{noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (NIFI-6510) Predictive Analytics for NiFi Metrics
[
https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yolanda M. Davis updated NIFI-6510:
---
Fix Version/s: 1.10.0
> Predictive Analytics for NiFi Metrics
> -
>
> Key: NIFI-6510
> URL: https://issues.apache.org/jira/browse/NIFI-6510
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Andrew Christianson
>Assignee: Yolanda M. Davis
>Priority: Major
> Fix For: 1.10.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> From Yolanda's email to the list:
>
> {noformat}
> Currently NiFi has lots of metrics available for areas including jvm and flow
> component usage (via component status) as well as provenance data which NiFi
> makes available either through the UI or reporting tasks (for consumption by
> other systems). Past discussions in the community cite users shipping this
> data to applications such as Prometheus, ELK stacks, or Ambari metrics for
> further analysis in order to capture/review performance issues, detect
> anomalies, and send alerts or notifications. These systems are efficient in
> capturing and helping to analyze these metrics however it requires
> customization work and knowledge of NiFi operations to provide meaningful
> analytics within a flow context.
> In speaking with Matt Burgess and Andy Christianson on this topic we feel
> that there is an opportunity to introduce an analytics framework that could
> provide users reasonable predictions on key performance indicators for flows,
> such as back pressure and flow rate, to help administrators improve
> operational management of NiFi clusters. This framework could offer several
> key features:
> - Provide a flexible internal analytics engine and model api which supports
> the addition of or enhancement to onboard models
> - Support integration of remote or cloud based ML models
> - Support both traditional and online (incremental) learning methods
> - Provide support for model caching (perhaps later inclusion into a model
> repository or registry)
> - UI enhancements to display prediction information either in existing
> summary data, new data visualizations, or directly within the flow/canvas
> (where applicable)
> For an initial target we thought that back pressure prediction would be a
> good starting point for this initiative, given that back pressure detection
> is a key indicator of flow performance and many of the metrics currently
> available would provide enough data points to create a reasonable performing
> model. We have some ideas on how this could be achieved however we wanted to
> discuss this more with the community to get thoughts about tackling this
> work, especially if there are specific use cases or other factors that should
> be considered.{noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (NIFI-6574) Add check for valid predictionIntervalMillis value
[ https://issues.apache.org/jira/browse/NIFI-6574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6574: --- Description: For backpressure prediction the predictionIntervalMillis should have a validity check for value provided. If invalid default value should be set with a warning in logs. (was: For the predictionIntervalMillis, add a validity check for value provided. If invalid default value should be set with an warning in logs.) > Add check for valid predictionIntervalMillis value > -- > > Key: NIFI-6574 > URL: https://issues.apache.org/jira/browse/NIFI-6574 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Yolanda M. Davis >Priority: Minor > > For backpressure prediction the predictionIntervalMillis should have a > validity check for value provided. If invalid default value should be set > with a warning in logs. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] bbende commented on a change in pull request #3679: NIFI-6089 Add Parquet record reader and writer
bbende commented on a change in pull request #3679: NIFI-6089 Add Parquet
record reader and writer
URL: https://github.com/apache/nifi/pull/3679#discussion_r319202703
##
File path:
nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-standard-record-utils/src/main/java/org/apache/nifi/record/listen/StandardSocketChannelRecordReader.java
##
@@ -48,13 +48,13 @@ public StandardSocketChannelRecordReader(final
SocketChannel socketChannel,
}
@Override
-public RecordReader createRecordReader(final FlowFile flowFile, final
ComponentLog logger) throws IOException, MalformedRecordException,
SchemaNotFoundException {
+public RecordReader createRecordReader(final ComponentLog logger) throws
IOException, MalformedRecordException, SchemaNotFoundException {
if (recordReader != null) {
Review comment:
I did attempt to make that change, however it caused failures in the unit
tests for ListenTCPRecord and was unclear at the time if it was just due to the
way the tests were setup, or a real problem. So I decided to leave the reader
being lazily created, but no longer requiring a FlowFile since we can call the
alternate method signature for creating a reader.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] tpalfy commented on a change in pull request #3651: NIFI-6539 Nifi stateless init from flow xml
tpalfy commented on a change in pull request #3651: NIFI-6539 Nifi stateless
init from flow xml
URL: https://github.com/apache/nifi/pull/3651#discussion_r319196394
##
File path:
nifi-stateless/nifi-stateless-core/src/main/java/org/apache/nifi/stateless/core/StatelessFlow.java
##
@@ -175,28 +194,32 @@ public StatelessFlow(final VersionedProcessGroup flow,
final ExtensionManager ex
componentMap.put(source.getId(), sourceComponent);
}
break;
+
case REMOTE_INPUT_PORT:
-throw new IllegalArgumentException("Unsupported source
type: " + source.getType());
+throw new IllegalArgumentException("Unsupported source
type: " + source.getId());
Review comment:
Shouldn't this be `source.getType()`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (NIFI-6585) Update tests to decouple model validation from status analytics & engine validation
[ https://issues.apache.org/jira/browse/NIFI-6585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6585: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Update tests to decouple model validation from status analytics & engine > validation > --- > > Key: NIFI-6585 > URL: https://issues.apache.org/jira/browse/NIFI-6585 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > The Connection Status Analytics object and engine test currently operates > more as an integration test between the model and those instances. We really > should mock the model and the extract functions (since separate tests exist > for that). -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Resolved] (NIFI-6586) Comments and Documentation for Admin Guide
[ https://issues.apache.org/jira/browse/NIFI-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis resolved NIFI-6586. Resolution: Fixed > Comments and Documentation for Admin Guide > -- > > Key: NIFI-6586 > URL: https://issues.apache.org/jira/browse/NIFI-6586 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > > -Update code with comments as needed. > -Include summary of analytics engine as well as configuration guide in admin > docs. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] achristianson opened a new pull request #3681: NIFI-6510 - Analytics framework
achristianson opened a new pull request #3681: NIFI-6510 - Analytics framework
URL: https://github.com/apache/nifi/pull/3681
Description of PR
```
Currently NiFi has lots of metrics available for areas including jvm and
flow component usage (via component status) as well as provenance data which
NiFi makes available either through the UI or reporting tasks (for consumption
by other systems). Past discussions in the community cite users shipping this
data to applications such as Prometheus, ELK stacks, or Ambari metrics for
further analysis in order to capture/review performance issues, detect
anomalies, and send alerts or notifications. These systems are efficient in
capturing and helping to analyze these metrics however it requires
customization work and knowledge of NiFi operations to provide meaningful
analytics within a flow context.
In speaking with Matt Burgess and Andy Christianson on this topic we feel
that there is an opportunity to introduce an analytics framework that could
provide users reasonable predictions on key performance indicators for flows,
such as back pressure and flow rate, to help administrators improve operational
management of NiFi clusters. This framework could offer several key features:
- Provide a flexible internal analytics engine and model api which supports
the addition of or enhancement to onboard models
- Support integration of remote or cloud based ML models
- Support both traditional and online (incremental) learning methods
- Provide support for model caching (perhaps later inclusion into a model
repository or registry)
- UI enhancements to display prediction information either in existing
summary data, new data visualizations, or directly within the flow/canvas
(where applicable)
For an initial target we thought that back pressure prediction would be a
good starting point for this initiative, given that back pressure detection is
a key indicator of flow performance and many of the metrics currently available
would provide enough data points to create a reasonable performing model. We
have some ideas on how this could be achieved however we wanted to discuss this
more with the community to get thoughts about tackling this work, especially if
there are specific use cases or other factors that should be considered
This closes NIFI-6510.
```
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically `master`)?
- [ ] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on both JDK 8 and
JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi-minifi-cpp] phrocker commented on issue #605: MINIFICPP-550 - Implement RocksDB controller service and component st…
phrocker commented on issue #605: MINIFICPP-550 - Implement RocksDB controller service and component st… URL: https://github.com/apache/nifi-minifi-cpp/pull/605#issuecomment-526277276 @bakaid I think this is a pretty important piece of functionality especially as it relates to dealing with state more gracefully on windows (CWEL needs for example ) -- is this ready for review? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] tpalfy commented on a change in pull request #3679: NIFI-6089 Add Parquet record reader and writer
tpalfy commented on a change in pull request #3679: NIFI-6089 Add Parquet
record reader and writer
URL: https://github.com/apache/nifi/pull/3679#discussion_r319080359
##
File path:
nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-standard-record-utils/src/main/java/org/apache/nifi/record/listen/StandardSocketChannelRecordReader.java
##
@@ -48,13 +48,13 @@ public StandardSocketChannelRecordReader(final
SocketChannel socketChannel,
}
@Override
-public RecordReader createRecordReader(final FlowFile flowFile, final
ComponentLog logger) throws IOException, MalformedRecordException,
SchemaNotFoundException {
+public RecordReader createRecordReader(final ComponentLog logger) throws
IOException, MalformedRecordException, SchemaNotFoundException {
if (recordReader != null) {
Review comment:
Shouldn't this recordReader be created in the constructor (as the interface
before the change has alluded to)?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] tpalfy commented on a change in pull request #3679: NIFI-6089 Add Parquet record reader and writer
tpalfy commented on a change in pull request #3679: NIFI-6089 Add Parquet
record reader and writer
URL: https://github.com/apache/nifi/pull/3679#discussion_r319172860
##
File path:
nifi-nar-bundles/nifi-parquet-bundle/nifi-parquet-processors/src/test/java/org/apache/nifi/parquet/TestParquetRecordSetWriter.java
##
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.parquet;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.commons.io.IOUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.schema.access.SchemaAccessUtils;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.record.MapRecord;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.util.MockComponentLog;
+import org.apache.nifi.util.MockConfigurationContext;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.util.HadoopInputFile;
+import org.apache.parquet.io.InputFile;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+
+public class TestParquetRecordSetWriter {
Review comment:
Not sure how relevant but doesn't cover the 2 cases where `avroSchema =
AvroTypeUtil.extractAvroSchema(recordSchema);`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] tpalfy commented on a change in pull request #3679: NIFI-6089 Add Parquet record reader and writer
tpalfy commented on a change in pull request #3679: NIFI-6089 Add Parquet
record reader and writer
URL: https://github.com/apache/nifi/pull/3679#discussion_r319123146
##
File path:
nifi-nar-bundles/nifi-parquet-bundle/nifi-parquet-processors/src/main/java/org/apache/nifi/parquet/record/ParquetRecordReader.java
##
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.parquet.record;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.nifi.avro.AvroTypeUtil;
+import org.apache.nifi.parquet.stream.NifiParquetInputFile;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.record.MapRecord;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.io.InputFile;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Map;
+
+public class ParquetRecordReader implements RecordReader {
+
+private GenericRecord lastParquetRecord;
+private RecordSchema recordSchema;
+
+private final InputStream inputStream;
+private final InputFile inputFile;
+private final ParquetReader parquetReader;
+
+public ParquetRecordReader(final InputStream inputStream, final long
inputLength, final Configuration configuration) throws IOException {
+if (inputLength < 0) {
+throw new IllegalArgumentException("Invalid input length of '" +
inputLength + "'. This record reader requires knowing " +
+"the length of the InputStream and cannot be used in some
cases where the length may not be known.");
+}
+
+this.inputStream = inputStream;
+
+inputFile = new NifiParquetInputFile(inputStream, inputLength);
+parquetReader =
AvroParquetReader.builder(inputFile).withConf(configuration).build();
+
+// Read the first record so that we can extract the schema
+lastParquetRecord = parquetReader.read();
+
+// TODO what if lastParquetRecord is null ?
Review comment:
Unresolved TODO
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] tpalfy commented on a change in pull request #3679: NIFI-6089 Add Parquet record reader and writer
tpalfy commented on a change in pull request #3679: NIFI-6089 Add Parquet
record reader and writer
URL: https://github.com/apache/nifi/pull/3679#discussion_r319145080
##
File path:
nifi-nar-bundles/nifi-parquet-bundle/nifi-parquet-processors/src/main/java/org/apache/nifi/parquet/utils/ParquetUtils.java
##
@@ -96,18 +97,40 @@
.allowableValues(org.apache.parquet.column.ParquetProperties.WriterVersion.values())
.build();
-public static List COMPRESSION_TYPES =
getCompressionTypes();
+public static final PropertyDescriptor AVRO_READ_COMPATIBILITY = new
PropertyDescriptor.Builder()
+.name("avro-read-compatibility")
+.displayName("Avro Read Compatibility")
+.description("Specifies the value for 'parquet.avro.compatible' in
the underlying Parquet library")
+.allowableValues("true", "false")
+.defaultValue("true")
+.required(true)
+.build();
-public static final PropertyDescriptor COMPRESSION_TYPE = new
PropertyDescriptor.Builder()
-.name("compression-type")
-.displayName("Compression Type")
-.description("The type of compression for the file being written.")
-.allowableValues(COMPRESSION_TYPES.toArray(new AllowableValue[0]))
-.defaultValue(COMPRESSION_TYPES.get(0).getValue())
+public static final String PARQUET_AVRO_ADD_LIST_ELEMENT_RECORDS =
"parquet.avro.add-list-element-records";
+
+public static final PropertyDescriptor AVRO_ADD_LIST_ELEMENT_RECORDS = new
PropertyDescriptor.Builder()
+.name("avro-add-list-element-records")
+.displayName("Avro Add List Element Records")
+.description("Specifies the value for
'parquet.avro.add-list-element-records' in the underlying Parquet library")
+.allowableValues("true", "false")
+.defaultValue("true")
+.required(true)
+.build();
+
+public static final String PARQUET_AVRO_WRITE_OLD_LIST_STRUCTURE =
"parquet.avro.write-old-list-structure";
+
+public static final PropertyDescriptor AVRO_WRITE_OLD_LIST_STRUCTURE = new
PropertyDescriptor.Builder()
+.name("avro-write-old-list-structure")
+.displayName("Avro Write Old List Structure")
+.description("Specifies the value for
'parquet.avro.write-old-list-structure' in the underlying Parquet library")
+.allowableValues("true", "false")
+.defaultValue("true")
.required(true)
.build();
-public static List getCompressionTypes() {
+public static List COMPRESSION_TYPES =
getCompressionTypes();
Review comment:
Minor: Could be final.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] tpalfy commented on a change in pull request #3679: NIFI-6089 Add Parquet record reader and writer
tpalfy commented on a change in pull request #3679: NIFI-6089 Add Parquet
record reader and writer
URL: https://github.com/apache/nifi/pull/3679#discussion_r319153864
##
File path:
nifi-nar-bundles/nifi-parquet-bundle/nifi-parquet-processors/src/main/java/org/apache/nifi/parquet/stream/NifiSeekableInputStream.java
##
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.parquet.stream;
+
+import org.apache.nifi.stream.io.ByteCountingInputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.parquet.io.DelegatingSeekableInputStream;
+
+import java.io.IOException;
+
+public class NifiSeekableInputStream extends DelegatingSeekableInputStream {
Review comment:
Maybe a simple unit test would be useful to cover the `seek` method.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] tpalfy commented on a change in pull request #3679: NIFI-6089 Add Parquet record reader and writer
tpalfy commented on a change in pull request #3679: NIFI-6089 Add Parquet
record reader and writer
URL: https://github.com/apache/nifi/pull/3679#discussion_r319088098
##
File path:
nifi-nar-bundles/nifi-parquet-bundle/nifi-parquet-processors/src/main/java/org/apache/nifi/parquet/ParquetReader.java
##
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.parquet;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.parquet.record.ParquetRecordReader;
+import org.apache.nifi.parquet.utils.ParquetConfig;
+import org.apache.nifi.parquet.utils.ParquetUtils;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.nifi.parquet.utils.ParquetUtils.applyCommonConfig;
+import static org.apache.nifi.parquet.utils.ParquetUtils.createParquetConfig;
+
+@Tags({"parquet", "parse", "record", "row", "reader"})
+@CapabilityDescription("Parses Parquet data and returns each Parquet record as
an separate Record object. " +
Review comment:
Minor: Typos: "an separate"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] tpalfy commented on a change in pull request #3679: NIFI-6089 Add Parquet record reader and writer
tpalfy commented on a change in pull request #3679: NIFI-6089 Add Parquet
record reader and writer
URL: https://github.com/apache/nifi/pull/3679#discussion_r319153527
##
File path:
nifi-nar-bundles/nifi-parquet-bundle/nifi-parquet-processors/src/main/java/org/apache/nifi/parquet/stream/NifiSeekableInputStream.java
##
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.parquet.stream;
+
+import org.apache.nifi.stream.io.ByteCountingInputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.parquet.io.DelegatingSeekableInputStream;
+
+import java.io.IOException;
+
+public class NifiSeekableInputStream extends DelegatingSeekableInputStream {
+
+private final ByteCountingInputStream input;
+
+public NifiSeekableInputStream(final ByteCountingInputStream input) {
+super(input);
+this.input = input;
+this.input.mark(Integer.MAX_VALUE);
+}
+
+@Override
+public long getPos() throws IOException {
+return input.getBytesConsumed();
+}
+
+@Override
+public void seek(long newPos) throws IOException {
+final long currentPos = getPos();
+if (newPos == currentPos) {
+return;
+}
+
+if (newPos > currentPos) {
+// seeking forward so skip ahead the difference from current pos
to new pos
+StreamUtils.skip(input, newPos - getPos());
+} else {
+// seeking backwards so first reset back to beginning of the
stream then seek
+input.reset();
+input.mark(Integer.MAX_VALUE);
+StreamUtils.skip(input, newPos - getPos());
Review comment:
Technically correct but maybe writing `StreamUtils.skip(input, newPos);`
would convey the intent better.
(It would also possible to simplify the `if-else` to something like this:
```java
if (newPos < currentPos) {
// seeking backwards so first reset back to beginning of the
stream then seek
input.reset();
input.mark(Integer.MAX_VALUE);
}
StreamUtils.skip(input, newPos - getPos());
```
)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi-minifi-cpp] jdye64 commented on issue #638: MINIFICPP-1018 - Fix linking issues
jdye64 commented on issue #638: MINIFICPP-1018 - Fix linking issues URL: https://github.com/apache/nifi-minifi-cpp/pull/638#issuecomment-526271094 ok - this does satisfy my needs but I'll wait until you look into the parallel builds before merging. Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi-cpp] bakaid edited a comment on issue #638: MINIFICPP-1018 - Fix linking issues
bakaid edited a comment on issue #638: MINIFICPP-1018 - Fix linking issues URL: https://github.com/apache/nifi-minifi-cpp/pull/638#issuecomment-526268424 The fix is not completely ready yet, I myself have seen some failures with parallel builds. I think some dependencies are wrong, I plan to look into it tomorrow. (Which means that unless it is not very urgent, it would be better to merge this after that.) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi-cpp] bakaid commented on issue #638: MINIFICPP-1018 - Fix linking issues
bakaid commented on issue #638: MINIFICPP-1018 - Fix linking issues URL: https://github.com/apache/nifi-minifi-cpp/pull/638#issuecomment-526268424 The fix is not completely ready yet, I myself have seen some failures with parallel builds. I think some dependencies are wrong, I plan to look into it tomorrow. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi-cpp] jdye64 commented on issue #638: MINIFICPP-1018 - Fix linking issues
jdye64 commented on issue #638: MINIFICPP-1018 - Fix linking issues URL: https://github.com/apache/nifi-minifi-cpp/pull/638#issuecomment-526267330 ok. It builds now. Interesting point however. It builds just fine with "make" but takes forever. If I try to run make -j16 however it fails fast. This is a beast of a workstation coming in at about $20K so wouldn't expect to see if fail like that. I'm running dual Intel Xeon Platinums as well so that was the reason for the -j16 to consume a single processor. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi-cpp] bakaid opened a new pull request #638: MINIFICPP-1018 - Fix linking issues
bakaid opened a new pull request #638: MINIFICPP-1018 - Fix linking issues URL: https://github.com/apache/nifi-minifi-cpp/pull/638 Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi-cpp] arpadboda edited a comment on issue #636: Minificpp-780 - Change GenerateFlowFile to allow 0b content FlowFiles
arpadboda edited a comment on issue #636: Minificpp-780 - Change GenerateFlowFile to allow 0b content FlowFiles URL: https://github.com/apache/nifi-minifi-cpp/pull/636#issuecomment-526239247 While checking this processor I found the following issues: -In case batch count is greater than one, the flowfiles generated in one batch are not unique -Handling of 0B size was error-prone: allocating zero size arrays, passing null pointers to callbacks, etc -A potential memory segmentation violation in case the file size is not multiple of 4. Because of these it was easier to completely rewrite the logic than fixing it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi-cpp] arpadboda commented on issue #636: Minificpp-780 - Change GenerateFlowFile to allow 0b content FlowFiles
arpadboda commented on issue #636: Minificpp-780 - Change GenerateFlowFile to allow 0b content FlowFiles URL: https://github.com/apache/nifi-minifi-cpp/pull/636#issuecomment-526239247 While checking this processor I found the following issues: -In case batch count is greater than one, the flowfiles generated in one batch are not unique -Handling of 0B size was error-prone: allocating zero size arrays, passing null pointers to callbacks, etc -A potential memory segmentation violation in case the file size is not multiple of 4. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi-cpp] arpadboda commented on issue #636: Minificpp-780 - Change GenerateFlowFile to allow 0b content FlowFiles
arpadboda commented on issue #636: Minificpp-780 - Change GenerateFlowFile to allow 0b content FlowFiles URL: https://github.com/apache/nifi-minifi-cpp/pull/636#issuecomment-526237971 WARNING: this PR depends on #637 - that one should be merged first, otherwise tests included in this are going to fail. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi-cpp] arpadboda opened a new pull request #637: MINIFICPP-1020 - PutFile fails to create empty files
arpadboda opened a new pull request #637: MINIFICPP-1020 - PutFile fails to create empty files URL: https://github.com/apache/nifi-minifi-cpp/pull/637 Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (MINIFICPP-1020) PutFile fails to create empty files
Arpad Boda created MINIFICPP-1020: - Summary: PutFile fails to create empty files Key: MINIFICPP-1020 URL: https://issues.apache.org/jira/browse/MINIFICPP-1020 Project: Apache NiFi MiNiFi C++ Issue Type: Bug Affects Versions: 0.6.0 Reporter: Arpad Boda Assignee: Arpad Boda Fix For: 0.7.0 PutFile processor fails to create empty files (in case the incoming flowfile has no content) and routes to failure (also rolls back session). -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi-minifi-cpp] arpadboda opened a new pull request #636: Minificpp 780
arpadboda opened a new pull request #636: Minificpp 780 URL: https://github.com/apache/nifi-minifi-cpp/pull/636 Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (NIFI-6588) Move graph bundle to profile in assembly
[ https://issues.apache.org/jira/browse/NIFI-6588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Bende resolved NIFI-6588. --- Fix Version/s: 1.10.0 Resolution: Fixed > Move graph bundle to profile in assembly > > > Key: NIFI-6588 > URL: https://issues.apache.org/jira/browse/NIFI-6588 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Major > Fix For: 1.10.0 > > Time Spent: 1h > Remaining Estimate: 0h > > To save space, we'll move the entire graph bundle into an optional profile. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6588) Move graph bundle to profile in assembly
[ https://issues.apache.org/jira/browse/NIFI-6588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918689#comment-16918689 ] ASF subversion and git services commented on NIFI-6588: --- Commit 2e554cbb67bd96e0a38a202e96cd9a58de5260a0 in nifi's branch refs/heads/master from Mike Thomsen [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=2e554cb ] NIFI-6588 Updated assembly pom file to move all graph components into a profile to save space in the convenience binary. > Move graph bundle to profile in assembly > > > Key: NIFI-6588 > URL: https://issues.apache.org/jira/browse/NIFI-6588 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > To save space, we'll move the entire graph bundle into an optional profile. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] bbende merged pull request #3669: NIFI-6588 Updated assembly pom file to move all graph components into…
bbende merged pull request #3669: NIFI-6588 Updated assembly pom file to move all graph components into… URL: https://github.com/apache/nifi/pull/3669 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] bbende commented on issue #3669: NIFI-6588 Updated assembly pom file to move all graph components into…
bbende commented on issue #3669: NIFI-6588 Updated assembly pom file to move all graph components into… URL: https://github.com/apache/nifi/pull/3669#issuecomment-526224788 Looks good, merging This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] scottyaslan commented on issue #3636: [WIP] Allow parameters to be used in property configuration
scottyaslan commented on issue #3636: [WIP] Allow parameters to be used in property configuration URL: https://github.com/apache/nifi/pull/3636#issuecomment-526221011 Reviewing... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (NIFI-6025) Track Enabled/Disabled State in Registry
[ https://issues.apache.org/jira/browse/NIFI-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Bende updated NIFI-6025: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Track Enabled/Disabled State in Registry > > > Key: NIFI-6025 > URL: https://issues.apache.org/jira/browse/NIFI-6025 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.8.0 >Reporter: Alan Jackoway >Assignee: Mark Payne >Priority: Major > Labels: SDLC > Fix For: 1.10.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > We often have little chunks of pipelines that are used for debugging/fixing > things that went wrong/etc. that we want to disable. I would love for > disabled/enabled to be a thing that gets committed to registry, but my little > test says it isn't. It would be nice for disabled state to be persisted in > registry so that we can differentiate between things that should usually be > disabled and things that are just stopped. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6025) Track Enabled/Disabled State in Registry
[ https://issues.apache.org/jira/browse/NIFI-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918667#comment-16918667 ] ASF subversion and git services commented on NIFI-6025: --- Commit 4ae1fec78a77be5b9b06bd79fa22ae020f3365bc in nifi's branch refs/heads/master from Mark Payne [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=4ae1fec ] NIFI-6025: Include Processor 'scheduled state' (i.e., Enabled or Disabled) in the VersionedProcessor when pushing to Flow Registry and take into account when updating flows on the NiFi side NIFI-6025: Include difference in Scheduled State as a Local Flow Difference This closes #3546. Signed-off-by: Bryan Bende > Track Enabled/Disabled State in Registry > > > Key: NIFI-6025 > URL: https://issues.apache.org/jira/browse/NIFI-6025 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.8.0 >Reporter: Alan Jackoway >Assignee: Mark Payne >Priority: Major > Labels: SDLC > Fix For: 1.10.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > We often have little chunks of pipelines that are used for debugging/fixing > things that went wrong/etc. that we want to disable. I would love for > disabled/enabled to be a thing that gets committed to registry, but my little > test says it isn't. It would be nice for disabled state to be persisted in > registry so that we can differentiate between things that should usually be > disabled and things that are just stopped. -- This message was sent by Atlassian Jira (v8.3.2#803003)
