[GitHub] nifi issue #2827: NIFI-5354 Changing ranger.version to 1.0.0 and ranger.hado...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/2827 @bbende thanks Bryan I will review shortly ---
[GitHub] nifi issue #2628: NIFI-4942 Add capability for encrypt-config tool to use se...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/2628 Spoke with @alopresto offline. He highlighted that is it important to maintain control of location of secure-hash.key file in order to prevent calling application from piping into a file that is controlled externally. We want to keep it as secured as possible. Concerning the documentation update recommendation that can be addressed in a separate item. +1 Will merge shortly ---
[GitHub] nifi issue #2628: NIFI-4942 Add capability for encrypt-config tool to use se...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/2628 @alopresto ran through test cases and this works as expected. One question for you is there a way to designate an output location for the secure-hash.key file? Such as if I want to pipe it to stdin or just to another location? Also as a side note I tested with -p (password) input where it may contain certain characters (@, -, and &). The '&' caused the script to fail but also to stall, needed to do a Control-C to break out of it. The below is resolved by simply including quotes around the password but may be good to document for users: `ydavis$ /Users/ydavis/dev/projects/nifi/nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.7.0-SNAPSHOT-bin/nifi-toolkit-1.7.0-SNAPSHOT/bin/encrypt-config.sh -v -m -b bootstrap.conf -n nifi-migrated-from-hash-key-break-2.properties -o nifi-migrated-from-hash-key-break-3.properties -p thisIs&ABadPassword4 -y '$s0$100801$j8z9NeI9DZEBTbCzOaQJbA$MI0iN/ZPQ5bk4YxcgJ2H95gCToQy3ZbIr7B6OMxB3oA' [1] 5576 -bash: ABadPassword4: command not found HW13535:conf ydavis$ 2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of nifi.properties 2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool:bootstrap.conf: bootstrap.conf 2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) nifi.properties: nifi-migrated-from-hash-key-break-2.properties 2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) nifi.properties: nifi-migrated-from-hash-key-break-3.properties 2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) login-identity-providers.xml: null 2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) login-identity-providers.xml: null 2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) authorizers.xml: null 2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) authorizers.xml: null 2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) flow.xml.gz: null 2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) flow.xml.gz: null 2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Key migration mode activated 2018/04/12 17:30:13 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 151 properties from /Users/ydavis/dev/tools/nifi-1.7.0-SNAPSHOT/conf/nifi-migrated-from-hash-key-break-2.properties 2018/04/12 17:30:13 ERROR [main] org.apache.nifi.properties.ConfigEncryptionTool: Encountered an error java.security.KeyException: Cannot derive key from empty/short password -- password must be at least 12 characters at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83) at org.codehaus.groovy.reflection.CachedConstructor.doConstructorInvoke(CachedConstructor.java:77) at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrap.callConstructor(ConstructorSite.java:84) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:60) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:235) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:247) at org.apache.nifi.properties.ConfigEncryptionTool.deriveKeyFromPassword(ConfigEncryptionTool.groovy:1493) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite$StaticMetaMethodSiteNoUnwrapNoCoerce.invoke(StaticMetaMethodSite.java:151) at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.callStatic(StaticMetaMethodSite.java:102) at org.codehaus.groovy.runtime.callsite.CallSiteArray.default
[GitHub] nifi issue #2628: NIFI-4942 Add capability for encrypt-config tool to use se...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/2628 @alopresto thanks for addressing this, happy to review. It does look like Travis is failing on a ratcheck related error in nifi-toolkit-encrypt-config ---
[GitHub] nifi pull request #1997: NIFI-4164 Adding a realistic time simulator process...
Github user YolandaMDavis commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1997#discussion_r166648706
--- Diff:
nifi-nar-bundles/nifi-simulator-bundle/nifi-simulator-processors/src/main/java/com/apache/nifi/processors/simulator/GenerateTimeSeriesFlowFile.java
---
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.apache.nifi.processors.simulator;
+
+import be.cetic.tsimulus.config.Configuration;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.joda.time.LocalDateTime;
+import scala.Some;
+import scala.Tuple3;
+import scala.collection.JavaConverters;
+
+import java.util.List;
+import java.util.Set;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.ArrayList;
+
+@Tags({"Simulator, Timeseries, IOT, Testing"})
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@CapabilityDescription("Generates realistic time series data using the
TSimulus time series generator, and places the values into the flowfile in a
CSV format.")
+public class GenerateTimeSeriesFlowFile extends AbstractProcessor {
+
+private Configuration simConfig = null;
+private boolean isTest = false;
+
+public static final PropertyDescriptor SIMULATOR_CONFIG = new
PropertyDescriptor
+.Builder().name("SIMULATOR_CONFIG")
+.displayName("Simulator Configuration File")
+.description("The JSON configuration file to use to configure
TSimulus")
+.required(true)
+.addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PRINT_HEADER = new
PropertyDescriptor
+.Builder().name("PRINT_HEADER")
+.displayName("Print Header")
+.description("Directs the processor whether to print a header
line or not.")
+.required(true)
+.allowableValues("true", "false")
+.defaultValue("false")
+.addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+.build();
+
+public static final Relationship SUCCESS = new Relationship.Builder()
+.name("Success")
+.description("When the flowfile is successfully generated")
+.build();
+
+private List descriptors;
+
+private Set relationships;
+
+@Override
+protected void init(final ProcessorInitializationContext context) {
+
+final List descriptors = new ArrayList<>();
+
+descriptors.add(SIMULATOR_CONFIG);
+descriptors.add(PRINT_HEADER);
+
+this.descriptors = Collections.unmodifiableList(descriptors);
+
+final Set relationships = new HashSet<>();
+relationships.add(SUCCESS);
+this.relationships = Collections.unmodifiableSet(relationships);
+}
+
+@Override
+public Set getRelationships() {
+return this.relationships;
+}
+
+@
[GitHub] nifi issue #2107: NIFI-4310 - added changes to support detection of reportin...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/2107 @markap14 thank you! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2065: NIFI-4255 - added flag to allow migration of existing (sou...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/2065 thank you @jtstorck ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #2107: NIFI-4310 - added changes to support detection of r...
GitHub user YolandaMDavis opened a pull request: https://github.com/apache/nifi/pull/2107 NIFI-4310 - added changes to support detection of reporting tasks and⦠⦠controller services during isEmpty flow check. Added testing scenarios Recommend the following tests scenarios are included: - Confirm cluster starts after flow election with empty flows on all nodes - Confirm start after election initiated for cluster having all but one node containing empty flows (one remaining node contains flow populated with root process group) - Confirm start after election initiated for cluster containing 1/2 nodes with empty flows (remaining halves would contain flows with only controller services). - Confirm start after election initiated for cluster containing 1/2 nodes with empty flows (remaining halves would contain flows with only reporting tasks). Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ x] Is your initial contribution a single, squashed commit? ### For code changes: - [x ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/YolandaMDavis/nifi NIFI-4310 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2107.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2107 commit 028e381e281d982cc6683c9a0949a3dc820b55d7 Author: Yolanda M. Davis Date: 2017-08-23T05:53:49Z NIFI-4310 - added changes to support detection of reporting tasks and controller services during isEmpty flow check. Added testing scenarios --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2065: NIFI-4255 - added flag to allow migration of existing (sou...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/2065 spoke with @jtstorck offline and we discussed refactoring a bit to ensure that the logic for determining acls are externalized and staged within the CompletableFuture chain. Committing those changes shortly --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #2065: NIFI-4255 - added flag to allow migration of existi...
Github user YolandaMDavis commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2065#discussion_r132969031
--- Diff:
nifi-toolkit/nifi-toolkit-zookeeper-migrator/src/main/java/org/apache/nifi/toolkit/zkmigrator/ZooKeeperMigrator.java
---
@@ -269,15 +269,23 @@ private String ensureNodeExists(ZooKeeper zooKeeper,
String path, CreateMode cre
}
}
-private DataStatAclNode transformNode(DataStatAclNode node, AuthMode
destinationAuthMode) {
-// For the NiFi use case, all nodes will be migrated to
CREATOR_ALL_ACL
+private DataStatAclNode transformNode(DataStatAclNode node, AuthMode
destinationAuthMode, boolean useExistingACL) {
+// If useExistingACL is false, for the NiFi use cases, all nodes
will be migrated to CREATOR_ALL_ACL
final DataStatAclNode migratedNode = new
DataStatAclNode(node.getPath(), node.getData(), node.getStat(),
-destinationAuthMode.equals(AuthMode.OPEN) ?
ZooDefs.Ids.OPEN_ACL_UNSAFE : ZooDefs.Ids.CREATOR_ALL_ACL,
+determineNodeACL(node,destinationAuthMode,useExistingACL),
node.getEphemeralOwner());
LOGGER.info("transformed original node {} to {}", node,
migratedNode);
return migratedNode;
}
+private List determineNodeACL(DataStatAclNode node, AuthMode
destinationAuthMode, boolean useExistingACL){
--- End diff --
@jtstorck Rather than have a separate stage I ended up combining this with
the previous method. Didn't really see the value of me having it separated out
in the first place .
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #2065: NIFI-4255 - added flag to allow migration of existi...
Github user YolandaMDavis commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2065#discussion_r132821753
--- Diff:
nifi-toolkit/nifi-toolkit-zookeeper-migrator/src/main/java/org/apache/nifi/toolkit/zkmigrator/ZooKeeperMigrator.java
---
@@ -269,15 +269,23 @@ private String ensureNodeExists(ZooKeeper zooKeeper,
String path, CreateMode cre
}
}
-private DataStatAclNode transformNode(DataStatAclNode node, AuthMode
destinationAuthMode) {
-// For the NiFi use case, all nodes will be migrated to
CREATOR_ALL_ACL
+private DataStatAclNode transformNode(DataStatAclNode node, AuthMode
destinationAuthMode, boolean useExistingACL) {
+// If useExistingACL is false, for the NiFi use cases, all nodes
will be migrated to CREATOR_ALL_ACL
final DataStatAclNode migratedNode = new
DataStatAclNode(node.getPath(), node.getData(), node.getStat(),
-destinationAuthMode.equals(AuthMode.OPEN) ?
ZooDefs.Ids.OPEN_ACL_UNSAFE : ZooDefs.Ids.CREATOR_ALL_ACL,
+determineNodeACL(node,destinationAuthMode,useExistingACL),
node.getEphemeralOwner());
LOGGER.info("transformed original node {} to {}", node,
migratedNode);
return migratedNode;
}
+private List determineNodeACL(DataStatAclNode node, AuthMode
destinationAuthMode, boolean useExistingACL){
--- End diff --
Sure no problem
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #2065: NIFI-4255 - added flag to allow migration of existi...
GitHub user YolandaMDavis opened a pull request: https://github.com/apache/nifi/pull/2065 NIFI-4255 - added flag to allow migration of existing (source) acls Testing Recommendations: Using a secured Zookeeper instance with predefined paths/znodes use the ZK Migrator to first export a json file containing nodes information. Then that json file can be edited to alter or add acls to existing paths. The file can then be reapplied to the source ZK server using the following example command: `./zk-migrator.sh -s -z :2181/nifi -f -k --ignore-source --use-existing-acl` Confirm that acls are applied as expected using the zkCli.sh (or some other ZK Client tool) Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/YolandaMDavis/nifi NIFI-4255 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2065.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2065 commit d4f4c78d669d6d2ab30692af15de7606191da545 Author: Yolanda M. Davis Date: 2017-08-08T04:03:42Z NIFI-4255 - added flag to allow migration of existing (source) acls to destination NIFI-4255 - update to documentation --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2046: NIFI-4022 - Enabled SASL auth scheme/ACL support for Curat...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/2046 @bbende thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #2046: NIFI-4022 - Enabled SASL auth scheme/ACL support fo...
GitHub user YolandaMDavis opened a pull request: https://github.com/apache/nifi/pull/2046 NIFI-4022 - Enabled SASL auth scheme/ACL support for Curator use Enhancement allows user to enable SASL based ACL's for nodes created via Curator for cluster management (e.g. leader election nodes, Cluster Coordinator/Primary Nodes). For testing would recommend the following actions: 1) Follow the updated administrator guide (included in PR as a separate commit) for enabling kerberos on Zookeeper (external or embedded) and NiFi 2)Testing with nifi nodes where principals vary across servers. For example nifi/instan...@realm.com vs nifi/instan...@realm.com. In this case the kerberos.removeHostFromPrincipal would need to be true (in both zookeeper.properties and nifi.properties) to ensure that the user will be normalized as n...@realm.com for acls. 3) Ensuring leader election scenarios work as expected with acls in place on the /nifi path (acl should be 'sasl', cdrwa and 'world', anyone r). Recommended scenario is removal of Cluster Coordinator from a cluster to ensure new coordinator is elected. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/YolandaMDavis/nifi NIFI-4022 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2046.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2046 commit 9e43229ed409527ffe3bab0b3bdb7584e64ce98e Author: Yolanda M. Davis Date: 2017-07-31T17:27:48Z NIFI-4022 - Initial update for SASL support for cluster management in Zookeeper commit 588a5ca995c46f94e893b249a787be7c8104e060 Author: Yolanda M. Davis Date: 2017-08-01T18:31:15Z NIFI-4022 - adding sasl documentation update and update to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2019: NIFI-4032: Managed Ranger Authorizer
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/2019 @mcgilman with the latest commit I was able to add a new node and see the users.xml file populated. The node started successfully and I was able to access the cluster without issue. +1 Will merge into master shortly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2019: NIFI-4032: Managed Ranger Authorizer
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/2019 I've worked through 3 Ranger configuration scenarios that leveraged the ldap user group provider, or the composite configurable user group provider (pairing the ldap provider with the file provider): 1) Using group authorizations for LDAP users (with no mapping for identities) alongside user authorizations for nodes . This is to cover cases where node identities may not be present in LDAP 2) Using mapped identities to ensure that user-group associations would still be properly resolved 3) Using the Composite Configurable User Group Provider to allow maintenance of node identities and groups in NiFi while allowing policies to be enforced via Ranger All three scenarios worked well with an established cluster. I was able to go from one scenarios to the next through changing configurations and updating policies without issue. However a bug was encountered on the third test case when I wanted to add a new node to the cluster. The process of adding a new node requires that no information that would seed the users.xml file be provided in configurations (e.g. Initial Admin, Node Identifiers, etc). Therefore the expectation is once the node attempts to join the cluster it would receive the necessary user information from the cluster to create it's own local version of the file. When using the ManagedRangerAuthorizer along with the Configurable provider it doesn't appear to have that functionality, since the users.xml generated was empty. This led to the node starting up fine however when attempting to access the UI from any node a proxy error occurred. Given the users.xml file was empty this error made sense because NiFi was unable to determine the users (node identities) or groups they should be mapped to, hence unable to apply the Ranger policy that allowed the nodes group to perform proxying. In speaking with @mcgilman offline this error was due to the ManagedRangerAuthorizer not extracting user group information for cases when it's paired with configurable user group providers. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2019: NIFI-4032: Managed Ranger Authorizer
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/2019 @mcgilman will take a look soon --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1938: NIFI-1763: Initial implementation of ConfluentSchemaRegist...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1938 @markap14 that now looks fixed. I was able to use the Confluence schema write strategy without issue. All is working as expected. +1 will merge in shortly --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1938: NIFI-1763: Initial implementation of ConfluentSchemaRegist...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1938 @markap14 thanks will check it out --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1938: NIFI-1763: Initial implementation of ConfluentSchemaRegist...
Github user YolandaMDavis commented on the issue:
https://github.com/apache/nifi/pull/1938
@markap14 Iâm not quite sure if I ran into a configuration problem on my
part or if this is truly an issue. Iâm testing the ConfluentSchemaRegistry
and do have it set up in NiFi with two small flows that places data onto a
kafka topic and reads from that same topic then saves data into the filesystem.
I did confirm that the Controller Service itself appears to work since I was
able to register a schema and get an AvroReader/RecordSetWriter and a
JsonTreeReader to refer to the schema within that registry. However when I
wanted to use the option to write the Schema using the Confluent Schema
Registry Reference (specifically within the AvroRecordSetWriter in my test)
Iâm getting a NullPointerException:
`2017-07-06 05:48:35,364 ERROR [Timer-Driven Process Thread-8]
o.a.n.p.k.pubsub.PublishKafkaRecord_0_10
PublishKafkaRecord_0_10[id=16230624-015d-1000-7474-292c6c453e5a] Failed to send
all message for
StandardFlowFileRecord[uuid=007abeb3-5d2e-4c11-b9ef-e279b7822348,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=149931562458
4-1, container=default, section=1], offset=683586,
length=20],offset=0,name=11741764036465,size=20] to Kafka; routing to failure
due to java.lang.NullPointerException: {}
java.lang.NullPointerException: null
at
org.apache.nifi.serialization.SchemaRegistryRecordSetWriter.getSchemaAccessWriter(SchemaRegistryRecordSetWriter.java:136)
at
org.apache.nifi.avro.AvroRecordSetWriter.createWriter(AvroRecordSetWriter.java:107)
at sun.reflect.GeneratedMethodAccessor374.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:89)
at com.sun.proxy.$Proxy131.createWriter(Unknown Source)
at
org.apache.nifi.processors.kafka.pubsub.PublisherLease.publish(PublisherLease.java:109)
at
org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_0_10$1.process(PublishKafkaRecord_0_10.java:338)
at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2136)
at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2106)
at
org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_0_10.onTrigger(PublishKafkaRecord_0_10.java:330)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)`
I did see a note about potentially experiencing an exception if the schema
didnât include the identifier and version however I wasnât quite expecting
the exception to be a Nullpointer. Is this expected behavior? I did confirm
that if I used another option (e.g. Set schema.name or Do Not Write Schema)
that I did not encounter this problem.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1956: NIFI-4135 - added hadoop-client and enhanced Authorizers e...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1956 @bbende did add the change to order (couldn't recreate the issue). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1938: NIFI-1763: Initial implementation of ConfluentSchemaRegist...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1938 @markap14 I can try this out as well --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1956: NIFI-4135 - added hadoop-client and enhanced Authorizers e...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1956 @bbende sorry for the delay (was away on vacation). Not a picky request at all; the weirdness I ran into was that the xsd could not resolve the property value if classpath was just before it. It is weird but I think it has something to do with property being unlimited and classpath being not required. I'll give it a try to see if I can resolve or at least recreate it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1957: NIFI-4137: Add support for UTF-8 in GetFTP and PutFTP
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1957 Ran tests using FileZilla on Windows 10 Server with flow containing GetFTP -> UpdateAttributes (for filename append) -> PutFTP. Worked as expected with the below use cases: Below demonstrates cases where UTF-8 was disabled then enabled  Files were stored as expected on FTP server:  +1 will merge shortly --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1957: NIFI-4137: Add support for UTF-8 in GetFTP and PutFTP
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1957 @mattyb149 will review --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1956: NIFI-4135 - added hadoop-client and enhanced Authorizers e...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1956 @bbende would appreciate if you could review. Thanks in advance! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1956: NIFI-4135 - added hadoop-client and enhanced Author...
GitHub user YolandaMDavis opened a pull request: https://github.com/apache/nifi/pull/1956 NIFI-4135 - added hadoop-client and enhanced Authorizers entity to su⦠â¦pport classpath for resources entry Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/YolandaMDavis/nifi NIFI-4135 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1956.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1956 commit ade54f9104186e1adf81875a8b777594a5aa7fe8 Author: Yolanda M. Davis Date: 2017-06-28T03:24:04Z NIFI-4135 - added hadoop-client and enhanced Authorizers entity to support classpath for resources entry --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1889: NIFI-3696 - Initial commit of ConfigMigration and FileMana...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1889 @brosander I've added updates to documentation as well as additional verbose information as recommended. please let me know if you see any additional issues --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1889: NIFI-3696 - Initial commit of ConfigMigration and F...
Github user YolandaMDavis commented on a diff in the pull request: https://github.com/apache/nifi/pull/1889#discussion_r120722963 --- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc --- @@ -1317,6 +1318,82 @@ cluster if restarted and the flow for the cluster has not changed. If the flow w the removed node should be deleted before restarting the node to allow it to obtain the cluster flow (otherwise an uninheritable flow file exception may occur). +=== File Manager + +The File Manager utility allows system administrators to take a backup of an existing NiFi installation, install a new version of NiFi +in a designated location (while migrating any previous configuration settings) or restore an installation from a previous backup. +File Manager supports NiFi version 1.0.0 and higher and is available in 'file-manager.bat' file for use on Windows machines. + +To show help: + + file-manager.sh -h + +The following are available options: + +* `-o,--operation ` File operation (install | backup | restore) +* `-b,--backupDir ` Backup NiFi Directory (used with backup or restore operation) +* `-c,--nifiCurrentDir ` Current NiFi Installation Directory (used optionally with install or restore operation) +* `-d,--nifiInstallDir ` NiFi Installation Directory (used with install or restore operation) +* `-i,--installFile ` NiFi Install File (used with install operation) +* `-r,--nifiRollbackDir ` NiFi Installation Directory (used with install or restore operation) +* `-t,--bootstrapConf ` Current NiFi Bootstrap Configuration File (used optionally) +* `-m,--moveRepositories` Allow repositories to be moved to new/restored nifi directory from existing installation, if available (used optionally with install or restore operation) +* `-x,--overwriteConfigs` Overwrite existing configuration directory with upgrade changes (used optionally with install or restore operation) +* `-v,--verbose` Verbose messaging (optional) +* `-h,--help` Print help info (optional) + +Example usage on Linux: + + # backup NiFi installation + # option -t may be provided to ensure backup of external boostrap.conf file + ./file-manager.sh + -o backup + âb /tmp/nifi_bak + âc /usr/nifi_old + -v + + # install NiFi using compressed tar file into /usr/nifi directory (should install as /usr/nifi/nifi-1.3.0). --- End diff -- @brosander agreed will update the admin guide and will also add verbose information during install execution. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1889: NIFI-3696 - initial commit of ConfigMigration and F...
GitHub user YolandaMDavis opened a pull request: https://github.com/apache/nifi/pull/1889 NIFI-3696 - initial commit of ConfigMigration and FileManager tools Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/YolandaMDavis/nifi NIFI-3696 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1889.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1889 commit f347869ceefd65d570fa5f9ab6d8daa6db2f4b57 Author: Yolanda M. Davis Date: 2017-05-31T14:17:23Z NIFI-3696 - initial commit of ConfigMigration and FileManager tools --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1866: NIFI-3984 - upgraded version to 0.2.1
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1866 @bbende can you please review? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1866: NIFI-3984 - upgraded version to 0.2.1
GitHub user YolandaMDavis opened a pull request: https://github.com/apache/nifi/pull/1866 NIFI-3984 - upgraded version to 0.2.1 Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/YolandaMDavis/nifi NIFI-3984 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1866.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1866 commit faceb5a8b278ce59c06e5f28bf0042cd6cdd1976 Author: Yolanda M. Davis Date: 2017-05-26T17:49:24Z NIFI-3984 - upgraded version --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1786: NIFI-2199 & NIFI-3112 - Handle propagation of stderr and a...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1786 @apiri validated ssh command on remote host with with start/stop/restart. All worked as expected, with NiFi starting appropriately and the IO stream for ssh was closed. Also validated that bootstrap errors are appropriately routed to the output stream and displayed before stream is closed. +1 will merge shortly. Thanks Aldrin! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1786: NIFI-2199 & NIFI-3112 - Handle propagation of stderr and a...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1786 @apiri will review --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1748: NIFI-3793 - switch to use project.version
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1748 @joewitt thanks switched to refer to 1.2.0-SNAPSHOT --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1748: NIFI-3793 - switch to use project.version
GitHub user YolandaMDavis opened a pull request: https://github.com/apache/nifi/pull/1748 NIFI-3793 - switch to use project.version Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/YolandaMDavis/nifi NIFI-3793 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1748.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1748 commit a50f71498bf57e191a24c1967bdd7d42ba3110f9 Author: Yolanda M. Davis Date: 2017-05-04T02:08:43Z NIFI-3793 - switch to use project.version --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1736: Nifi 3774
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1736 @thesolson did the merge but corrected one small thing, tick marks on controller were still missing. hope is was ok to apply. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1736: Nifi 3774
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1736 Thanks Sarah for the updates +1 Will squash and merge this shortly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1736: Nifi 3774
Github user YolandaMDavis commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1736#discussion_r114452389
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1176,6 +1176,146 @@ Secure Prompt
If you prefer not to provide the password or raw key in the command-line
invocation of the tool, leaving these arguments absent will prompt a secure
console read of the password (by default) or raw key (if the `-r` flag is
provided at invocation).
+[[admin-toolkit]]
+Administrative Tools
+
+The admin toolkit contains command line utilities for administrators to
support NiFi maintenance in standalone
+and clustered environments. These utilities include:
+
+* Notify -- The notification tool allows administrators to send bulletins
to the NiFi UI using the command line.
+* Node Manager -- The node manager tool allows administrators to perform a
status check on a node as well as to connect, disconnect, or remove nodes that
are part of a cluster.
+
+The admin toolkit is bundled with the nifi-toolkit and can be executed
with scripts found in the _bin_ folder.
+
+=== Prerequisites for Running Admin Toolkit in a Secure Environment
+For secured nodes and clusters, two policies should be configured in
advance:
+
+* Access the controller â A user that will have access to these
utilities should be authorized in NiFi by creating an âaccess
+the controllerâ policy ('/controller') with both view and modify rights.
+* Proxy user request â If not previously set nodeâs identity (the DN
value of the nodeâs certificate) should be authorized to proxy requests on
behalf of a user
+
+When executing either the notify or node manager tools in a secured
environment the `proxyDN` flag option should be used in order to properly
identify the user that was authorized to execute these commands. In non-secure
environments, or if running the status operation on the Node Manager tool, the
flag is ignored.
+
+=== Notify
+Notify allows administrators to send messages as bulletins to NiFi.
Notify is supported on NiFi version 1.2.0 and higher.
+The notification tool is also available in a _notify.bat_ file for use on
Windows machines.
+
+To send notifications:
+
+ notify.sh -d {$NIFI_HOME} âb {nifi bootstrap file path} -m {message}
[-l {level}][-v]
+
+To show help:
+
+ notify.sh -h
+
+The following are available options:
+
+* `-b,--bootstrapConf ` The _bootstrap.conf_ file to persist master
key
+* `-d,--nifiInstallDir ` NiFi Root Folder (required)
+* `-p,--proxyDN ` Proxy or User DN (required for secured nodes)
+* `-m,--message ` Bulletin message (required)
+* `-l,--level ` Status level of bulletin â INFO, WARN, ERROR
+* `-v,--verbose` Verbose messaging (optional)
+* `-h,--help` Help Text (optional)
+
+Example usage on Linux:
+
+ ./notify.sh -d /usr/nifi/nifi_current -b
/usr/nifi/nifi_current/conf/bootstrap.conf -m "Test Message Server 1" -l "WARN"
âp âydavis@nifiâ -v
+
+Example usage on Windows:
+
+ notify.bat -v -d "C:\\Program Files\\nifi\\nifi-1.2.0-SNAPSHOT" -b
"C:\\Program Files\\nifi\\nifi-1.2.0-SNAPSHOT\\conf\\bootstrap.conf" -m "Test
Message Server 1" -v
+
+Executing the above command line should result in a bulletin appearing in
NiFi:
+
+image::nifi-notifications.png["NiFi Notifications"]
+
+=== Node Manager
+
+Node manager supports connecting, disconnecting and removing a node when
in a cluster (an error message
+displays if the node is not part of a cluster) as well as obtaining the
status of a node. When nodes are disconnected
+from a cluster and need to be
+connected or removed, a list of urls of connected nodes should be provided
to send the required command to
+the active cluster. Node Manager supports NiFi version 1.0.0 and higher.
Node Manager is also available in
+'node-manager.bat' file for use on Windows machines.
+
+To connect, disconnect, or remove a node from a cluster:
+
+ node-manager.sh -d {$NIFI_HOME} âb { nifi bootstrap file path}
+ -o {remove|disconnect|connect|status} [-u {url list}] [-p {proxy name}]
[-v]
+
+To show help:
+
+ node-manager.sh -h
+
+The following are available options:
+
+* `-b,--bootstrapConf ` The _bootstrap.conf_ file to persist master
key
+* `-d,--nifiInstallDir ` NiFi Root Folder (required)
+* `-p,--proxyDN ` Proxy or User DN (required for secured nodes doing
connect, disconnect and remove operations)
+* `-o, --operation ` Operations supported: status, connect (cluster),
disconnect (cluster), remove (cluster)
+* `-u,--clusterUrls ` Comma delimited list of act
[GitHub] nifi pull request #1736: Nifi 3774
Github user YolandaMDavis commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1736#discussion_r114451504
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1176,6 +1176,146 @@ Secure Prompt
If you prefer not to provide the password or raw key in the command-line
invocation of the tool, leaving these arguments absent will prompt a secure
console read of the password (by default) or raw key (if the `-r` flag is
provided at invocation).
+[[admin-toolkit]]
+Administrative Tools
+
+The admin toolkit contains command line utilities for administrators to
support NiFi maintenance in standalone
+and clustered environments. These utilities include:
+
+* Notify -- The notification tool allows administrators to send bulletins
to the NiFi UI using the command line.
+* Node Manager -- The node manager tool allows administrators to perform a
status check on a node as well as to connect, disconnect, or remove nodes that
are part of a cluster.
+
+The admin toolkit is bundled with the nifi-toolkit and can be executed
with scripts found in the _bin_ folder.
+
+=== Prerequisites for Running Admin Toolkit in a Secure Environment
+For secured nodes and clusters, two policies should be configured in
advance:
+
+* Access the controller â A user that will have access to these
utilities should be authorized in NiFi by creating an âaccess
+the controllerâ policy ('/controller') with both view and modify rights.
+* Proxy user request â If not previously set nodeâs identity (the DN
value of the nodeâs certificate) should be authorized to proxy requests on
behalf of a user
+
+When executing either the notify or node manager tools in a secured
environment the `proxyDN` flag option should be used in order to properly
identify the user that was authorized to execute these commands. In non-secure
environments, or if running the status operation on the Node Manager tool, the
flag is ignored.
+
+=== Notify
+Notify allows administrators to send messages as bulletins to NiFi.
Notify is supported on NiFi version 1.2.0 and higher.
+The notification tool is also available in a _notify.bat_ file for use on
Windows machines.
+
+To send notifications:
+
+ notify.sh -d {$NIFI_HOME} âb {nifi bootstrap file path} -m {message}
[-l {level}][-v]
--- End diff --
missing space between `[-l {level}]` and `[-v]`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1736: Nifi 3774
Github user YolandaMDavis commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1736#discussion_r114451288
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1176,6 +1176,146 @@ Secure Prompt
If you prefer not to provide the password or raw key in the command-line
invocation of the tool, leaving these arguments absent will prompt a secure
console read of the password (by default) or raw key (if the `-r` flag is
provided at invocation).
+[[admin-toolkit]]
+Administrative Tools
+
+The admin toolkit contains command line utilities for administrators to
support NiFi maintenance in standalone
+and clustered environments. These utilities include:
+
+* Notify -- The notification tool allows administrators to send bulletins
to the NiFi UI using the command line.
+* Node Manager -- The node manager tool allows administrators to perform a
status check on a node as well as to connect, disconnect, or remove nodes that
are part of a cluster.
+
+The admin toolkit is bundled with the nifi-toolkit and can be executed
with scripts found in the _bin_ folder.
+
+=== Prerequisites for Running Admin Toolkit in a Secure Environment
+For secured nodes and clusters, two policies should be configured in
advance:
+
+* Access the controller â A user that will have access to these
utilities should be authorized in NiFi by creating an âaccess
+the controllerâ policy ('/controller') with both view and modify rights.
--- End diff --
`/controller` doesn't appear to use the tick marks per @alopresto comment
"Should mark /controller with '``'.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1736: Nifi 3774
Github user YolandaMDavis commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1736#discussion_r114450875
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1176,6 +1176,146 @@ Secure Prompt
If you prefer not to provide the password or raw key in the command-line
invocation of the tool, leaving these arguments absent will prompt a secure
console read of the password (by default) or raw key (if the `-r` flag is
provided at invocation).
+[[admin-toolkit]]
+Administrative Tools
+
+The admin toolkit contains command line utilities for administrators to
support NiFi maintenance in standalone
+and clustered environments. These utilities include:
+
+* Notify -- The notification tool allows administrators to send bulletins
to the NiFi UI using the command line.
+* Node Manager -- The node manager tool allows administrators to perform a
status check on a node as well as to connect, disconnect, or remove nodes that
are part of a cluster.
+
+The admin toolkit is bundled with the nifi-toolkit and can be executed
with scripts found in the _bin_ folder.
+
+=== Prerequisites for Running Admin Toolkit in a Secure Environment
+For secured nodes and clusters, two policies should be configured in
advance:
+
+* Access the controller â A user that will have access to these
utilities should be authorized in NiFi by creating an âaccess
+the controllerâ policy ('/controller') with both view and modify rights.
+* Proxy user request â If not previously set nodeâs identity (the DN
value of the nodeâs certificate) should be authorized to proxy requests on
behalf of a user
+
+When executing either the notify or node manager tools in a secured
environment the `proxyDN` flag option should be used in order to properly
identify the user that was authorized to execute these commands. In non-secure
environments, or if running the status operation on the Node Manager tool, the
flag is ignored.
+
+=== Notify
+Notify allows administrators to send messages as bulletins to NiFi.
Notify is supported on NiFi version 1.2.0 and higher.
+The notification tool is also available in a _notify.bat_ file for use on
Windows machines.
+
+To send notifications:
+
+ notify.sh -d {$NIFI_HOME} âb {nifi bootstrap file path} -m {message}
[-l {level}][-v]
+
+To show help:
+
+ notify.sh -h
+
+The following are available options:
+
+* `-b,--bootstrapConf ` The _bootstrap.conf_ file to persist master
key
+* `-d,--nifiInstallDir ` NiFi Root Folder (required)
+* `-p,--proxyDN ` Proxy or User DN (required for secured nodes)
+* `-m,--message ` Bulletin message (required)
+* `-l,--level ` Status level of bulletin â INFO, WARN, ERROR
+* `-v,--verbose` Verbose messaging (optional)
+* `-h,--help` Help Text (optional)
+
+Example usage on Linux:
+
+ ./notify.sh -d /usr/nifi/nifi_current -b
/usr/nifi/nifi_current/conf/bootstrap.conf -m "Test Message Server 1" -l "WARN"
âp âydavis@nifiâ -v
+
+Example usage on Windows:
+
+ notify.bat -v -d "C:\\Program Files\\nifi\\nifi-1.2.0-SNAPSHOT" -b
"C:\\Program Files\\nifi\\nifi-1.2.0-SNAPSHOT\\conf\\bootstrap.conf" -m "Test
Message Server 1" -v
+
+Executing the above command line should result in a bulletin appearing in
NiFi:
+
+image::nifi-notifications.png["NiFi Notifications"]
+
+=== Node Manager
+
+Node manager supports connecting, disconnecting and removing a node when
in a cluster (an error message
+displays if the node is not part of a cluster) as well as obtaining the
status of a node. When nodes are disconnected
+from a cluster and need to be
+connected or removed, a list of urls of connected nodes should be provided
to send the required command to
+the active cluster. Node Manager supports NiFi version 1.0.0 and higher.
Node Manager is also available in
+'node-manager.bat' file for use on Windows machines.
+
+To connect, disconnect, or remove a node from a cluster:
+
+ node-manager.sh -d {$NIFI_HOME} âb { nifi bootstrap file path}
+ -o {remove|disconnect|connect|status} [-u {url list}] [-p {proxy name}]
[-v]
+
+To show help:
+
+ node-manager.sh -h
+
+The following are available options:
+
+* `-b,--bootstrapConf ` The _bootstrap.conf_ file to persist master
key
--- End diff --
@thesolson I think this was copied in from another document. This should
read:
-b, --bootstrapConf Existing Bootstrap Configuration file (required)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
ena
[GitHub] nifi pull request #1736: Nifi 3774
Github user YolandaMDavis commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1736#discussion_r114450829
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1176,6 +1176,146 @@ Secure Prompt
If you prefer not to provide the password or raw key in the command-line
invocation of the tool, leaving these arguments absent will prompt a secure
console read of the password (by default) or raw key (if the `-r` flag is
provided at invocation).
+[[admin-toolkit]]
+Administrative Tools
+
+The admin toolkit contains command line utilities for administrators to
support NiFi maintenance in standalone
+and clustered environments. These utilities include:
+
+* Notify -- The notification tool allows administrators to send bulletins
to the NiFi UI using the command line.
+* Node Manager -- The node manager tool allows administrators to perform a
status check on a node as well as to connect, disconnect, or remove nodes that
are part of a cluster.
+
+The admin toolkit is bundled with the nifi-toolkit and can be executed
with scripts found in the _bin_ folder.
+
+=== Prerequisites for Running Admin Toolkit in a Secure Environment
+For secured nodes and clusters, two policies should be configured in
advance:
+
+* Access the controller â A user that will have access to these
utilities should be authorized in NiFi by creating an âaccess
+the controllerâ policy ('/controller') with both view and modify rights.
+* Proxy user request â If not previously set nodeâs identity (the DN
value of the nodeâs certificate) should be authorized to proxy requests on
behalf of a user
+
+When executing either the notify or node manager tools in a secured
environment the `proxyDN` flag option should be used in order to properly
identify the user that was authorized to execute these commands. In non-secure
environments, or if running the status operation on the Node Manager tool, the
flag is ignored.
+
+=== Notify
+Notify allows administrators to send messages as bulletins to NiFi.
Notify is supported on NiFi version 1.2.0 and higher.
+The notification tool is also available in a _notify.bat_ file for use on
Windows machines.
+
+To send notifications:
+
+ notify.sh -d {$NIFI_HOME} âb {nifi bootstrap file path} -m {message}
[-l {level}][-v]
+
+To show help:
+
+ notify.sh -h
+
+The following are available options:
+
+* `-b,--bootstrapConf ` The _bootstrap.conf_ file to persist master
key
--- End diff --
@thesolson I think this was copied in from another document. This should
read:
`-b, --bootstrapConf ` Existing Bootstrap Configuration file (required)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1736: Nifi 3774
Github user YolandaMDavis commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1736#discussion_r114433000
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1176,6 +1176,148 @@ Secure Prompt
If you prefer not to provide the password or raw key in the command-line
invocation of the tool, leaving these arguments absent will prompt a secure
console read of the password (by default) or raw key (if the `-r` flag is
provided at invocation).
+[[admin-toolkit]]
+Administrative Tools
+
+The admin toolkit contains command line utilities for administrators to
support NiFi maintenance in standalone
+and clustered environments. These utilities include:
+
+* Notify -- The notification tool allows administrators to send bulletins
to the NiFi UI using the command line.
+* Node Manager -- The node manager tool allows administrators to safely
disconnect or remove nodes from a NiFi cluster.
+
+The admin toolkit is bundled with the nifi-toolkit and can be executed
with scripts found in the _bin_ folder.
+
+=== Prerequisites for Running Admin Toolkit in a Secure Environment
+For secured nodes and clusters, two policies should be configured in
advance:
+
+* Access the controller â A user that will have access to these
utilities should be authorized in NiFi by creating an âaccess the
controllerâ policy (/controller) with both view and modify rights.
+* Proxy user request â If not previously set nodeâs identity ( the DN
value of the nodeâs certificate) should be authorized to proxy requests on
behalf of a user
+
+When executing either the notify or node manager tools in a secured
environment the proxyDN flag option should be used in order to properly
identify the user that was authorized to execute these commands. In non-secure
environments the flag will be ignored.
+
+=== Notify
+Notify allows administrators to send messages as bulletins to NiFi.
Notify is supported on NiFi version 1.2.0 and higher.
+The notification tool is also available in a notify.bat file for use on
Windows machines.
+
+
+
+To send notifications:
+
+ notify.sh -d {nifi root folder} âb {nifi bootstrap file path} -m
{message} [-l {level}][-v]
+
+
+To show help:
+
+ notify.sh -h
+
+The following are available options:
+
+* -b | --bootstrapConf - Existing Bootstrap Configuration file (required)
+* -d | --nifiInstallDir â NiFi Root Folder (required)
+* -p | --proxyDN â Proxy or User DN (required for secured nodes)
+* -m | --message â Bulletin message (required)
+* -l | --level â Status level of bulletin â INFO, WARN, ERROR
+* -v | --verbose â Verbose messaging (optional)
+* -h | --help â Help Text (optional)
+
+Example usage on Linux:
+
+ ./notify.sh -d /usr/nifi/nifi_current -b
/usr/nifi/nifi_current/conf/bootstrap.conf -m "Test Message Server 1" -l "WARN"
âp âydavis@nifiâ -v
+
+Example usage on Windows:
+
+ notify.bat -v -d "C:\\Program Files\\nifi\\nifi-1.2.0-SNAPSHOT" -b
"C:\\Program Files\\nifi\\nifi-1.2.0-SNAPSHOT\\conf\\bootstrap.conf" -m "Test
Message Server 1" -v
+
+Executing the above command line should result in a bulletin appearing in
NiFi:
+
+image::nifi-notifications.png["NiFi Notifications"]
+
+=== Node Manager
+
+Node manager supports connecting, disconnecting and removing a node when
in a cluster (an error message
+displays if the node is not part of a cluster) as well as obtaining the
status of a node. When nodes are disconnected
+from a cluster and need to be
+connected or removed, a list of urls of connected nodes should be provided
to send the required command to
+the active cluster. Node Manager supports NiFi version 1.0.0 and higher.
Node Manager is also available in
+node-manager.bat file for use on Windows machines.
+
+To connect, disconnect, or remove a node from a cluster:
+
+ node-manager.sh -d {nifi root folder} âb { nifi bootstrap file path}
+ -o {remove|disconnect|connect|status} [-u {url list}] [-p {proxy
name}][-v]
+
+To show help:
+
+ node-manager.sh -h
+
+The following are available options:
+
+* -b | --bootstrapConf - Existing Bootstrap Configuration file (required)
+* -d | --nifiInstallDir â NiFi Root Folder (required)
+* -p | --proxyDN â Proxy or User DN (required for secured nodes doing
connect, disconnect and remove operations)
+* -o | --operation- Operations supported: status, connect (cluster),
disconnect(cluster), remove (cluster)
+* -u | --urlList â Comma delimited list of active urls for cluster
(optional). Not required for disconnecting a node ye
[GitHub] nifi pull request #1736: Nifi 3774
Github user YolandaMDavis commented on a diff in the pull request: https://github.com/apache/nifi/pull/1736#discussion_r114432596 --- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc --- @@ -1176,6 +1176,148 @@ Secure Prompt If you prefer not to provide the password or raw key in the command-line invocation of the tool, leaving these arguments absent will prompt a secure console read of the password (by default) or raw key (if the `-r` flag is provided at invocation). +[[admin-toolkit]] +Administrative Tools + +The admin toolkit contains command line utilities for administrators to support NiFi maintenance in standalone +and clustered environments. These utilities include: + +* Notify -- The notification tool allows administrators to send bulletins to the NiFi UI using the command line. +* Node Manager -- The node manager tool allows administrators to safely disconnect or remove nodes from a NiFi cluster. + +The admin toolkit is bundled with the nifi-toolkit and can be executed with scripts found in the _bin_ folder. + +=== Prerequisites for Running Admin Toolkit in a Secure Environment +For secured nodes and clusters, two policies should be configured in advance: + +* Access the controller â A user that will have access to these utilities should be authorized in NiFi by creating an âaccess the controllerâ policy (/controller) with both view and modify rights. +* Proxy user request â If not previously set nodeâs identity ( the DN value of the nodeâs certificate) should be authorized to proxy requests on behalf of a user + +When executing either the notify or node manager tools in a secured environment the proxyDN flag option should be used in order to properly identify the user that was authorized to execute these commands. In non-secure environments the flag will be ignored. --- End diff -- "In non-secure environments (or if running the status operation on the Node Manager tool) the flag will be ignored." --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1736: Nifi 3774
Github user YolandaMDavis commented on a diff in the pull request: https://github.com/apache/nifi/pull/1736#discussion_r114432248 --- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc --- @@ -1176,6 +1176,148 @@ Secure Prompt If you prefer not to provide the password or raw key in the command-line invocation of the tool, leaving these arguments absent will prompt a secure console read of the password (by default) or raw key (if the `-r` flag is provided at invocation). +[[admin-toolkit]] +Administrative Tools + +The admin toolkit contains command line utilities for administrators to support NiFi maintenance in standalone +and clustered environments. These utilities include: + +* Notify -- The notification tool allows administrators to send bulletins to the NiFi UI using the command line. +* Node Manager -- The node manager tool allows administrators to safely disconnect or remove nodes from a NiFi cluster. --- End diff -- missing some functionality here. Should read: Node Manager -- The node manager tool allows administrators to perform a status check on a node as well as connect, disconnect or remove nodes that are part of a cluster. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1736: Nifi 3774
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1736 @thesolson thanks will review --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1686: NIFI-3594 Encrypted provenance repository implementation
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1686 @alopresto actually Andy just noticed there's a conflict. If you could resolve, squash and push I'll be happy to continue with merging into master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1686: NIFI-3594 Encrypted provenance repository implementation
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1686 @alopresto ran tests with the following corner cases: -flow running with normal shutdown invoked. NiFi shutdown and started without issue -flow running with kill -9 on process invoked. NiFi process was terminated and I was able to start without issue -flow running and disk exceeded space - I was able to shutdown NiFi clear out disk problem and start nifi without issue Also validated that switch between EWAPR and PPR (and vice versa) along with removal of content, flowfile and provenance repos would prevent classcastexception seen previously. +1 will merge shortly --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1686: NIFI-3594 Encrypted provenance repository implementation
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1686 @alopresto moving to a separate ticket makes sense. I have one more test to run (run out of disk space per @markap14 suggestion) and will provide any feedback shortly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1726: NIFI-3765 - Added a 'status' operation to Node Mana...
GitHub user YolandaMDavis opened a pull request:
https://github.com/apache/nifi/pull/1726
NIFI-3765 - Added a 'status' operation to Node Manager
The command for status is the following:
./node-manager.sh -d {nifi-installation-dir} -b {nifi-bootstrap-file} -o
status
NodeManager will print any of the following messages based on the returned
response or received exception:
When Site Returns Response Code == 200:
**NiFi Node is running and available.**
**When Site Returns Response Code != 200**
NiFi Node returned Response Code: {code} with reason {response body}
**When exception occurs attempting to connect to the node**
NiFi Node could not be reached due to exception {exception message}
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with NIFI- where is the JIRA number
you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/YolandaMDavis/nifi NIFI-3765
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/1726.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1726
commit 9a142cd4140e4e08e3120a151ef787135f9474eb
Author: Yolanda M. Davis
Date: 2017-05-01T23:33:42Z
NIFI-3765 - added status operation to NodeManager
commit 18429232fe8bfe260a556ea16625315e0c31ed22
Author: Yolanda M. Davis
Date: 2017-05-01T23:41:30Z
NIFI-3765 - updated help text
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1686: NIFI-3594 Encrypted provenance repository implementation
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1686 @alopresto Ran a test where I had an existing unencrypted provenance repo and switched to the encrypted provider. The issues [you've documented](https://github.com/alopresto/nifi/blob/276f31aa85d9ec401fb34f939238c6646e187ae6/nifi-docs/src/main/asciidoc/user-guide.adoc#potential-issues) did occur (where switching between configuration would lead to errors in the log and when querying). However I'm wondering two things: 1) Can there be a friendlier error message in the logs for those errors? I saw lots of errors around ClassCastException between the old and new Record Readers. Just wondering if NiFi could simply say detected a mismatch and that it could be due to a change in repo providers? 2) I also saw the UI error (see below), however I expected to eventually be able to retrieve the events that were subject to encryption. I have a super short and small flow that runs every 10s. Perhaps provenance hasn't been completely overwritten with the newer encrypted files in my case but each time I attempt to query provenance I get the ClassCastException.   If clearing the provenance repo when changing configurations would resolve the issue can that step be included in the documentation (if needed I would recommend including that step in the issues section)? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1686: NIFI-3594 Encrypted provenance repository implementation
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1686 @alopresto happy to do some functional testing of this as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1697: NIFI-3695 - added proxy dn flag to support providing autho...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1697 Looks good to me, thanks @mcgilman ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1697: NIFI-3695 - added proxy dn flag to support providing autho...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1697 @mcgilman I addressed your comments above in the latest commit. Please let me know if you have any additional questions or feedback. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1697: NIFI-3695 - added proxy dn flag to support providing autho...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1697 @mcgilman understood. It does makes sense to have more polite messaging given this new case. I'll work on those changes and update. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1697: NIFI-3695 - added proxy dn flag to support providing autho...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1697 @mcgilman and @brosander appreciate if you could take a look at this enhancement. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1697: NIFI-3695 - added proxy dn flag to support providin...
GitHub user YolandaMDavis opened a pull request: https://github.com/apache/nifi/pull/1697 NIFI-3695 - added proxy dn flag to support providing authorized usern⦠â¦ame for secured environments An updated version of documentation, with usage of the proxy flag is attached: [Admin Toolkit for NiFi.docx](https://github.com/apache/nifi/files/958755/Admin.Toolkit.for.NiFi.docx) Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/YolandaMDavis/nifi NIFI-3695-proxy Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1697.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1697 commit 41e7ee738448613b4c7f0e1ccadfe89741b6befb Author: Yolanda M. Davis Date: 2017-04-26T14:22:18Z NIFI-3695 - added proxy dn flag to support providing authorized username for secured environments --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1669: NIFI-3695 - created the nifi admin toolkit which includes ...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1669 @alopresto I addressed the comments to apply LdapName as well as other minor corrections suggested --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1669: NIFI-3695 - created the nifi admin toolkit which includes ...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1669 @alopresto thanks so much for your feedback. I've made changes for your review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1669: NIFI-3695 - created the nifi admin toolkit which includes ...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1669 @brosander I've chatted with @thesolson about documentation and she was willing to help there. She will create a separate request in Jira for that effort to add admin-toolkit info to the admin guide. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1669: NIFI-3695 - created the nifi admin toolkit which includes ...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1669 @brosander concerning the binaries they have a very small footprint so I wasn't too concerned about including them (32K, 27K and 1K). I will work to incorporate the cert generation test and documentation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1669: NIFI-3695 - created the nifi admin toolkit which in...
Github user YolandaMDavis commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1669#discussion_r112242565
--- Diff:
nifi-toolkit/nifi-toolkit-admin/src/main/groovy/org/apache/nifi/toolkit/admin/client/NiFiClientUtil.groovy
---
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.admin.client
+
+import com.google.common.collect.Lists
+import com.sun.jersey.api.client.Client
+import com.sun.jersey.api.client.ClientResponse
+import com.sun.jersey.api.client.WebResource
+import org.apache.nifi.util.NiFiProperties
+import org.apache.nifi.util.StringUtils
+import org.apache.nifi.web.api.dto.NodeDTO
+import org.apache.nifi.web.api.entity.ClusterEntity
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+public class NiFiClientUtil {
+
+private static final Logger logger =
LoggerFactory.getLogger(NiFiClientUtil.class)
+private final static String GET_CLUSTER_ENDPOINT
="/nifi-api/controller/cluster"
+
+public static Boolean isCluster(final NiFiProperties niFiProperties){
+String clusterNode =
niFiProperties.getProperty(NiFiProperties.CLUSTER_IS_NODE)
+return Boolean.valueOf(clusterNode)
+}
+
+public static String getUrl(NiFiProperties niFiProperties, String
endpoint){
+
+final StringBuilder urlBuilder = new StringBuilder();
+
+
if(!StringUtils.isEmpty(niFiProperties.getProperty(NiFiProperties.WEB_HTTPS_PORT))){
+urlBuilder.append("https://";)
+
urlBuilder.append(StringUtils.isEmpty(niFiProperties.getProperty(NiFiProperties.WEB_HTTPS_HOST))
? "localhost": niFiProperties.getProperty(NiFiProperties.WEB_HTTPS_HOST))
+urlBuilder.append(":")
+
urlBuilder.append(StringUtils.isEmpty(niFiProperties.getProperty(NiFiProperties.WEB_HTTPS_PORT))
? "8081" : niFiProperties.getProperty(NiFiProperties.WEB_HTTPS_PORT))
+}else{
+urlBuilder.append("http://";)
+
urlBuilder.append(StringUtils.isEmpty(niFiProperties.getProperty(NiFiProperties.WEB_HTTP_HOST))
? "localhost": niFiProperties.getProperty(NiFiProperties.WEB_HTTP_HOST))
+urlBuilder.append(":")
+
urlBuilder.append(StringUtils.isEmpty(niFiProperties.getProperty(NiFiProperties.WEB_HTTPS_PORT))
? "8080": niFiProperties.getProperty(NiFiProperties.WEB_HTTPS_PORT))
+}
+
+if(!StringUtils.isEmpty(endpoint)) {
+urlBuilder.append(endpoint)
+}
+
+urlBuilder.toString()
+}
+
+public static String getUrl(NiFiProperties niFiProperties, NodeDTO
nodeDTO, String endpoint){
+
+final StringBuilder urlBuilder = new StringBuilder();
+
if(!StringUtils.isEmpty(niFiProperties.getProperty(NiFiProperties.WEB_HTTPS_PORT))){
+urlBuilder.append("https://";)
+
+}else{
+urlBuilder.append("http://";)
+}
+urlBuilder.append(nodeDTO.address)
+urlBuilder.append(":")
+urlBuilder.append(nodeDTO.apiPort)
+
+if(!StringUtils.isEmpty(endpoint)) {
+urlBuilder.append(endpoint)
+}
+
+urlBuilder.toString()
+}
+
+public static ClusterEntity getCluster(final Client client,
NiFiProperties niFiProperties, List activeUrls){
+
+if(activeUrls.isEmpty()){
+final String url = getUrl(niFiProperties,null)
+activeUrls.add(url)
+}
+
+for(String activeUrl: activeUrls) {
+
+try {
+
+String url = activeUrl + GET_CLUSTER_ENDPOINT
+final WebResource webResource = client.resource(url)
+final ClientResponse response =
webResource.type(
[GitHub] nifi issue #1678: NIFI-3694 updated latest logback,jetty,moved servlet-api a...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1678 @joewitt ran local test using smaller maxFileSize, maxHistory and totalSizeCap settings. Created a flow that used LogAttribute heavily. nifi-app.log appeared to rotate and was purged as expected. +1 Will merge into master shortly --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1678: NIFI-3694 updated latest logback,jetty,moved servlet-api a...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1678 @joewitt I will take a look, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1669: NIFI-3695 - created the nifi admin toolkit which includes ...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1669 @brosander thanks so much for reviewing! Concerning documentation I do have a link attached. Will also look at the code changes you suggested. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1669: NIFI-3695 - created the nifi admin toolkit which in...
Github user YolandaMDavis commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1669#discussion_r111814074
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/groovy/org/apache/nifi/web/StandardNiFiServiceFacadeSpec.groovy
---
@@ -49,6 +53,7 @@ class StandardNiFiServiceFacadeSpec extends Specification
{
SecurityContextHolder.getContext().setAuthentication(null);
}
--- End diff --
The full suite of tests were previously ignored, however I had a test that
could work.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1674: NIFI-3708 - removed typo from character escape rege...
GitHub user YolandaMDavis opened a pull request: https://github.com/apache/nifi/pull/1674 NIFI-3708 - removed typo from character escape regex Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/YolandaMDavis/nifi NIFI-3708 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1674.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1674 commit f6beba1e6ec7028ed7583db08e88a7c80d25b7d4 Author: Yolanda M. Davis Date: 2017-04-14T15:19:24Z NIFI-3708 - removed typo from character escape regex --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1669: NIFI-3695 - created the nifi admin toolkit which in...
GitHub user YolandaMDavis opened a pull request: https://github.com/apache/nifi/pull/1669 NIFI-3695 - created the nifi admin toolkit which includes shell scrip⦠â¦ts and classes to support notification and basic node management in standalone and clustered nifi. Documentation [Admin Toolkit for NiFi.docx](https://github.com/apache/nifi/files/917612/Admin.Toolkit.for.NiFi.docx) and use case information is attached Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [X] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/YolandaMDavis/nifi NIFI-3695 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1669.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1669 commit c6e66279d0d7e9b803f51ce9715262b51e25d8d1 Author: Yolanda M. Davis Date: 2017-02-07T15:28:15Z NIFI-3695 - created the nifi admin toolkit which includes shell scripts and classes to support notification and basic node management in standalone and clustered nifi. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1630: NIFI-3613: Fixed threading bug in JoltTransformJSON and ad...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1630 @markap14 reviewed and ran unit/performance tests with no issue. +1 will merge shortly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1630: NIFI-3613: Fixed threading bug in JoltTransformJSON and ad...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1630 @markap14 I will take a look, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1348: NIFI-3010 - Incorporate Expression Language support for Jo...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1348 @scottyaslan @mcgilman @moranr would appreciate your input, especially on recent styling changes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1303: NIFI-2994: Provide default position for RPGs
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1303 Re-tested with several combinations of empty values, system behaved as expected, returning bad requests on invalid input and return 201 Created when position values were available or position object not included. +1 Will merge shortly. Thanks @mcgilman --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1303: NIFI-2994: Provide default position for RPGs
Github user YolandaMDavis commented on the issue:
https://github.com/apache/nifi/pull/1303
Testing with a Rest Client API I was able to produce a null pointer
exception for the following case:
"position":{"x":null,"y":null}
"position":{"x":"","y":""}
The stack trace in both cases were the following:
`
java.lang.NullPointerException: null
at
org.apache.nifi.web.dao.impl.StandardRemoteProcessGroupDAO.updateRemoteProcessGroup(StandardRemoteProcessGroupDAO.java:376)
~[classes/:na]
at
org.apache.nifi.web.dao.impl.StandardRemoteProcessGroupDAO.createRemoteProcessGroup(StandardRemoteProcessGroupDAO.java:87)
~[classes/:na]
at
org.apache.nifi.web.dao.impl.StandardRemoteProcessGroupDAO$$FastClassBySpringCGLIB$$4a6f5661.invoke()
~[classes/:na]
at
org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
~[spring-core-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at
org.apache.nifi.audit.RemoteProcessGroupAuditor.createRemoteProcessGroupAdvice(RemoteProcessGroupAuditor.java:68)
~[classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[na:1.8.0_111]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[na:1.8.0_111]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.8.0_111]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_111]
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68)
~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at
org.apache.nifi.web.dao.impl.StandardRemoteProcessGroupDAO$$EnhancerBySpringCGLIB$$97a3cf4b.createRemoteProcessGroup()
~[classes/:na]
at
org.apache.nifi.web.StandardNiFiServiceFacade.lambda$createRemoteProcessGroup$102(StandardNiFiServiceFacade.java:1579)
~[classes/:1.2.0-SNAPSHOT]
at
org.apache.nifi.web.StandardNiFiServiceFacade.lambda$createComponent$86(StandardNiFiServiceFacade.java:1333)
~[classes/:1.2.0-SNAPSHOT]
at
org.apache.nifi.web.revision.NaiveRevisionManager.updateRevision(NaiveRevisionManager.java:121)
~[nifi-web-optimistic-locking-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
at
org.apache.nifi.web.StandardNiFiServiceFacade.createComponent(StandardNiFiServiceFacade.java:1331)
~[classes/:1.2.0-SNAPSHOT]
at
org.apache.nifi.web.StandardNiFiServiceFacade.createRemoteProcessGroup(StandardNiFiServiceFacade.java:1576)
~[classes/:1.2.0-SNAPSHOT]
at
org.apache.nifi.web.StandardNiFiServiceFacade$$FastClassBySpringCGLIB$$358780e0.invoke()
~[classes/:1.2.0-SNAPSHOT]
at
org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
~[spring-core-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at
org.apache.nifi.web.NiFiServiceFacadeLock.
[GitHub] nifi issue #1303: NIFI-2994: Provide default position for RPGs
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1303 @mcgilman reviewing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1319: NIFI-3167: Delete button in cluster table wrapping to next...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1319 Tested with 3 node cluster on Firefox, Safari and Chrome. Confirmed existing in Firefox was resolved with patch. +1 Will merge into master shortly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1319: NIFI-3167: Delete button in cluster table wrapping to next...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1319 @mcgilman reviewing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1348: NIFI-3010 - Incorporate Expression Language support...
GitHub user YolandaMDavis opened a pull request:
https://github.com/apache/nifi/pull/1348
NIFI-3010 - Incorporate Expression Language support for JoltTransformJSON
This PR contains several feature enhancements:
- Support of expression language within the specification
- Support of expression language testing using Advanced UI with new Test
Attributes dialog
- Update of Jolt library to version 0.1
- Added support for new Modifier library (Jolt Beta functionality)
For Testing:
Expression language can now be referenced within a Jolt Specification.
Expressions can be written on either side on the left hand side ( matching
input json keys) or the right hand side (output of json data) of the
specification. The Advanced UI can test expressions that reference attributes
using the Attributes dialog which allows users to provide key/value pairs that
will be resolved when testing a transformation.
https://cloud.githubusercontent.com/assets/1371858/21375258/4e9f1878-c6fa-11e6-9a45-2213e6f42107.png";>
https://cloud.githubusercontent.com/assets/1371858/21375261/52c60d44-c6fa-11e6-9fff-9e815c6e7ce9.png";>
Jolt 0.1 has added support for transformation of data within a given schema
through the Modifier library. Modifier introduces more sophisticated data
manipulation capabilities including for strings, lists, type conversion, etc.
https://cloud.githubusercontent.com/assets/1371858/21375249/4080ccb4-c6fa-11e6-8e34-c41c33644aa9.png";>
Below is an example chain specification that can be used against Twitter
payload for testing. When testing in the Advanced UI ensure that attributes
referenced in expression language are provided for testing (${rating.var} and
${id.var}) and that left side variable content match actual input (e.g. id.var
should equal 'id' for twitter json to match). Custom attributes can also be
configured with nifi to use this specification within an actual flow.
`
[{
"operation": "shift",
"spec": {
"created_at": "created_date_time",
"id": "tweet_id",
"text": "tweet_text",
"user": {
"${id.var}": "user_id"
}
}
},
{
"operation": "default",
"spec":{
"user-rating" : 4,
"twitter-rating" : "${rating.var}"
}
},{
"operation": "modify-default-beta",
"spec": {
"~in_reply_to_status_id": 0,
"~in_reply_to_status_id_str": 0,
"~in_reply_to_user_id": "",
"~in_reply_to_user_id_str": 0,
"~in_reply_to_screen_name": ""
}
},{
"operation": "modify-define-beta",
"spec": {
"user_nearby_address":"na"
}
},{
"operation": "modify-overwrite-beta",
"spec": {
"truncate" : true
}
}
]
`
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with NIFI- where is the JIRA number
you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [x] Is your initial contribution a single, squashed commit?
### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in
which it is rendered?
### N
[GitHub] nifi issue #1261: NIFI-3024 Added key migration for sensitive processor prop...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1261 @alopresto agreed with this assessment and the move to open to a separate PR. All other tests ran successfully for me beyond this issue +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1261: NIFI-3024 Added key migration for sensitive processor prop...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1261 made it passed that issue however on subsequent test encoutered the following (includes command executed): HW11205:nifi-1.1.0 ydavis$ /Users/ydavis/dev/tools/nifi-1.1.0/toolkit/nifi-toolkit-1.1.0-pr-1261/bin/encrypt-config.sh -b /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/bootstrap.conf -n /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/nifi.properties -x -f /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/flow.xml.gz -s thisIsASpecialPassword 2016/11/23 13:27:59 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source nifi.properties and destination nifi.properties are identical [/Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/nifi.properties] so the original will be overwritten 2016/11/23 13:27:59 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source flow.xml.gz and destination flow.xml.gz are identical [/Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/flow.xml.gz] so the original will be overwritten 2016/11/23 13:27:59 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 121 properties from /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/nifi.properties 2016/11/23 13:27:59 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 121 properties from /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/nifi.properties 2016/11/23 13:28:00 INFO [main] org.apache.nifi.properties.ProtectedNiFiProperties: There are 1 protected properties of 4 sensitive properties (25%) 2016/11/23 13:28:00 INFO [main] org.apache.nifi.properties.AESSensitivePropertyProvider: AES Sensitive Property Provider decrypted a sensitive value successfully 2016/11/23 13:28:00 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Loaded NiFiProperties instance with 120 properties pad block corrupted --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1261: NIFI-3024 Added key migration for sensitive processor prop...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1261 @alopresto I think I isolated part of the issue. Please confirm my use of this command: /Users/ydavis/dev/tools/nifi-1.1.0/toolkit/nifi-toolkit-1.1.0-pr-1261/bin/encrypt-config.sh -b /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/bootstrap.conf -n /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/nifi.properties -x -f /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/flow.xml.gz -s thisIsADifferentPassword Please confirm my use of -x in this context. I thought this would prevent updating nifi.properties? The end result did show an update to nifi-properties that excluded the nifi.senstivei.props.key.protected value: # security properties# nifi.sensitive.props.key=9ASs7Dpzqm1pOZVR||tJbvXzRFOwGxabcssIkA1l3p3tHnOdo2fPkPXOL3TJgZNkzURMO3qw nifi.sensitive.props.key.protected= nifi.sensitive.props.algorithm=PBEWITHMD5AND128BITAES-CBC-OPENSSL nifi.sensitive.props.provider=BC nifi.sensitive.props.additional.keys= --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1261: NIFI-3024 Added key migration for sensitive processor prop...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1261 was able to get more info I think on the problem in an attempt to recover with migrating: HW11205:nifi-1.1.0 ydavis$ /Users/ydavis/dev/tools/nifi-1.1.0/toolkit/nifi-toolkit-1.1.0-pr-1261/bin/encrypt-config.sh -b /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/bootstrap.conf -n /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/nifi.properties -f /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/flow.xml.gz -s thisIsABadPassword -m -w whatever12345! -p whomever12345! -v 2016/11/23 11:22:25 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of nifi.properties 2016/11/23 11:22:25 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source nifi.properties and destination nifi.properties are identical [/Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/nifi.properties] so the original will be overwritten 2016/11/23 11:22:25 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of flow.xml.gz 2016/11/23 11:22:25 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source flow.xml.gz and destination flow.xml.gz are identical [/Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/flow.xml.gz] so the original will be overwritten 2016/11/23 11:22:25 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool:bootstrap.conf: /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/bootstrap.conf 2016/11/23 11:22:25 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) nifi.properties: /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/nifi.properties 2016/11/23 11:22:25 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) nifi.properties: /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/nifi.properties 2016/11/23 11:22:25 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) login-identity-providers.xml: null 2016/11/23 11:22:25 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) login-identity-providers.xml: null 2016/11/23 11:22:25 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) flow.xml.gz: /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/flow.xml.gz 2016/11/23 11:22:25 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) flow.xml.gz: /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/flow.xml.gz 2016/11/23 11:22:25 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Key migration mode activated 2016/11/23 11:22:25 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 121 properties from /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/nifi.properties 2016/11/23 11:22:26 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 121 properties from /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/nifi.properties 2016/11/23 11:22:26 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Loaded NiFiProperties instance with 121 properties 2016/11/23 11:22:26 ERROR [main] org.apache.nifi.properties.ConfigEncryptionTool: Encountered an error javax.crypto.BadPaddingException: pad block corrupted at org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher$BufferedGenericBlockCipher.doFinal(Unknown Source) at org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher.engineDoFinal(Unknown Source) at javax.crypto.Cipher.doFinal(Cipher.java:2165) at javax.crypto.Cipher$doFinal$2.call(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125) at org.apache.nifi.properties.ConfigEncryptionTool.decryptFlowElement(ConfigEncryptionTool.groovy:542) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:384
[GitHub] nifi issue #1261: NIFI-3024 Added key migration for sensitive processor prop...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1261 Encountered issue while attempting the below test cases (3 node cluster): #initial encryption /Users/ydavis/dev/tools/nifi-1.1.0/toolkit/nifi-toolkit-1.1.0-pr-1261/bin/encrypt-config.sh -b /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/bootstrap.conf -n /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/nifi.properties -f /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/flow.xml.gz -s thisIsABadPassword -p whomever12345! -v /Users/ydavis/dev/tools/nifi-1.1.0/toolkit/nifi-toolkit-1.1.0-pr-1261/bin/encrypt-config.sh -b /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/bootstrap.conf -n /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/nifi.properties -f /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/flow.xml.gz -s thisIsABadPassword -p whomever12345! -v /Users/ydavis/dev/tools/nifi-1.1.0/toolkit/nifi-toolkit-1.1.0-pr-1261/bin/encrypt-config.sh -b /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-3/conf/bootstrap.conf -n /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-3/conf/nifi.properties -f /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-3/conf/flow.xml.gz -s thisIsABadPassword -p whomever12345! -v #Migration /Users/ydavis/dev/tools/nifi-1.1.0/toolkit/nifi-toolkit-1.1.0-pr-1261/bin/encrypt-config.sh -b /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/bootstrap.conf -n /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/nifi.properties -f /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/flow.xml.gz -s thisIsABadPassword -m -w whomever12345! -p whatever12345! -v /Users/ydavis/dev/tools/nifi-1.1.0/toolkit/nifi-toolkit-1.1.0-pr-1261/bin/encrypt-config.sh -b /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/bootstrap.conf -n /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/nifi.properties -f /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/flow.xml.gz -s thisIsABadPassword -m -w whomever12345! -p whatever12345! -v /Users/ydavis/dev/tools/nifi-1.1.0/toolkit/nifi-toolkit-1.1.0-pr-1261/bin/encrypt-config.sh -b /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-3/conf/bootstrap.conf -n /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-3/conf/nifi.properties -f /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-3/conf/flow.xml.gz -s thisIsABadPassword -m -w whomever12345! -p whatever12345! -v #Update all encrypt passwords exclude others /Users/ydavis/dev/tools/nifi-1.1.0/toolkit/nifi-toolkit-1.1.0-pr-1261/bin/encrypt-config.sh -b /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/bootstrap.conf -n /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/nifi.properties -x -f /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-1/conf/flow.xml.gz -s thisIsADifferentPassword /Users/ydavis/dev/tools/nifi-1.1.0/toolkit/nifi-toolkit-1.1.0-pr-1261/bin/encrypt-config.sh -b /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/bootstrap.conf -n /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/nifi.properties -x -f /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/flow.xml.gz -s thisIsADifferentPassword /Users/ydavis/dev/tools/nifi-1.1.0/toolkit/nifi-toolkit-1.1.0-pr-1261/bin/encrypt-config.sh -b /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-3/conf/bootstrap.conf -n /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-3/conf/nifi.properties -x -f /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-3/conf/flow.xml.gz -s thisIsADifferentPassword All 3 above worked successfully and cluster was able to start and stop each time as well as run flow. I attempted my fourth test case to change 1 node's senstive key using the command below: /Users/ydavis/dev/tools/nifi-1.1.0/toolkit/nifi-toolkit-1.1.0-pr-1261/bin/encrypt-config.sh -b /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/bootstrap.conf -n /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/nifi.properties -x -f /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/flow.xml.gz -s thisIsASpecialPassword On this run config tool reported the following error: HW11205:nifi-1.1.0 ydavis$ /Users/ydavis/dev/tools/nifi-1.1.0/toolkit/nifi-toolkit-1.1.0-pr-1261/bin/encrypt-config.sh -b /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/bootstrap.conf -n /Users/ydavis/dev/tools/nifi-1.1.0/cluster/nifi-1.1.0-pr-1261-2/conf/nifi.properties -x -f /
[GitHub] nifi issue #1261: NIFI-3024 Added key migration for sensitive processor prop...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1261 @alopresto thanks Andy will confirm --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1261: NIFI-3024 Added key migration for sensitive processor prop...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1261 @alopresto reviewing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1221: NIFI-3024 - Encrypt-toolkit flow.xml.gz support
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1221 @olegz I think @alopresto is taking over this PR and may create a new one, per his comment --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1238: NIFI-3051 Fixed issue serializing commented or empty login...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1238 @alopresto was able to run tests both in IDE and Maven and reconfirmed test scenarios worked as expected. +1 Ready for merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1238: NIFI-3051 Fixed issue serializing commented or empty login...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1238 @alopresto agreed with fixing the test. I've actually logged the issue here: https://issues.apache.org/jira/browse/NIFI-3060 Once that is addressed and you've updated your test I can reevaluate. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1238: NIFI-3051 Fixed issue serializing commented or empty login...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1238 @alopresto thanks for the update. Was able to retest with the following scenarios: encryption with fully commented file (as in the original template without edits) encryption with one provider (using an Ldap Provider class) commented out, another provider available encryption with ldap provider available All scenarios work as expected. +1 Will merge shortly into master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1238: NIFI-3051 Fixed issue serializing commented or empty login...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1238 @alopresto just thought of a particular scenario. If a user defines a provider with an identifier not equal to "ldap-provider" however has securable properties like those found in the ldap-provider template the tool currently does not catch that. I'm wondering if it's possible to do something similar to what happens in nifi.properties where it searches for the value of password as a property and encrypts/decrypts that value? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1238: NIFI-3051 Fixed issue serializing commented or empty login...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1238 @alopresto actually correction, maven run of this test worked however individual execution of test in my IDE does not so this may not be an issue, probably a configuration problem on my end. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1238: NIFI-3051 Fixed issue serializing commented or empty login...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1238 @alopresto I rebased this with master and I have 7 tests are failing. I believe this may be due to recent changes in logging for toolkit, since all failed tests rely on assertions against events in the TestAppender class which doesn't get populated during the run. Also I ran ConfigEncryptionToolTest in master (without this change) and the test is failing there as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1238: NIFI-3051 Fixed issue serializing commented or empty login...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1238 @alopresto will review --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1216: NIFI-2654 Enabled encryption coverage for login-identity-p...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1216 @alopresto I was able to test encrypt-config update with the following scenarios: tested encryption using new -l flag along with -n, - b, and -p flags (no migration). nifi.properties and login provider identity files encrypted properties as expected. Master key appears as expected in bootstrap.con tested encryption using -l, -n, and - b flag with migration of password using -m -w. nifi.properties and login provider identity files encrypted as expected. Master key appears as expected in bootstrap.conf Also tested encrypted configurations with secured NiFi against OpenLDAP (SIMPLE and TLS). In both scenarios NiFi was able to decrypt values and access LDAP for authentication as needed. +1 Thanks @alopresto! Will merge this in shortly --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1216: NIFI-2654 Enabled encryption coverage for login-identity-p...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1216 @alopresto will review --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1066: NIFI-2823 - Adding nifiDnPrefix, nifiDnSuffix options to s...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1066 @brosander was able to run tests to generate certs with prefix and suffix +1 Will merge into master shortly --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1066: NIFI-2823 - Adding nifiDnPrefix, nifiDnSuffix options to s...
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1066 @brosander will review --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1186: NIFI-2652 [WIP] Handle encrypted config key migration
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1186 @alopresto re-ran the unit tests and scenarios looks good +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1186: NIFI-2652 [WIP] Handle encrypted config key migration
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1186 @alopresto just to add clarity on unit test failures, I am specifically receiving a problem when running the ConfigEncryptionToolTest. Several tests are failing for me and the failure appears to be due to a generated key that looks truncated. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1186: NIFI-2652 [WIP] Handle encrypted config key migration
Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/1186 @alopresto I also received problems running unit test but confirmed that I could migrate keys using several scenarios: raw key -> raw key password -> password password -> raw key raw key -> password Also tried negative conditions including incorrect old password/rawkey and received expected exception. Once unit test issue is resolved I can reevaluate for merge. Thanks @alopresto! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
