[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit
[ https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15665181#comment-15665181 ] ASF subversion and git services commented on NIFI-2957: --- Commit 7e237341810763ed4631fec8f568c04d8f928788 in nifi's branch refs/heads/master from [~jtstorck] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=7e23734 ] NIFI-2957 Corrections of typos, documentation, and strings Updates made regarding Intellij code inspections This closes #1218 Signed-off-by: Bryan Rosander > ZooKeeper migration toolkit > --- > > Key: NIFI-2957 > URL: https://issues.apache.org/jira/browse/NIFI-2957 > Project: Apache NiFi > Issue Type: New Feature > Components: Tools and Build >Affects Versions: 1.0.0, 0.7.1 >Reporter: Jeff Storck >Assignee: Jeff Storck > Fix For: 1.2.0 > > > When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the > embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be > migrated. > Initial considerations: > * Username/password protection of nodes is not supported in NiFi 1.x.. Those > nodes that are configured that way in ZooKeeper need to be migrated to have > an open ACL. > * The toolkit will support a mode to read data from a configurable root node > in a source ZooKeeper, and the data will be written to a file designated via > CLI. > * The toolkit will support a mode to write data to a destination ZooKeeper > * The toolkit will not allow data to be written to the same ZooKeeper from > which the source data was obtained. > * The toolkit will not support reconnecting to ZooKeeper if it is > disconnected. The user can rerun the tool. > * The toolkit will support ZooKeepers configured with Kerberos. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit
[ https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15665183#comment-15665183 ] ASF GitHub Bot commented on NIFI-2957: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/1218 > ZooKeeper migration toolkit > --- > > Key: NIFI-2957 > URL: https://issues.apache.org/jira/browse/NIFI-2957 > Project: Apache NiFi > Issue Type: New Feature > Components: Tools and Build >Affects Versions: 1.0.0, 0.7.1 >Reporter: Jeff Storck >Assignee: Jeff Storck > Fix For: 1.2.0 > > > When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the > embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be > migrated. > Initial considerations: > * Username/password protection of nodes is not supported in NiFi 1.x.. Those > nodes that are configured that way in ZooKeeper need to be migrated to have > an open ACL. > * The toolkit will support a mode to read data from a configurable root node > in a source ZooKeeper, and the data will be written to a file designated via > CLI. > * The toolkit will support a mode to write data to a destination ZooKeeper > * The toolkit will not allow data to be written to the same ZooKeeper from > which the source data was obtained. > * The toolkit will not support reconnecting to ZooKeeper if it is > disconnected. The user can rerun the tool. > * The toolkit will support ZooKeepers configured with Kerberos. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit
[ https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15665176#comment-15665176 ] ASF GitHub Bot commented on NIFI-2957: -- Github user brosander commented on the issue: https://github.com/apache/nifi/pull/1218 +1 will merge > ZooKeeper migration toolkit > --- > > Key: NIFI-2957 > URL: https://issues.apache.org/jira/browse/NIFI-2957 > Project: Apache NiFi > Issue Type: New Feature > Components: Tools and Build >Affects Versions: 1.0.0, 0.7.1 >Reporter: Jeff Storck >Assignee: Jeff Storck > Fix For: 1.2.0 > > > When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the > embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be > migrated. > Initial considerations: > * Username/password protection of nodes is not supported in NiFi 1.x.. Those > nodes that are configured that way in ZooKeeper need to be migrated to have > an open ACL. > * The toolkit will support a mode to read data from a configurable root node > in a source ZooKeeper, and the data will be written to a file designated via > CLI. > * The toolkit will support a mode to write data to a destination ZooKeeper > * The toolkit will not allow data to be written to the same ZooKeeper from > which the source data was obtained. > * The toolkit will not support reconnecting to ZooKeeper if it is > disconnected. The user can rerun the tool. > * The toolkit will support ZooKeepers configured with Kerberos. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit
[ https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15665102#comment-15665102 ] ASF GitHub Bot commented on NIFI-2957: -- Github user brosander commented on the issue: https://github.com/apache/nifi/pull/1218 Reviewing > ZooKeeper migration toolkit > --- > > Key: NIFI-2957 > URL: https://issues.apache.org/jira/browse/NIFI-2957 > Project: Apache NiFi > Issue Type: New Feature > Components: Tools and Build >Affects Versions: 1.0.0, 0.7.1 >Reporter: Jeff Storck >Assignee: Jeff Storck > Fix For: 1.2.0 > > > When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the > embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be > migrated. > Initial considerations: > * Username/password protection of nodes is not supported in NiFi 1.x.. Those > nodes that are configured that way in ZooKeeper need to be migrated to have > an open ACL. > * The toolkit will support a mode to read data from a configurable root node > in a source ZooKeeper, and the data will be written to a file designated via > CLI. > * The toolkit will support a mode to write data to a destination ZooKeeper > * The toolkit will not allow data to be written to the same ZooKeeper from > which the source data was obtained. > * The toolkit will not support reconnecting to ZooKeeper if it is > disconnected. The user can rerun the tool. > * The toolkit will support ZooKeepers configured with Kerberos. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit
[ https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15664568#comment-15664568 ] ASF GitHub Bot commented on NIFI-2957: -- GitHub user jtstorck opened a pull request: https://github.com/apache/nifi/pull/1218 NIFI-2957 Corrections of typos, documentation, and strings Updates made regarding Intellij code inspections You can merge this pull request into a Git repository by running: $ git pull https://github.com/jtstorck/nifi NIFI-2957 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1218.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1218 commit eb97e44cfee5fcc2860b4e651f76319f1971364b Author: Jeff Storck Date: 2016-11-14T18:02:16Z NIFI-2957 Corrections of typos, documentation, and strings Updates made regarding Intellij code inspections > ZooKeeper migration toolkit > --- > > Key: NIFI-2957 > URL: https://issues.apache.org/jira/browse/NIFI-2957 > Project: Apache NiFi > Issue Type: New Feature > Components: Tools and Build >Affects Versions: 1.0.0, 0.7.1 >Reporter: Jeff Storck >Assignee: Jeff Storck > Fix For: 1.2.0 > > > When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the > embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be > migrated. > Initial considerations: > * Username/password protection of nodes is not supported in NiFi 1.x.. Those > nodes that are configured that way in ZooKeeper need to be migrated to have > an open ACL. > * The toolkit will support a mode to read data from a configurable root node > in a source ZooKeeper, and the data will be written to a file designated via > CLI. > * The toolkit will support a mode to write data to a destination ZooKeeper > * The toolkit will not allow data to be written to the same ZooKeeper from > which the source data was obtained. > * The toolkit will not support reconnecting to ZooKeeper if it is > disconnected. The user can rerun the tool. > * The toolkit will support ZooKeepers configured with Kerberos. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit
[ https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15657907#comment-15657907 ] ASF GitHub Bot commented on NIFI-2957: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/1193 > ZooKeeper migration toolkit > --- > > Key: NIFI-2957 > URL: https://issues.apache.org/jira/browse/NIFI-2957 > Project: Apache NiFi > Issue Type: New Feature > Components: Tools and Build >Affects Versions: 1.0.0, 0.7.1 >Reporter: Jeff Storck >Assignee: Jeff Storck > Fix For: 1.2.0 > > > When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the > embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be > migrated. > Initial considerations: > * Username/password protection of nodes is not supported in NiFi 1.x.. Those > nodes that are configured that way in ZooKeeper need to be migrated to have > an open ACL. > * The toolkit will support a mode to read data from a configurable root node > in a source ZooKeeper, and the data will be written to a file designated via > CLI. > * The toolkit will support a mode to write data to a destination ZooKeeper > * The toolkit will not allow data to be written to the same ZooKeeper from > which the source data was obtained. > * The toolkit will not support reconnecting to ZooKeeper if it is > disconnected. The user can rerun the tool. > * The toolkit will support ZooKeepers configured with Kerberos. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit
[ https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15657902#comment-15657902 ] ASF subversion and git services commented on NIFI-2957: --- Commit ed6e03399f63d53aafdc2faadfa0878ff0b42188 in nifi's branch refs/heads/master from [~jtstorck] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=ed6e033 ] NIFI-2957 ZooKeeper Migration Toolkit Reads from and writes to a Zookeeper that is open or secured via user/password digest or SASL Supports persisting of data obtained from Zookeeper to a file or standard out as JSON Supports sending of Zookeeper data to Zookeeper from a file or standard out as JSON Does not allow data obtained from a source Zookeeper to be written back to the same Zookeeper connect string and path This closes #1193 Signed-off-by: Bryan Rosander > ZooKeeper migration toolkit > --- > > Key: NIFI-2957 > URL: https://issues.apache.org/jira/browse/NIFI-2957 > Project: Apache NiFi > Issue Type: New Feature > Components: Tools and Build >Affects Versions: 1.0.0, 0.7.1 >Reporter: Jeff Storck >Assignee: Jeff Storck > Fix For: 1.2.0 > > > When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the > embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be > migrated. > Initial considerations: > * Username/password protection of nodes is not supported in NiFi 1.x.. Those > nodes that are configured that way in ZooKeeper need to be migrated to have > an open ACL. > * The toolkit will support a mode to read data from a configurable root node > in a source ZooKeeper, and the data will be written to a file designated via > CLI. > * The toolkit will support a mode to write data to a destination ZooKeeper > * The toolkit will not allow data to be written to the same ZooKeeper from > which the source data was obtained. > * The toolkit will not support reconnecting to ZooKeeper if it is > disconnected. The user can rerun the tool. > * The toolkit will support ZooKeepers configured with Kerberos. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit
[ https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15657785#comment-15657785 ] ASF GitHub Bot commented on NIFI-2957: -- Github user jtstorck commented on the issue: https://github.com/apache/nifi/pull/1193 Also removed tests that specify a port for Zookeeper in favor of tests that check the connect string and path parsing from the command line. > ZooKeeper migration toolkit > --- > > Key: NIFI-2957 > URL: https://issues.apache.org/jira/browse/NIFI-2957 > Project: Apache NiFi > Issue Type: New Feature > Components: Tools and Build >Affects Versions: 1.0.0, 0.7.1 >Reporter: Jeff Storck >Assignee: Jeff Storck > Fix For: 1.2.0 > > > When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the > embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be > migrated. > Initial considerations: > * Username/password protection of nodes is not supported in NiFi 1.x.. Those > nodes that are configured that way in ZooKeeper need to be migrated to have > an open ACL. > * The toolkit will support a mode to read data from a configurable root node > in a source ZooKeeper, and the data will be written to a file designated via > CLI. > * The toolkit will support a mode to write data to a destination ZooKeeper > * The toolkit will not allow data to be written to the same ZooKeeper from > which the source data was obtained. > * The toolkit will not support reconnecting to ZooKeeper if it is > disconnected. The user can rerun the tool. > * The toolkit will support ZooKeepers configured with Kerberos. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit
[ https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15657514#comment-15657514 ] ASF GitHub Bot commented on NIFI-2957: -- Github user jtstorck commented on the issue: https://github.com/apache/nifi/pull/1193 @brosander New commit on the PR addresses the CNFE when connecting to ZK, and added some documentation to the code in ZooKeeperMigrator that writes to Zookeeper. > ZooKeeper migration toolkit > --- > > Key: NIFI-2957 > URL: https://issues.apache.org/jira/browse/NIFI-2957 > Project: Apache NiFi > Issue Type: New Feature > Components: Tools and Build >Affects Versions: 1.0.0, 0.7.1 >Reporter: Jeff Storck >Assignee: Jeff Storck > Fix For: 1.2.0 > > > When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the > embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be > migrated. > Initial considerations: > * Username/password protection of nodes is not supported in NiFi 1.x.. Those > nodes that are configured that way in ZooKeeper need to be migrated to have > an open ACL. > * The toolkit will support a mode to read data from a configurable root node > in a source ZooKeeper, and the data will be written to a file designated via > CLI. > * The toolkit will support a mode to write data to a destination ZooKeeper > * The toolkit will not allow data to be written to the same ZooKeeper from > which the source data was obtained. > * The toolkit will not support reconnecting to ZooKeeper if it is > disconnected. The user can rerun the tool. > * The toolkit will support ZooKeepers configured with Kerberos. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit
[
https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15656081#comment-15656081
]
ASF GitHub Bot commented on NIFI-2957:
--
Github user brosander commented on the issue:
https://github.com/apache/nifi/pull/1193
With the new build I'm seeing errors running though the steps
[here](https://github.com/jtstorck/docker-zk-krb#run-the-zk-migrator-to-read-the-protected-nodes-for-client-from-zookeeper).
```
2016-11-11 04:13:03,655 WARN [main-SendThread(zk-kerberos:2181)]
org.apache.zookeeper.ClientCnxn Session 0x0 for server null, unexpected error,
closing socket connection and attempting reconnect
java.lang.NoClassDefFoundError: org/apache/log4j/Logger
at org.apache.zookeeper.Login.(Login.java:44)
~[zookeeper-3.4.6.jar:3.4.6-1569965]
at
org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslClient(ZooKeeperSaslClient.java:226)
~[zookeeper-3.4.6.jar:3.4.6-1569965]
at
org.apache.zookeeper.client.ZooKeeperSaslClient.(ZooKeeperSaslClient.java:131)
~[zookeeper-3.4.6.jar:3.4.6-1569965]
at
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:949)
~[zookeeper-3.4.6.jar:3.4.6-1569965]
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
~[zookeeper-3.4.6.jar:3.4.6-1569965]
Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Logger
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
~[na:1.8.0_111]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
~[na:1.8.0_111]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
~[na:1.8.0_111]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
~[na:1.8.0_111]
... 5 common frames omitted
2016-11-11 04:13:03,713 INFO [main] o.a.n.t.zkmigrator.ZooKeeperMigrator
Persisting data from source ZooKeeper:
ZooKeeperEndpointConfig{connectString=zk-kerberos:2181, path=/}
Exception in thread "main" java.lang.RuntimeException: unable to perform
operation: KeeperErrorCode = ConnectionLoss for /
at
org.apache.nifi.toolkit.zkmigrator.ZooKeeperMigratorMain.main(ZooKeeperMigratorMain.java:145)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1500)
at
org.apache.nifi.toolkit.zkmigrator.ZooKeeperMigrator.getNode(ZooKeeperMigrator.java:188)
at
org.apache.nifi.toolkit.zkmigrator.ZooKeeperMigrator.readZooKeeper(ZooKeeperMigrator.java:95)
at
org.apache.nifi.toolkit.zkmigrator.ZooKeeperMigratorMain.main(ZooKeeperMigratorMain.java:137)
```
Unfortunately it looks like Zookeeper has [hard log4j
dependencies](https://github.com/apache/zookeeper/blob/release-3.4.6/src/java/main/org/apache/zookeeper/Login.java#L44).
> ZooKeeper migration toolkit
> ---
>
> Key: NIFI-2957
> URL: https://issues.apache.org/jira/browse/NIFI-2957
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Tools and Build
>Affects Versions: 1.0.0, 0.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
> Fix For: 1.2.0
>
>
> When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the
> embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be
> migrated.
> Initial considerations:
> * Username/password protection of nodes is not supported in NiFi 1.x.. Those
> nodes that are configured that way in ZooKeeper need to be migrated to have
> an open ACL.
> * The toolkit will support a mode to read data from a configurable root node
> in a source ZooKeeper, and the data will be written to a file designated via
> CLI.
> * The toolkit will support a mode to write data to a destination ZooKeeper
> * The toolkit will not allow data to be written to the same ZooKeeper from
> which the source data was obtained.
> * The toolkit will not support reconnecting to ZooKeeper if it is
> disconnected. The user can rerun the tool.
> * The toolkit will support ZooKeepers configured with Kerberos.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit
[ https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15655694#comment-15655694 ] ASF GitHub Bot commented on NIFI-2957: -- Github user jtstorck commented on the issue: https://github.com/apache/nifi/pull/1193 @brosander Updated the logging config to lessen the logging framework's output at startup and added a config to the assembly's classpath dir. > ZooKeeper migration toolkit > --- > > Key: NIFI-2957 > URL: https://issues.apache.org/jira/browse/NIFI-2957 > Project: Apache NiFi > Issue Type: New Feature > Components: Tools and Build >Affects Versions: 1.0.0, 0.7.1 >Reporter: Jeff Storck >Assignee: Jeff Storck > Fix For: 1.2.0 > > > When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the > embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be > migrated. > Initial considerations: > * Username/password protection of nodes is not supported in NiFi 1.x.. Those > nodes that are configured that way in ZooKeeper need to be migrated to have > an open ACL. > * The toolkit will support a mode to read data from a configurable root node > in a source ZooKeeper, and the data will be written to a file designated via > CLI. > * The toolkit will support a mode to write data to a destination ZooKeeper > * The toolkit will not allow data to be written to the same ZooKeeper from > which the source data was obtained. > * The toolkit will not support reconnecting to ZooKeeper if it is > disconnected. The user can rerun the tool. > * The toolkit will support ZooKeepers configured with Kerberos. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit
[ https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15655515#comment-15655515 ] ASF GitHub Bot commented on NIFI-2957: -- Github user jtstorck commented on a diff in the pull request: https://github.com/apache/nifi/pull/1193#discussion_r87508185 --- Diff: nifi-toolkit/nifi-toolkit-zookeeper-migrator/src/test/resources/logback.groovy --- @@ -0,0 +1,36 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +import ch.qos.logback.classic.encoder.PatternLayoutEncoder --- End diff -- Yes, I'll look into that. > ZooKeeper migration toolkit > --- > > Key: NIFI-2957 > URL: https://issues.apache.org/jira/browse/NIFI-2957 > Project: Apache NiFi > Issue Type: New Feature > Components: Tools and Build >Affects Versions: 1.0.0, 0.7.1 >Reporter: Jeff Storck >Assignee: Jeff Storck > Fix For: 1.2.0 > > > When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the > embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be > migrated. > Initial considerations: > * Username/password protection of nodes is not supported in NiFi 1.x.. Those > nodes that are configured that way in ZooKeeper need to be migrated to have > an open ACL. > * The toolkit will support a mode to read data from a configurable root node > in a source ZooKeeper, and the data will be written to a file designated via > CLI. > * The toolkit will support a mode to write data to a destination ZooKeeper > * The toolkit will not allow data to be written to the same ZooKeeper from > which the source data was obtained. > * The toolkit will not support reconnecting to ZooKeeper if it is > disconnected. The user can rerun the tool. > * The toolkit will support ZooKeepers configured with Kerberos. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit
[ https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15655358#comment-15655358 ] ASF GitHub Bot commented on NIFI-2957: -- Github user brosander commented on a diff in the pull request: https://github.com/apache/nifi/pull/1193#discussion_r87499434 --- Diff: nifi-toolkit/nifi-toolkit-zookeeper-migrator/src/test/resources/logback.groovy --- @@ -0,0 +1,36 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +import ch.qos.logback.classic.encoder.PatternLayoutEncoder --- End diff -- I like the dsl config option here but can we put it in the classpath/ folder of the toolkit assembly? That should get picked up still be editable without recompile. > ZooKeeper migration toolkit > --- > > Key: NIFI-2957 > URL: https://issues.apache.org/jira/browse/NIFI-2957 > Project: Apache NiFi > Issue Type: New Feature > Components: Tools and Build >Affects Versions: 1.0.0, 0.7.1 >Reporter: Jeff Storck >Assignee: Jeff Storck > Fix For: 1.2.0 > > > When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the > embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be > migrated. > Initial considerations: > * Username/password protection of nodes is not supported in NiFi 1.x.. Those > nodes that are configured that way in ZooKeeper need to be migrated to have > an open ACL. > * The toolkit will support a mode to read data from a configurable root node > in a source ZooKeeper, and the data will be written to a file designated via > CLI. > * The toolkit will support a mode to write data to a destination ZooKeeper > * The toolkit will not allow data to be written to the same ZooKeeper from > which the source data was obtained. > * The toolkit will not support reconnecting to ZooKeeper if it is > disconnected. The user can rerun the tool. > * The toolkit will support ZooKeepers configured with Kerberos. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit
[ https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15655346#comment-15655346 ] ASF GitHub Bot commented on NIFI-2957: -- Github user brosander commented on the issue: https://github.com/apache/nifi/pull/1193 @jtstorck having both logback and log4j on the classpath is causing slf4j to complain: ``` SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/nifi-toolkit/lib/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/nifi-toolkit/lib/slf4j-log4j12-1.7.12.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder] ``` > ZooKeeper migration toolkit > --- > > Key: NIFI-2957 > URL: https://issues.apache.org/jira/browse/NIFI-2957 > Project: Apache NiFi > Issue Type: New Feature > Components: Tools and Build >Affects Versions: 1.0.0, 0.7.1 >Reporter: Jeff Storck >Assignee: Jeff Storck > Fix For: 1.2.0 > > > When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the > embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be > migrated. > Initial considerations: > * Username/password protection of nodes is not supported in NiFi 1.x.. Those > nodes that are configured that way in ZooKeeper need to be migrated to have > an open ACL. > * The toolkit will support a mode to read data from a configurable root node > in a source ZooKeeper, and the data will be written to a file designated via > CLI. > * The toolkit will support a mode to write data to a destination ZooKeeper > * The toolkit will not allow data to be written to the same ZooKeeper from > which the source data was obtained. > * The toolkit will not support reconnecting to ZooKeeper if it is > disconnected. The user can rerun the tool. > * The toolkit will support ZooKeepers configured with Kerberos. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit
[ https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15654616#comment-15654616 ] ASF GitHub Bot commented on NIFI-2957: -- Github user jtstorck commented on the issue: https://github.com/apache/nifi/pull/1193 @brosander Code has been updated to allow sending data to the root of a Zookeeper server. Also, you shouldn't have to specify the port when sending data if Zookeeper is running on the default port. > ZooKeeper migration toolkit > --- > > Key: NIFI-2957 > URL: https://issues.apache.org/jira/browse/NIFI-2957 > Project: Apache NiFi > Issue Type: New Feature > Components: Tools and Build >Affects Versions: 1.0.0, 0.7.1 >Reporter: Jeff Storck >Assignee: Jeff Storck > Fix For: 1.2.0 > > > When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the > embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be > migrated. > Initial considerations: > * Username/password protection of nodes is not supported in NiFi 1.x.. Those > nodes that are configured that way in ZooKeeper need to be migrated to have > an open ACL. > * The toolkit will support a mode to read data from a configurable root node > in a source ZooKeeper, and the data will be written to a file designated via > CLI. > * The toolkit will support a mode to write data to a destination ZooKeeper > * The toolkit will not allow data to be written to the same ZooKeeper from > which the source data was obtained. > * The toolkit will not support reconnecting to ZooKeeper if it is > disconnected. The user can rerun the tool. > * The toolkit will support ZooKeepers configured with Kerberos. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit
[
https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15654447#comment-15654447
]
ASF GitHub Bot commented on NIFI-2957:
--
Github user brosander commented on the issue:
https://github.com/apache/nifi/pull/1193
@jtstorck I'm having some problems trying to save a path (/node) and then
restore at the same path in a new zookeeper instance.
Here's what I'm getting (with a teardown and reinstantiation of the
zookeeper container between receive and send commands) :
```
zk-migrator.sh -z zoo1/node -r -f /out/out.json
zk-migrator.sh -z zoo1:2181/ -s -f /out/out.json
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/opt/nifi-toolkit/lib/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/nifi-toolkit/lib/slf4j-log4j12-1.7.12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type
[ch.qos.logback.classic.util.ContextSelectorStaticBinder]
2016-11-10 16:14:26,590 INFO [main] o.a.n.t.zkmigrator.ZooKeeperMigrator
Source data was obtained from ZooKeeper:
ZooKeeperEndpointConfig{connectString=zoo1, path=/node}
2016-11-10 16:14:26,621 INFO [ForkJoinPool.commonPool-worker-1]
o.a.n.t.zkmigrator.ZooKeeperMigrator transformed original node
DataStatAclNode{path=/node, acls=[31,s{'world,'anyone}
], ephemeralOwner=0} to DataStatAclNode{path=//node,
acls=[31,s{'world,'anyone}
], ephemeralOwner=0}
Exception in thread "main" java.lang.RuntimeException: unable to perform
operation: java.lang.IllegalArgumentException: Invalid path string "//node"
caused by empty node name specified @1
at
org.apache.nifi.toolkit.zkmigrator.ZooKeeperMigratorMain.main(ZooKeeperMigratorMain.java:145)
Caused by: java.util.concurrent.ExecutionException:
java.lang.IllegalArgumentException: Invalid path string "//node" caused by
empty node name specified @1
at
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at
org.apache.nifi.toolkit.zkmigrator.ZooKeeperMigrator.writeZooKeeper(ZooKeeperMigrator.java:171)
at
org.apache.nifi.toolkit.zkmigrator.ZooKeeperMigratorMain.main(ZooKeeperMigratorMain.java:139)
Caused by: java.lang.IllegalArgumentException: Invalid path string "//node"
caused by empty node name specified @1
at
org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:99)
at
org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:35)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:766)
at
org.apache.nifi.toolkit.zkmigrator.ZooKeeperMigrator.ensureNodeExists(ZooKeeperMigrator.java:214)
at
org.apache.nifi.toolkit.zkmigrator.ZooKeeperMigrator.lambda$null$5(ZooKeeperMigrator.java:159)
at
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1595)
at
java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1582)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at
java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at
java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
```
> ZooKeeper migration toolkit
> ---
>
> Key: NIFI-2957
> URL: https://issues.apache.org/jira/browse/NIFI-2957
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Tools and Build
>Affects Versions: 1.0.0, 0.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
> Fix For: 1.2.0
>
>
> When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the
> embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be
> migrated.
> Initial considerations:
> * Username/password protection of nodes is not supported in NiFi 1.x.. Those
> nodes that are configured that way in ZooKeeper need to be migrated to have
> an open ACL.
> * The toolkit will support a mode to read data from a configurable root node
> in a source ZooKeeper, and the data will be written to a file designated via
> CLI.
> * The toolkit will support a mode to wri
[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit
[
https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15654390#comment-15654390
]
ASF GitHub Bot commented on NIFI-2957:
--
Github user brosander commented on the issue:
https://github.com/apache/nifi/pull/1193
@jtstorck I'm getting logging messages in stdout when I don't specify a
file:
```
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/opt/nifi-toolkit/lib/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/nifi-toolkit/lib/slf4j-log4j12-1.7.12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type
[ch.qos.logback.classic.util.ContextSelectorStaticBinder]
2016-11-10 15:53:36,446 INFO [main] o.a.n.t.zkmigrator.ZooKeeperMigrator
Persisting data from source ZooKeeper:
ZooKeeperEndpointConfig{connectString=zoo1, path=/node}
[
{
"connectString": "zoo1",
"path": "/node"
},
{
"path": "/node",
"data": [
99,
111,
110,
116,
101,
110,
116
],
"stat": {
"czxid": 9,
"mzxid": 9,
"ctime": 1478792367151,
"mtime": 1478792367151,
"version": 0,
"cversion": 0,
"aversion": 0,
"ephemeralOwner": 0,
"dataLength": 7,
"numChildren": 0,
"pzxid": 9
},
"acls": [
{
"perms": 31,
"id": {
"scheme": "world",
"id": "anyone"
}
}
],
"ephemeralOwner": 0
}
]2016-11-10 15:53:36,496 INFO [main] o.a.n.t.zkmigrator.ZooKeeperMigrator 1
nodes read from ZooKeeperEndpointConfig{connectString=zoo1, path=/node}
```
> ZooKeeper migration toolkit
> ---
>
> Key: NIFI-2957
> URL: https://issues.apache.org/jira/browse/NIFI-2957
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Tools and Build
>Affects Versions: 1.0.0, 0.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
> Fix For: 1.2.0
>
>
> When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the
> embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be
> migrated.
> Initial considerations:
> * Username/password protection of nodes is not supported in NiFi 1.x.. Those
> nodes that are configured that way in ZooKeeper need to be migrated to have
> an open ACL.
> * The toolkit will support a mode to read data from a configurable root node
> in a source ZooKeeper, and the data will be written to a file designated via
> CLI.
> * The toolkit will support a mode to write data to a destination ZooKeeper
> * The toolkit will not allow data to be written to the same ZooKeeper from
> which the source data was obtained.
> * The toolkit will not support reconnecting to ZooKeeper if it is
> disconnected. The user can rerun the tool.
> * The toolkit will support ZooKeepers configured with Kerberos.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit
[ https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15651843#comment-15651843 ] ASF GitHub Bot commented on NIFI-2957: -- GitHub user jtstorck opened a pull request: https://github.com/apache/nifi/pull/1193 NIFI-2957 ZooKeeper Migration Toolkit Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. Reads from and writes to a Zookeeper that is open or secured via user/password digest or SASL Supports persisting of data obtained from Zookeeper to a file or standard out as JSON Supports sending of Zookeeper data to Zookeeper from a file or standard out as JSON Does not allow data obtained from a source Zookeeper to be written back to the same Zookeeper connect string and path You can merge this pull request into a Git repository by running: $ git pull https://github.com/jtstorck/nifi NIFI-2957 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1193.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1193 commit 7b7bec120164bb2eb8fb7b0018ad5b5fa4fed6c7 Author: Jeff Storck Date: 2016-10-27T16:21:20Z NIFI-2957 ZooKeeper Migration Toolkit Reads from and writes to a Zookeeper that is open or secured via user/password digest or SASL Supports persisting of data obtained from Zookeeper to a file or standard out as JSON Supports sending of Zookeeper data to Zookeeper from a file or standard out as JSON Does not allow data obtained from a source Zookeeper to be written back to the same Zookeeper connect string and path > ZooKeeper migration toolkit > --- > > Key: NIFI-2957 > URL: https://issues.apache.org/jira/browse/NIFI-2957 > Project: Apache NiFi > Issue Type: New Feature > Components: Tools and Build >Affects Versions: 1.0.0, 0.7.1 >Reporter: Jeff Storck >Assignee: Jeff Storck > Fix For: 1.2.0 > > > When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the > embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be > migrated. > Initial considerations: > * Username/password protection of nodes is not supported in NiFi 1.x.. Those > nodes that are configured that way in ZooKeeper need to be migrated to have > an open ACL. > * The toolkit will support a mode to read data from a configurable root node > in a source ZooKeeper, and the data will be written to a file designated via > CLI. > * The toolkit will support a mode to write data to a destination ZooKeeper > * The toolkit will not allow data to be written to the same ZooKeeper from > which the source data was obtained. > * The toolkit will not support reconnecting to ZooKeeper if it is > disconnected. The user can rerun the tool. > * The toolkit will support ZooKeepers configured with Kerberos. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
