[GitHub] nifi-minifi pull request #78: MINIFI-251 - Added explicit Java runtime argum...
GitHub user trixpan opened a pull request: https://github.com/apache/nifi-minifi/pull/78 MINIFI-251 - Added explicit Java runtime argument to default bootstra⦠â¦.conf to avoid blocking on VM deployment. Thank you for submitting a contribution to Apache NiFi - MiNiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi-minifi folder? - [X] ~~~Have you written or updated unit tests to verify your changes?~~~ - [X] ~~~If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?~~~ - [X] ~~~If applicable, have you updated the LICENSE file, including the main LICENSE file under minifi-assembly?~~~ - [X] ~~~If applicable, have you updated the NOTICE file, including the main NOTICE file found under minifi-assembly?~~~ ### For documentation related changes: - [X] ~~~Have you ensured that format looks appropriate for the output in which it is rendered?~~~ ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/trixpan/nifi-minifi MINIFI-251 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi/pull/78.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #78 commit 26f0cde711ecb34cf556b1e43d7246ea34c9e2da Author: Andre F de Miranda Date: 2017-03-25T06:04:39Z MINIFI-251 - Added explicit Java runtime argument to default bootstrap.conf to avoid blocking on VM deployment. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request #77: MINIFI-251 - Added explicit Java runtime argum...
Github user trixpan closed the pull request at: https://github.com/apache/nifi-minifi/pull/77 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request #77: MINFI-251 - Added explicit Java runtime argume...
GitHub user trixpan opened a pull request: https://github.com/apache/nifi-minifi/pull/77 MINFI-251 - Added explicit Java runtime argument to default bootstrap⦠â¦.conf to avoid blocking on VM deployment. Thank you for submitting a contribution to Apache NiFi - MiNiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi-minifi folder? - [X] ~~~Have you written or updated unit tests to verify your changes?~~~ - [X] ~~~If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?~~~ - [X] ~~~If applicable, have you updated the LICENSE file, including the main LICENSE file under minifi-assembly?~~~ - [X] ~~~If applicable, have you updated the NOTICE file, including the main NOTICE file found under minifi-assembly?~~~ ### For documentation related changes: - [X] ~~~Have you ensured that format looks appropriate for the output in which it is rendered?~~~ ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/trixpan/nifi-minifi MINIFI-251 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi/pull/77.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #77 commit f12a70c1c692f6d07ecf4eae13a9215552a3dd77 Author: Andre F de Miranda Date: 2017-03-25T06:04:39Z MINFI-251 - Added explicit Java runtime argument to default bootstrap.conf to avoid blocking on VM deployment. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3413) Implement a GetChangeDataCapture processor
[ https://issues.apache.org/jira/browse/NIFI-3413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941343#comment-15941343 ] ASF GitHub Bot commented on NIFI-3413: -- Github user phrocker commented on the issue: https://github.com/apache/nifi/pull/1618 @mattyb149 Cool stuff. I used your template because it was an easier place to start. Seems pretty good. I'll continue to test at my leisure but I'm happy with it presently. I tried to break it by stopping mysql mid cycle and BinaryLogClient handled it gracefully and all was well. Besides the NPE above I'm happy thus far. I'll continue to play with it for my own edification. Let me know if you wish me to test the NPE again or not. > Implement a GetChangeDataCapture processor > -- > > Key: NIFI-3413 > URL: https://issues.apache.org/jira/browse/NIFI-3413 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess > > Database systems such as MySQL, Oracle, and SQL Server allow access to their > transactional logs and such, in order for external clients to have a "change > data capture" (CDC) capability. I propose a GetChangeDataCapture processor to > enable this in NiFi. > The processor would be configured with a DBCPConnectionPool controller > service, as well as a Database Type property (similar to the one in > QueryDatabaseTable) for database-specific handling. Additional properties > might include the CDC table name, etc. Additional database-specific > properties could be handled using dynamic properties (and the documentation > should reflect this). > The processor would accept no incoming connections (it is a "Get" or source > processor), would be intended to run on the primary node only as a single > threaded processor, and would generate a flow file for each operation > (INSERT, UPDATE, DELETE, e,g,) in one or some number of formats (JSON, e.g.). > The flow files would be transferred in time order (to enable a replication > solution, for example), perhaps with some auto-incrementing attribute to also > indicate order if need be. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi issue #1618: NIFI-3413: Add GetChangeDataCaptureMySQL processor
Github user phrocker commented on the issue: https://github.com/apache/nifi/pull/1618 @mattyb149 Cool stuff. I used your template because it was an easier place to start. Seems pretty good. I'll continue to test at my leisure but I'm happy with it presently. I tried to break it by stopping mysql mid cycle and BinaryLogClient handled it gracefully and all was well. Besides the NPE above I'm happy thus far. I'll continue to play with it for my own edification. Let me know if you wish me to test the NPE again or not. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-79) Add support for bulletin level 'NONE'
[ https://issues.apache.org/jira/browse/NIFI-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941305#comment-15941305 ] ASF GitHub Bot commented on NIFI-79: Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/1622 @mcgilman would you be able to give me a hand by reviewing this PR? I was able to reproduce the desired behavior but not sure if the implementation is adequate > Add support for bulletin level 'NONE' > - > > Key: NIFI-79 > URL: https://issues.apache.org/jira/browse/NIFI-79 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework, Core UI >Reporter: Joseph Witt >Assignee: Andre F de Miranda >Priority: Trivial > > sometimes you just don't care what a processor has to say -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi issue #1622: NIFI-79 - Introduces the ability to set Bulletin Level to ...
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/1622 @mcgilman would you be able to give me a hand by reviewing this PR? I was able to reproduce the desired behavior but not sure if the implementation is adequate --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-79) Add support for bulletin level 'NONE'
[ https://issues.apache.org/jira/browse/NIFI-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941297#comment-15941297 ] ASF GitHub Bot commented on NIFI-79: GitHub user trixpan opened a pull request: https://github.com/apache/nifi/pull/1622 NIFI-79 - Introduces the ability to set Bulletin Level to NONE Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [X] Have you written or updated unit tests to verify your changes? - [X] ~~~If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?~~~ - [X] ~~~If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?~~~ - [X] ~~~If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?~~~ - [X] ~~~If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?~~~ ### For documentation related changes: - [X] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/trixpan/nifi NIFI-79 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1622.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1622 commit 162d27eca532f298e6341cf63041548b532ef389 Author: Andre F de Miranda Date: 2017-03-24T22:47:42Z NIFI-79 - Introduces the ability to set Bulletin Level to NONE > Add support for bulletin level 'NONE' > - > > Key: NIFI-79 > URL: https://issues.apache.org/jira/browse/NIFI-79 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework, Core UI >Reporter: Joseph Witt >Assignee: Andre F de Miranda >Priority: Trivial > > sometimes you just don't care what a processor has to say -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi pull request #1622: NIFI-79 - Introduces the ability to set Bulletin Le...
GitHub user trixpan opened a pull request: https://github.com/apache/nifi/pull/1622 NIFI-79 - Introduces the ability to set Bulletin Level to NONE Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [X] Have you written or updated unit tests to verify your changes? - [X] ~~~If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?~~~ - [X] ~~~If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?~~~ - [X] ~~~If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?~~~ - [X] ~~~If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?~~~ ### For documentation related changes: - [X] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/trixpan/nifi NIFI-79 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1622.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1622 commit 162d27eca532f298e6341cf63041548b532ef389 Author: Andre F de Miranda Date: 2017-03-24T22:47:42Z NIFI-79 - Introduces the ability to set Bulletin Level to NONE --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3413) Implement a GetChangeDataCapture processor
[ https://issues.apache.org/jira/browse/NIFI-3413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941265#comment-15941265 ] ASF GitHub Bot commented on NIFI-3413: -- Github user phrocker commented on the issue: https://github.com/apache/nifi/pull/1618 @mattyb149 I began with an empty password for my user and thus set an empty string. I saw an NPE in the logs, which did go away once I set a password for my user and subsequently set it in processor config. 2017-03-24 18:19:53,150 ERROR [blc-localhost:3306] org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[blc-localhost:3306,5,main]: java.lang.NullPointerException 2017-03-24 18:19:53,155 ERROR [blc-localhost:3306] org.apache.nifi.NiFi java.lang.NullPointerException: null at com.github.shyiko.mysql.binlog.network.protocol.command.AuthenticateCommand.passwordCompatibleWithMySQL411(AuthenticateCommand.java:89) ~[mysql-binlog-connector-java-0.8.1.jar:na] at com.github.shyiko.mysql.binlog.network.protocol.command.AuthenticateCommand.toByteArray(AuthenticateCommand.java:70) ~[mysql-binlog-connector-java-0.8.1.jar:na] at com.github.shyiko.mysql.binlog.network.protocol.PacketChannel.write(PacketChannel.java:65) ~[mysql-binlog-connector-java-0.8.1.jar:na] at com.github.shyiko.mysql.binlog.BinaryLogClient.authenticate(BinaryLogClient.java:546) ~[mysql-binlog-connector-java-0.8.1.jar:na] at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:407) ~[mysql-binlog-connector-java-0.8.1.jar:na] at com.github.shyiko.mysql.binlog.BinaryLogClient$5.run(BinaryLogClient.java:635) ~[mysql-binlog-connector-java-0.8.1.jar:na] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121] > Implement a GetChangeDataCapture processor > -- > > Key: NIFI-3413 > URL: https://issues.apache.org/jira/browse/NIFI-3413 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess > > Database systems such as MySQL, Oracle, and SQL Server allow access to their > transactional logs and such, in order for external clients to have a "change > data capture" (CDC) capability. I propose a GetChangeDataCapture processor to > enable this in NiFi. > The processor would be configured with a DBCPConnectionPool controller > service, as well as a Database Type property (similar to the one in > QueryDatabaseTable) for database-specific handling. Additional properties > might include the CDC table name, etc. Additional database-specific > properties could be handled using dynamic properties (and the documentation > should reflect this). > The processor would accept no incoming connections (it is a "Get" or source > processor), would be intended to run on the primary node only as a single > threaded processor, and would generate a flow file for each operation > (INSERT, UPDATE, DELETE, e,g,) in one or some number of formats (JSON, e.g.). > The flow files would be transferred in time order (to enable a replication > solution, for example), perhaps with some auto-incrementing attribute to also > indicate order if need be. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi issue #1618: NIFI-3413: Add GetChangeDataCaptureMySQL processor
Github user phrocker commented on the issue: https://github.com/apache/nifi/pull/1618 @mattyb149 I began with an empty password for my user and thus set an empty string. I saw an NPE in the logs, which did go away once I set a password for my user and subsequently set it in processor config. 2017-03-24 18:19:53,150 ERROR [blc-localhost:3306] org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[blc-localhost:3306,5,main]: java.lang.NullPointerException 2017-03-24 18:19:53,155 ERROR [blc-localhost:3306] org.apache.nifi.NiFi java.lang.NullPointerException: null ââat com.github.shyiko.mysql.binlog.network.protocol.command.AuthenticateCommand.passwordCompatibleWithMySQL411(AuthenticateCommand.java:89) ~[mysql-binlog-connector-java-0.8.1.jar:na] ââat com.github.shyiko.mysql.binlog.network.protocol.command.AuthenticateCommand.toByteArray(AuthenticateCommand.java:70) ~[mysql-binlog-connector-java-0.8.1.jar:na] ââat com.github.shyiko.mysql.binlog.network.protocol.PacketChannel.write(PacketChannel.java:65) ~[mysql-binlog-connector-java-0.8.1.jar:na] ââat com.github.shyiko.mysql.binlog.BinaryLogClient.authenticate(BinaryLogClient.java:546) ~[mysql-binlog-connector-java-0.8.1.jar:na] ââat com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:407) ~[mysql-binlog-connector-java-0.8.1.jar:na] ââat com.github.shyiko.mysql.binlog.BinaryLogClient$5.run(BinaryLogClient.java:635) ~[mysql-binlog-connector-java-0.8.1.jar:na] ââat java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121] --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (NIFI-3304) In the Administration Guide, there is an incorrect JAVA 8 parameter for bootstrap.conf file
[ https://issues.apache.org/jira/browse/NIFI-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Moser updated NIFI-3304: Status: Patch Available (was: Open) > In the Administration Guide, there is an incorrect JAVA 8 parameter for > bootstrap.conf file > --- > > Key: NIFI-3304 > URL: https://issues.apache.org/jira/browse/NIFI-3304 > Project: Apache NiFi > Issue Type: Bug > Components: Documentation & Website >Reporter: David A. Wynne >Assignee: Michael Moser >Priority: Minor > > In the Administration guide, section Bootstrap properties, there is a > paragraph about Java 8 handling of codecache, the listed properties are > accurate for Java version 7, but for Java version 8, one has changed > This is correct for Java version 7: > java.arg.8=-XX:CodeCacheFlushingMinimumFreeSpace=10m > But for Java version 8, it should be > java.arg.8=-XX:CodeCacheMinimumFreeSpace=10m -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3304) In the Administration Guide, there is an incorrect JAVA 8 parameter for bootstrap.conf file
[ https://issues.apache.org/jira/browse/NIFI-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941208#comment-15941208 ] ASF GitHub Bot commented on NIFI-3304: -- GitHub user mosermw opened a pull request: https://github.com/apache/nifi/pull/1621 NIFI-3304 removed unneeded Java 8 codecache info from Admin Guide Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mosermw/nifi nifi-3304 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1621.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1621 commit 858bc4dc49e299f815c5ad3ae5e6dbdcebfcd575 Author: Mike Moser Date: 2017-03-24T21:44:07Z NIFI-3304 removed unneeded Java 8 codecache info from Admin Guide > In the Administration Guide, there is an incorrect JAVA 8 parameter for > bootstrap.conf file > --- > > Key: NIFI-3304 > URL: https://issues.apache.org/jira/browse/NIFI-3304 > Project: Apache NiFi > Issue Type: Bug > Components: Documentation & Website >Reporter: David A. Wynne >Assignee: Michael Moser >Priority: Minor > > In the Administration guide, section Bootstrap properties, there is a > paragraph about Java 8 handling of codecache, the listed properties are > accurate for Java version 7, but for Java version 8, one has changed > This is correct for Java version 7: > java.arg.8=-XX:CodeCacheFlushingMinimumFreeSpace=10m > But for Java version 8, it should be > java.arg.8=-XX:CodeCacheMinimumFreeSpace=10m -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi pull request #1621: NIFI-3304 removed unneeded Java 8 codecache info fr...
GitHub user mosermw opened a pull request: https://github.com/apache/nifi/pull/1621 NIFI-3304 removed unneeded Java 8 codecache info from Admin Guide Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mosermw/nifi nifi-3304 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1621.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1621 commit 858bc4dc49e299f815c5ad3ae5e6dbdcebfcd575 Author: Mike Moser Date: 2017-03-24T21:44:07Z NIFI-3304 removed unneeded Java 8 codecache info from Admin Guide --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3413) Implement a GetChangeDataCapture processor
[ https://issues.apache.org/jira/browse/NIFI-3413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941189#comment-15941189 ] ASF GitHub Bot commented on NIFI-3413: -- Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi/pull/1618#discussion_r107914632 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/impl/mysql/event/io/DeleteRowsWriter.java --- @@ -0,0 +1,90 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard.db.impl.mysql.event.io; + +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processors.standard.GetChangeDataCaptureMySQL; +import org.apache.nifi.processors.standard.db.event.ColumnDefinition; +import org.apache.nifi.processors.standard.db.impl.mysql.event.DeleteRowsEventInfo; + +import java.io.IOException; +import java.io.Serializable; +import java.util.BitSet; +import java.util.concurrent.atomic.AtomicLong; + +import static org.apache.nifi.processors.standard.db.impl.mysql.MySQLCDCUtils.getWritableObject; + +/** + * A writer class to output MySQL binlog "delete rows" events to flow file(s). + */ +public class DeleteRowsWriter extends AbstractBinlogTableEventWriter { + +/** + * Creates and transfers a new flow file whose contents are the JSON-serialized value of the specified event, and the sequence ID attribute set + * + * @param session A reference to a ProcessSession from which the flow file(s) will be created and transferred + * @param eventInfo An event whose value will become the contents of the flow file + * @return The next available CDC sequence ID for use by the CDC processor + */ +public long writeEvent(final ProcessSession session, final DeleteRowsEventInfo eventInfo, final long currentSequenceId) { +final AtomicLong seqId = new AtomicLong(currentSequenceId); --- End diff -- The correct answer is that I'm usually missing something... > Implement a GetChangeDataCapture processor > -- > > Key: NIFI-3413 > URL: https://issues.apache.org/jira/browse/NIFI-3413 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess > > Database systems such as MySQL, Oracle, and SQL Server allow access to their > transactional logs and such, in order for external clients to have a "change > data capture" (CDC) capability. I propose a GetChangeDataCapture processor to > enable this in NiFi. > The processor would be configured with a DBCPConnectionPool controller > service, as well as a Database Type property (similar to the one in > QueryDatabaseTable) for database-specific handling. Additional properties > might include the CDC table name, etc. Additional database-specific > properties could be handled using dynamic properties (and the documentation > should reflect this). > The processor would accept no incoming connections (it is a "Get" or source > processor), would be intended to run on the primary node only as a single > threaded processor, and would generate a flow file for each operation > (INSERT, UPDATE, DELETE, e,g,) in one or some number of formats (JSON, e.g.). > The flow files would be transferred in time order (to enable a replication > solution, for example), perhaps with some auto-incrementing attribute to also > indicate order if need be. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3413) Implement a GetChangeDataCapture processor
[ https://issues.apache.org/jira/browse/NIFI-3413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941184#comment-15941184 ] ASF GitHub Bot commented on NIFI-3413: -- Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi/pull/1618#discussion_r107921926 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetChangeDataCaptureMySQL.java --- @@ -0,0 +1,879 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard; + +import com.github.shyiko.mysql.binlog.BinaryLogClient; +import com.github.shyiko.mysql.binlog.event.Event; +import com.github.shyiko.mysql.binlog.event.EventHeaderV4; +import com.github.shyiko.mysql.binlog.event.EventType; +import com.github.shyiko.mysql.binlog.event.QueryEventData; +import com.github.shyiko.mysql.binlog.event.RotateEventData; +import com.github.shyiko.mysql.binlog.event.TableMapEventData; +import org.apache.nifi.annotation.behavior.DynamicProperties; +import org.apache.nifi.annotation.behavior.DynamicProperty; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.Stateful; +import org.apache.nifi.annotation.behavior.TriggerSerially; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.annotation.lifecycle.OnStopped; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.PropertyValue; +import org.apache.nifi.components.state.Scope; +import org.apache.nifi.components.state.StateManager; +import org.apache.nifi.components.state.StateMap; +import org.apache.nifi.distributed.cache.client.Deserializer; +import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient; +import org.apache.nifi.distributed.cache.client.Serializer; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.processor.AbstractSessionFactoryProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessSessionFactory; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.processors.standard.db.CDCException; +import org.apache.nifi.processors.standard.db.event.ColumnDefinition; +import org.apache.nifi.processors.standard.db.event.RowEventException; +import org.apache.nifi.processors.standard.db.event.TableInfo; +import org.apache.nifi.processors.standard.db.event.TableInfoCacheKey; +import org.apache.nifi.processors.standard.db.event.io.EventWriter; +import org.apache.nifi.processors.standard.db.impl.mysql.event.BeginTransactionEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.RawBinlogEvent; +import org.apache.nifi.processors.standard.db.impl.mysql.BinlogEventListener; +import org.apache.nifi.processors.standard.db.impl.mysql.event.BinlogEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.CommitTransactionEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.DeleteRowsEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.SchemaChangeEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.UpdateRowsEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.InsertRowsEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.io.BeginTransactionEventWriter; +import org.apache.nifi.processors.st
[jira] [Commented] (NIFI-3413) Implement a GetChangeDataCapture processor
[ https://issues.apache.org/jira/browse/NIFI-3413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941187#comment-15941187 ] ASF GitHub Bot commented on NIFI-3413: -- Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi/pull/1618#discussion_r107908219 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/event/ColumnDefinition.java --- @@ -0,0 +1,70 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard.db.event; + +/** + * A class that specifies a definition for a relational table column, including type, name, etc. + */ +public class ColumnDefinition { + +private byte type; +private String name = ""; + +public ColumnDefinition(byte type) { --- End diff -- Is there a way to avoid a byte and use an enumeration? I may answer this myself as I continue the review... > Implement a GetChangeDataCapture processor > -- > > Key: NIFI-3413 > URL: https://issues.apache.org/jira/browse/NIFI-3413 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess > > Database systems such as MySQL, Oracle, and SQL Server allow access to their > transactional logs and such, in order for external clients to have a "change > data capture" (CDC) capability. I propose a GetChangeDataCapture processor to > enable this in NiFi. > The processor would be configured with a DBCPConnectionPool controller > service, as well as a Database Type property (similar to the one in > QueryDatabaseTable) for database-specific handling. Additional properties > might include the CDC table name, etc. Additional database-specific > properties could be handled using dynamic properties (and the documentation > should reflect this). > The processor would accept no incoming connections (it is a "Get" or source > processor), would be intended to run on the primary node only as a single > threaded processor, and would generate a flow file for each operation > (INSERT, UPDATE, DELETE, e,g,) in one or some number of formats (JSON, e.g.). > The flow files would be transferred in time order (to enable a replication > solution, for example), perhaps with some auto-incrementing attribute to also > indicate order if need be. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3413) Implement a GetChangeDataCapture processor
[ https://issues.apache.org/jira/browse/NIFI-3413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941182#comment-15941182 ] ASF GitHub Bot commented on NIFI-3413: -- Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi/pull/1618#discussion_r107909853 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/event/TableInfo.java --- @@ -0,0 +1,146 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard.db.event; + +import org.apache.nifi.distributed.cache.client.exception.DeserializationException; +import org.apache.nifi.distributed.cache.client.exception.SerializationException; + +import java.io.IOException; +import java.io.OutputStream; +import java.util.ArrayList; +import java.util.List; +import java.util.stream.Collectors; + +/** + * A POJO for holding table information related to update events. + */ +public class TableInfo { + +final static String DB_TABLE_NAME_DELIMITER = "@!@"; + +private String databaseName; +private String tableName; +private Long tableId; +private List columns; + +public TableInfo(String databaseName, String tableName, Long tableId, List columns) { +this.databaseName = databaseName; +this.tableName = tableName; +this.tableId = tableId; +this.columns = columns; +} + +public String getDatabaseName() { +return databaseName; +} + +public String getTableName() { +return tableName; +} + +public void setTableName(String tableName) { +this.tableName = tableName; +} + +public Long getTableId() { +return tableId; +} + +public List getColumns() { +return columns; +} + +public void setColumns(List columns) { +this.columns = columns; +} + +@Override +public boolean equals(Object o) { +if (this == o) return true; +if (o == null || getClass() != o.getClass()) return false; + +TableInfo tableInfo = (TableInfo) o; + +if (!databaseName.equals(tableInfo.databaseName)) return false; +if (!tableName.equals(tableInfo.tableName)) return false; +if (!tableId.equals(tableInfo.tableId)) return false; +return columns != null ? columns.equals(tableInfo.columns) : tableInfo.columns == null; +} + +@Override +public int hashCode() { +int result = databaseName.hashCode(); +result = 31 * result + tableName.hashCode(); +result = 31 * result + tableId.hashCode(); +result = 31 * result + (columns != null ? columns.hashCode() : 0); +return result; +} + +public static class Serializer implements org.apache.nifi.distributed.cache.client.Serializer { + +@Override +public void serialize(TableInfo value, OutputStream output) throws SerializationException, IOException { +StringBuilder sb = new StringBuilder(value.getDatabaseName()); +sb.append(DB_TABLE_NAME_DELIMITER); +sb.append(value.getTableName()); +sb.append(DB_TABLE_NAME_DELIMITER); +sb.append(value.getTableId()); +List columnDefinitions = value.getColumns(); +if (columnDefinitions != null && !columnDefinitions.isEmpty()) { +sb.append(DB_TABLE_NAME_DELIMITER); +sb.append(columnDefinitions.stream().map((col) -> col.getName() + DB_TABLE_NAME_DELIMITER + col.getType()).collect(Collectors.joining(DB_TABLE_NAME_DELIMITER))); +} +output.write(sb.toString().getBytes()); +} +} + +publ
[jira] [Commented] (NIFI-3413) Implement a GetChangeDataCapture processor
[ https://issues.apache.org/jira/browse/NIFI-3413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941188#comment-15941188 ] ASF GitHub Bot commented on NIFI-3413: -- Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi/pull/1618#discussion_r107917776 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/impl/mysql/event/io/InsertRowsWriter.java --- @@ -0,0 +1,90 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard.db.impl.mysql.event.io; + +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processors.standard.GetChangeDataCaptureMySQL; +import org.apache.nifi.processors.standard.db.event.ColumnDefinition; +import org.apache.nifi.processors.standard.db.impl.mysql.event.InsertRowsEventInfo; + +import java.io.IOException; +import java.io.Serializable; +import java.util.BitSet; +import java.util.concurrent.atomic.AtomicLong; + +import static org.apache.nifi.processors.standard.db.impl.mysql.MySQLCDCUtils.getWritableObject; + +/** + * A writer class to output MySQL binlog "write rows" (aka INSERT) events to flow file(s). + */ +public class InsertRowsWriter extends AbstractBinlogTableEventWriter { + +/** + * Creates and transfers a new flow file whose contents are the JSON-serialized value of the specified event, and the sequence ID attribute set + * + * @param session A reference to a ProcessSession from which the flow file(s) will be created and transferred + * @param eventInfo An event whose value will become the contents of the flow file + * @return The next available CDC sequence ID for use by the CDC processor + */ +public long writeEvent(final ProcessSession session, final InsertRowsEventInfo eventInfo, final long currentSequenceId) { +final AtomicLong seqId = new AtomicLong(currentSequenceId); +for (Serializable[] row : eventInfo.getRows()) { + +FlowFile flowFile = session.create(); +flowFile = session.write(flowFile, outputStream -> { + +super.startJson(outputStream, eventInfo); +super.writeJson(eventInfo); + +final BitSet bitSet = eventInfo.getIncludedColumns(); +writeRow(eventInfo, row, bitSet); + +super.endJson(); +}); + +flowFile = session.putAllAttributes(flowFile, getCommonAttributes(seqId.get(), eventInfo)); +session.transfer(flowFile, GetChangeDataCaptureMySQL.REL_SUCCESS); +seqId.getAndIncrement(); +} +return seqId.get(); +} + +protected void writeRow(InsertRowsEventInfo event, Serializable[] row, BitSet includedColumns) throws IOException { +jsonGenerator.writeArrayFieldStart("columns"); +int i = includedColumns.nextSetBit(0); +while (i != -1) { +jsonGenerator.writeStartObject(); +jsonGenerator.writeNumberField("id", i + 1); +ColumnDefinition columnDefinition = event.getColumnByIndex(i); +Integer columnType = null; +if (columnDefinition != null) { +jsonGenerator.writeStringField("name", columnDefinition.getName()); +columnType = (int) columnDefinition.getType(); --- End diff -- I left a comment earlier about this. Could we avoid the autoboxing and make make type a Long instead of an enumeration? > Implement a GetChangeDataCapture processor > -- > > Key: NIFI-3413 > URL: https://issues.apache.org/jira/browse/NIFI-3413 > Project: Apache NiFi > Issue Type: New Fe
[jira] [Commented] (NIFI-3413) Implement a GetChangeDataCapture processor
[ https://issues.apache.org/jira/browse/NIFI-3413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941186#comment-15941186 ] ASF GitHub Bot commented on NIFI-3413: -- Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi/pull/1618#discussion_r107912643 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/impl/mysql/event/io/AbstractBinlogTableEventWriter.java --- @@ -0,0 +1,65 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard.db.impl.mysql.event.io; + +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processors.standard.GetChangeDataCaptureMySQL; +import org.apache.nifi.processors.standard.db.impl.mysql.event.BinlogTableEventInfo; + +import java.io.IOException; + +/** + * An abstract base class for writing MYSQL binlog events into flow file(s), e.g. + */ +public abstract class AbstractBinlogTableEventWriter extends AbstractBinlogEventWriter { + +protected void writeJson(T event) throws IOException { +super.writeJson(event); +if (event.getDatabaseName() != null) { +jsonGenerator.writeStringField("database", event.getDatabaseName()); +} else { +jsonGenerator.writeNullField("database"); +} +if (event.getTableName() != null) { +jsonGenerator.writeStringField("table_name", event.getTableName()); +} else { +jsonGenerator.writeNullField("table_name"); +} +if (event.getTableId() != null) { +jsonGenerator.writeNumberField("table_id", event.getTableId()); +} else { +jsonGenerator.writeNullField("table_id"); +} +} + +// Default implementation for binlog events +@Override +public long writeEvent(ProcessSession session, T eventInfo, long currentSequenceId) { +FlowFile flowFile = session.create(); +flowFile = session.write(flowFile, (outputStream) -> { +super.startJson(outputStream, eventInfo); +super.writeJson(eventInfo); --- End diff -- Isn't the super.writeJson(event) encapsulaed by AbstractBinlogEventWriter.writeEvent(...) when it calls writeJson(T)? > Implement a GetChangeDataCapture processor > -- > > Key: NIFI-3413 > URL: https://issues.apache.org/jira/browse/NIFI-3413 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess > > Database systems such as MySQL, Oracle, and SQL Server allow access to their > transactional logs and such, in order for external clients to have a "change > data capture" (CDC) capability. I propose a GetChangeDataCapture processor to > enable this in NiFi. > The processor would be configured with a DBCPConnectionPool controller > service, as well as a Database Type property (similar to the one in > QueryDatabaseTable) for database-specific handling. Additional properties > might include the CDC table name, etc. Additional database-specific > properties could be handled using dynamic properties (and the documentation > should reflect this). > The processor would accept no incoming connections (it is a "Get" or source > processor), would be intended to run on the primary node only as a single > threaded processor, and would generate a flow file for each operation > (INSERT, UPDATE, DELETE, e,g,) in one or some number of formats (JSON, e.g.). > The flow files would be transferred in time order (to enable a replication > solution, for example), perhaps with some auto-incrementing attribute to also > indicate order if need be. -- This message was sent by Atlass
[GitHub] nifi pull request #1618: NIFI-3413: Add GetChangeDataCaptureMySQL processor
Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi/pull/1618#discussion_r107914548 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/impl/mysql/event/io/DeleteRowsWriter.java --- @@ -0,0 +1,90 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard.db.impl.mysql.event.io; + +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processors.standard.GetChangeDataCaptureMySQL; +import org.apache.nifi.processors.standard.db.event.ColumnDefinition; +import org.apache.nifi.processors.standard.db.impl.mysql.event.DeleteRowsEventInfo; + +import java.io.IOException; +import java.io.Serializable; +import java.util.BitSet; +import java.util.concurrent.atomic.AtomicLong; + +import static org.apache.nifi.processors.standard.db.impl.mysql.MySQLCDCUtils.getWritableObject; + +/** + * A writer class to output MySQL binlog "delete rows" events to flow file(s). + */ +public class DeleteRowsWriter extends AbstractBinlogTableEventWriter { + +/** + * Creates and transfers a new flow file whose contents are the JSON-serialized value of the specified event, and the sequence ID attribute set + * + * @param session A reference to a ProcessSession from which the flow file(s) will be created and transferred + * @param eventInfo An event whose value will become the contents of the flow file + * @return The next available CDC sequence ID for use by the CDC processor + */ +public long writeEvent(final ProcessSession session, final DeleteRowsEventInfo eventInfo, final long currentSequenceId) { +final AtomicLong seqId = new AtomicLong(currentSequenceId); --- End diff -- Why use atomic long? Seems to me that CAS is unnecessary unless I'm missing something. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3413) Implement a GetChangeDataCapture processor
[ https://issues.apache.org/jira/browse/NIFI-3413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941191#comment-15941191 ] ASF GitHub Bot commented on NIFI-3413: -- Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi/pull/1618#discussion_r107914548 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/impl/mysql/event/io/DeleteRowsWriter.java --- @@ -0,0 +1,90 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard.db.impl.mysql.event.io; + +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processors.standard.GetChangeDataCaptureMySQL; +import org.apache.nifi.processors.standard.db.event.ColumnDefinition; +import org.apache.nifi.processors.standard.db.impl.mysql.event.DeleteRowsEventInfo; + +import java.io.IOException; +import java.io.Serializable; +import java.util.BitSet; +import java.util.concurrent.atomic.AtomicLong; + +import static org.apache.nifi.processors.standard.db.impl.mysql.MySQLCDCUtils.getWritableObject; + +/** + * A writer class to output MySQL binlog "delete rows" events to flow file(s). + */ +public class DeleteRowsWriter extends AbstractBinlogTableEventWriter { + +/** + * Creates and transfers a new flow file whose contents are the JSON-serialized value of the specified event, and the sequence ID attribute set + * + * @param session A reference to a ProcessSession from which the flow file(s) will be created and transferred + * @param eventInfo An event whose value will become the contents of the flow file + * @return The next available CDC sequence ID for use by the CDC processor + */ +public long writeEvent(final ProcessSession session, final DeleteRowsEventInfo eventInfo, final long currentSequenceId) { +final AtomicLong seqId = new AtomicLong(currentSequenceId); --- End diff -- Why use atomic long? Seems to me that CAS is unnecessary unless I'm missing something. > Implement a GetChangeDataCapture processor > -- > > Key: NIFI-3413 > URL: https://issues.apache.org/jira/browse/NIFI-3413 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess > > Database systems such as MySQL, Oracle, and SQL Server allow access to their > transactional logs and such, in order for external clients to have a "change > data capture" (CDC) capability. I propose a GetChangeDataCapture processor to > enable this in NiFi. > The processor would be configured with a DBCPConnectionPool controller > service, as well as a Database Type property (similar to the one in > QueryDatabaseTable) for database-specific handling. Additional properties > might include the CDC table name, etc. Additional database-specific > properties could be handled using dynamic properties (and the documentation > should reflect this). > The processor would accept no incoming connections (it is a "Get" or source > processor), would be intended to run on the primary node only as a single > threaded processor, and would generate a flow file for each operation > (INSERT, UPDATE, DELETE, e,g,) in one or some number of formats (JSON, e.g.). > The flow files would be transferred in time order (to enable a replication > solution, for example), perhaps with some auto-incrementing attribute to also > indicate order if need be. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi pull request #1618: NIFI-3413: Add GetChangeDataCaptureMySQL processor
Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi/pull/1618#discussion_r107920928 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetChangeDataCaptureMySQL.java --- @@ -0,0 +1,879 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard; + +import com.github.shyiko.mysql.binlog.BinaryLogClient; +import com.github.shyiko.mysql.binlog.event.Event; +import com.github.shyiko.mysql.binlog.event.EventHeaderV4; +import com.github.shyiko.mysql.binlog.event.EventType; +import com.github.shyiko.mysql.binlog.event.QueryEventData; +import com.github.shyiko.mysql.binlog.event.RotateEventData; +import com.github.shyiko.mysql.binlog.event.TableMapEventData; +import org.apache.nifi.annotation.behavior.DynamicProperties; +import org.apache.nifi.annotation.behavior.DynamicProperty; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.Stateful; +import org.apache.nifi.annotation.behavior.TriggerSerially; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.annotation.lifecycle.OnStopped; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.PropertyValue; +import org.apache.nifi.components.state.Scope; +import org.apache.nifi.components.state.StateManager; +import org.apache.nifi.components.state.StateMap; +import org.apache.nifi.distributed.cache.client.Deserializer; +import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient; +import org.apache.nifi.distributed.cache.client.Serializer; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.processor.AbstractSessionFactoryProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessSessionFactory; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.processors.standard.db.CDCException; +import org.apache.nifi.processors.standard.db.event.ColumnDefinition; +import org.apache.nifi.processors.standard.db.event.RowEventException; +import org.apache.nifi.processors.standard.db.event.TableInfo; +import org.apache.nifi.processors.standard.db.event.TableInfoCacheKey; +import org.apache.nifi.processors.standard.db.event.io.EventWriter; +import org.apache.nifi.processors.standard.db.impl.mysql.event.BeginTransactionEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.RawBinlogEvent; +import org.apache.nifi.processors.standard.db.impl.mysql.BinlogEventListener; +import org.apache.nifi.processors.standard.db.impl.mysql.event.BinlogEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.CommitTransactionEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.DeleteRowsEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.SchemaChangeEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.UpdateRowsEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.InsertRowsEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.io.BeginTransactionEventWriter; +import org.apache.nifi.processors.standard.db.impl.mysql.event.io.CommitTransactionEventWriter; +import org.apache.nifi.processors.standard.db.impl.mysql.event.io.DeleteRowsWriter; +import org.apache.nifi.processors.standard.db.impl.mysql.event.io.InsertRowsWriter; +import
[GitHub] nifi pull request #1618: NIFI-3413: Add GetChangeDataCaptureMySQL processor
Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi/pull/1618#discussion_r107917776 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/impl/mysql/event/io/InsertRowsWriter.java --- @@ -0,0 +1,90 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard.db.impl.mysql.event.io; + +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processors.standard.GetChangeDataCaptureMySQL; +import org.apache.nifi.processors.standard.db.event.ColumnDefinition; +import org.apache.nifi.processors.standard.db.impl.mysql.event.InsertRowsEventInfo; + +import java.io.IOException; +import java.io.Serializable; +import java.util.BitSet; +import java.util.concurrent.atomic.AtomicLong; + +import static org.apache.nifi.processors.standard.db.impl.mysql.MySQLCDCUtils.getWritableObject; + +/** + * A writer class to output MySQL binlog "write rows" (aka INSERT) events to flow file(s). + */ +public class InsertRowsWriter extends AbstractBinlogTableEventWriter { + +/** + * Creates and transfers a new flow file whose contents are the JSON-serialized value of the specified event, and the sequence ID attribute set + * + * @param session A reference to a ProcessSession from which the flow file(s) will be created and transferred + * @param eventInfo An event whose value will become the contents of the flow file + * @return The next available CDC sequence ID for use by the CDC processor + */ +public long writeEvent(final ProcessSession session, final InsertRowsEventInfo eventInfo, final long currentSequenceId) { +final AtomicLong seqId = new AtomicLong(currentSequenceId); +for (Serializable[] row : eventInfo.getRows()) { + +FlowFile flowFile = session.create(); +flowFile = session.write(flowFile, outputStream -> { + +super.startJson(outputStream, eventInfo); +super.writeJson(eventInfo); + +final BitSet bitSet = eventInfo.getIncludedColumns(); +writeRow(eventInfo, row, bitSet); + +super.endJson(); +}); + +flowFile = session.putAllAttributes(flowFile, getCommonAttributes(seqId.get(), eventInfo)); +session.transfer(flowFile, GetChangeDataCaptureMySQL.REL_SUCCESS); +seqId.getAndIncrement(); +} +return seqId.get(); +} + +protected void writeRow(InsertRowsEventInfo event, Serializable[] row, BitSet includedColumns) throws IOException { +jsonGenerator.writeArrayFieldStart("columns"); +int i = includedColumns.nextSetBit(0); +while (i != -1) { +jsonGenerator.writeStartObject(); +jsonGenerator.writeNumberField("id", i + 1); +ColumnDefinition columnDefinition = event.getColumnByIndex(i); +Integer columnType = null; +if (columnDefinition != null) { +jsonGenerator.writeStringField("name", columnDefinition.getName()); +columnType = (int) columnDefinition.getType(); --- End diff -- I left a comment earlier about this. Could we avoid the autoboxing and make make type a Long instead of an enumeration? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3413) Implement a GetChangeDataCapture processor
[ https://issues.apache.org/jira/browse/NIFI-3413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941181#comment-15941181 ] ASF GitHub Bot commented on NIFI-3413: -- Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi/pull/1618#discussion_r107921330 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetChangeDataCaptureMySQL.java --- @@ -0,0 +1,879 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard; + +import com.github.shyiko.mysql.binlog.BinaryLogClient; +import com.github.shyiko.mysql.binlog.event.Event; +import com.github.shyiko.mysql.binlog.event.EventHeaderV4; +import com.github.shyiko.mysql.binlog.event.EventType; +import com.github.shyiko.mysql.binlog.event.QueryEventData; +import com.github.shyiko.mysql.binlog.event.RotateEventData; +import com.github.shyiko.mysql.binlog.event.TableMapEventData; +import org.apache.nifi.annotation.behavior.DynamicProperties; +import org.apache.nifi.annotation.behavior.DynamicProperty; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.Stateful; +import org.apache.nifi.annotation.behavior.TriggerSerially; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.annotation.lifecycle.OnStopped; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.PropertyValue; +import org.apache.nifi.components.state.Scope; +import org.apache.nifi.components.state.StateManager; +import org.apache.nifi.components.state.StateMap; +import org.apache.nifi.distributed.cache.client.Deserializer; +import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient; +import org.apache.nifi.distributed.cache.client.Serializer; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.processor.AbstractSessionFactoryProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessSessionFactory; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.processors.standard.db.CDCException; +import org.apache.nifi.processors.standard.db.event.ColumnDefinition; +import org.apache.nifi.processors.standard.db.event.RowEventException; +import org.apache.nifi.processors.standard.db.event.TableInfo; +import org.apache.nifi.processors.standard.db.event.TableInfoCacheKey; +import org.apache.nifi.processors.standard.db.event.io.EventWriter; +import org.apache.nifi.processors.standard.db.impl.mysql.event.BeginTransactionEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.RawBinlogEvent; +import org.apache.nifi.processors.standard.db.impl.mysql.BinlogEventListener; +import org.apache.nifi.processors.standard.db.impl.mysql.event.BinlogEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.CommitTransactionEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.DeleteRowsEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.SchemaChangeEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.UpdateRowsEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.InsertRowsEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.io.BeginTransactionEventWriter; +import org.apache.nifi.processors.st
[jira] [Commented] (NIFI-3413) Implement a GetChangeDataCapture processor
[ https://issues.apache.org/jira/browse/NIFI-3413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941183#comment-15941183 ] ASF GitHub Bot commented on NIFI-3413: -- Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi/pull/1618#discussion_r107910775 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/event/TableInfoCacheKey.java --- @@ -0,0 +1,95 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard.db.event; + +import org.apache.nifi.distributed.cache.client.exception.SerializationException; + +import java.io.IOException; +import java.io.OutputStream; + +import static org.apache.nifi.processors.standard.db.event.TableInfo.DB_TABLE_NAME_DELIMITER; + +/** + * This class represents a key in a cache that contains information (column definitions, e.g.) for a database table + */ +public class TableInfoCacheKey { + +private final String databaseName; +private final String tableName; +private final long tableId; +private final String uuidPrefix; + +public TableInfoCacheKey(String uuidPrefix, String databaseName, String tableName, long tableId) { +this.uuidPrefix = uuidPrefix; +this.databaseName = databaseName; +this.tableName = tableName; +this.tableId = tableId; +} + +@Override +public boolean equals(Object o) { +if (this == o) return true; +if (o == null || getClass() != o.getClass()) return false; + +TableInfoCacheKey that = (TableInfoCacheKey) o; + +if (tableId != that.tableId) return false; +if (databaseName != null ? !databaseName.equals(that.databaseName) : that.databaseName != null) return false; --- End diff -- what if that.databaseName is null? I don't believe java.lang.String checks null. Could use EqualsBuilder from commons-lang, which does the null check since commons-lang3 is in the pom. > Implement a GetChangeDataCapture processor > -- > > Key: NIFI-3413 > URL: https://issues.apache.org/jira/browse/NIFI-3413 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess > > Database systems such as MySQL, Oracle, and SQL Server allow access to their > transactional logs and such, in order for external clients to have a "change > data capture" (CDC) capability. I propose a GetChangeDataCapture processor to > enable this in NiFi. > The processor would be configured with a DBCPConnectionPool controller > service, as well as a Database Type property (similar to the one in > QueryDatabaseTable) for database-specific handling. Additional properties > might include the CDC table name, etc. Additional database-specific > properties could be handled using dynamic properties (and the documentation > should reflect this). > The processor would accept no incoming connections (it is a "Get" or source > processor), would be intended to run on the primary node only as a single > threaded processor, and would generate a flow file for each operation > (INSERT, UPDATE, DELETE, e,g,) in one or some number of formats (JSON, e.g.). > The flow files would be transferred in time order (to enable a replication > solution, for example), perhaps with some auto-incrementing attribute to also > indicate order if need be. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3413) Implement a GetChangeDataCapture processor
[ https://issues.apache.org/jira/browse/NIFI-3413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941190#comment-15941190 ] ASF GitHub Bot commented on NIFI-3413: -- Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi/pull/1618#discussion_r107920928 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetChangeDataCaptureMySQL.java --- @@ -0,0 +1,879 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard; + +import com.github.shyiko.mysql.binlog.BinaryLogClient; +import com.github.shyiko.mysql.binlog.event.Event; +import com.github.shyiko.mysql.binlog.event.EventHeaderV4; +import com.github.shyiko.mysql.binlog.event.EventType; +import com.github.shyiko.mysql.binlog.event.QueryEventData; +import com.github.shyiko.mysql.binlog.event.RotateEventData; +import com.github.shyiko.mysql.binlog.event.TableMapEventData; +import org.apache.nifi.annotation.behavior.DynamicProperties; +import org.apache.nifi.annotation.behavior.DynamicProperty; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.Stateful; +import org.apache.nifi.annotation.behavior.TriggerSerially; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.annotation.lifecycle.OnStopped; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.PropertyValue; +import org.apache.nifi.components.state.Scope; +import org.apache.nifi.components.state.StateManager; +import org.apache.nifi.components.state.StateMap; +import org.apache.nifi.distributed.cache.client.Deserializer; +import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient; +import org.apache.nifi.distributed.cache.client.Serializer; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.processor.AbstractSessionFactoryProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessSessionFactory; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.processors.standard.db.CDCException; +import org.apache.nifi.processors.standard.db.event.ColumnDefinition; +import org.apache.nifi.processors.standard.db.event.RowEventException; +import org.apache.nifi.processors.standard.db.event.TableInfo; +import org.apache.nifi.processors.standard.db.event.TableInfoCacheKey; +import org.apache.nifi.processors.standard.db.event.io.EventWriter; +import org.apache.nifi.processors.standard.db.impl.mysql.event.BeginTransactionEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.RawBinlogEvent; +import org.apache.nifi.processors.standard.db.impl.mysql.BinlogEventListener; +import org.apache.nifi.processors.standard.db.impl.mysql.event.BinlogEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.CommitTransactionEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.DeleteRowsEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.SchemaChangeEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.UpdateRowsEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.InsertRowsEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.io.BeginTransactionEventWriter; +import org.apache.nifi.processors.st
[GitHub] nifi pull request #1618: NIFI-3413: Add GetChangeDataCaptureMySQL processor
Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi/pull/1618#discussion_r107921330 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetChangeDataCaptureMySQL.java --- @@ -0,0 +1,879 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard; + +import com.github.shyiko.mysql.binlog.BinaryLogClient; +import com.github.shyiko.mysql.binlog.event.Event; +import com.github.shyiko.mysql.binlog.event.EventHeaderV4; +import com.github.shyiko.mysql.binlog.event.EventType; +import com.github.shyiko.mysql.binlog.event.QueryEventData; +import com.github.shyiko.mysql.binlog.event.RotateEventData; +import com.github.shyiko.mysql.binlog.event.TableMapEventData; +import org.apache.nifi.annotation.behavior.DynamicProperties; +import org.apache.nifi.annotation.behavior.DynamicProperty; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.Stateful; +import org.apache.nifi.annotation.behavior.TriggerSerially; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.annotation.lifecycle.OnStopped; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.PropertyValue; +import org.apache.nifi.components.state.Scope; +import org.apache.nifi.components.state.StateManager; +import org.apache.nifi.components.state.StateMap; +import org.apache.nifi.distributed.cache.client.Deserializer; +import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient; +import org.apache.nifi.distributed.cache.client.Serializer; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.processor.AbstractSessionFactoryProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessSessionFactory; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.processors.standard.db.CDCException; +import org.apache.nifi.processors.standard.db.event.ColumnDefinition; +import org.apache.nifi.processors.standard.db.event.RowEventException; +import org.apache.nifi.processors.standard.db.event.TableInfo; +import org.apache.nifi.processors.standard.db.event.TableInfoCacheKey; +import org.apache.nifi.processors.standard.db.event.io.EventWriter; +import org.apache.nifi.processors.standard.db.impl.mysql.event.BeginTransactionEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.RawBinlogEvent; +import org.apache.nifi.processors.standard.db.impl.mysql.BinlogEventListener; +import org.apache.nifi.processors.standard.db.impl.mysql.event.BinlogEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.CommitTransactionEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.DeleteRowsEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.SchemaChangeEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.UpdateRowsEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.InsertRowsEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.io.BeginTransactionEventWriter; +import org.apache.nifi.processors.standard.db.impl.mysql.event.io.CommitTransactionEventWriter; +import org.apache.nifi.processors.standard.db.impl.mysql.event.io.DeleteRowsWriter; +import org.apache.nifi.processors.standard.db.impl.mysql.event.io.InsertRowsWriter; +import
[GitHub] nifi pull request #1618: NIFI-3413: Add GetChangeDataCaptureMySQL processor
Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi/pull/1618#discussion_r107909853 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/event/TableInfo.java --- @@ -0,0 +1,146 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard.db.event; + +import org.apache.nifi.distributed.cache.client.exception.DeserializationException; +import org.apache.nifi.distributed.cache.client.exception.SerializationException; + +import java.io.IOException; +import java.io.OutputStream; +import java.util.ArrayList; +import java.util.List; +import java.util.stream.Collectors; + +/** + * A POJO for holding table information related to update events. + */ +public class TableInfo { + +final static String DB_TABLE_NAME_DELIMITER = "@!@"; + +private String databaseName; +private String tableName; +private Long tableId; +private List columns; + +public TableInfo(String databaseName, String tableName, Long tableId, List columns) { +this.databaseName = databaseName; +this.tableName = tableName; +this.tableId = tableId; +this.columns = columns; +} + +public String getDatabaseName() { +return databaseName; +} + +public String getTableName() { +return tableName; +} + +public void setTableName(String tableName) { +this.tableName = tableName; +} + +public Long getTableId() { +return tableId; +} + +public List getColumns() { +return columns; +} + +public void setColumns(List columns) { +this.columns = columns; +} + +@Override +public boolean equals(Object o) { +if (this == o) return true; +if (o == null || getClass() != o.getClass()) return false; + +TableInfo tableInfo = (TableInfo) o; + +if (!databaseName.equals(tableInfo.databaseName)) return false; +if (!tableName.equals(tableInfo.tableName)) return false; +if (!tableId.equals(tableInfo.tableId)) return false; +return columns != null ? columns.equals(tableInfo.columns) : tableInfo.columns == null; +} + +@Override +public int hashCode() { +int result = databaseName.hashCode(); +result = 31 * result + tableName.hashCode(); +result = 31 * result + tableId.hashCode(); +result = 31 * result + (columns != null ? columns.hashCode() : 0); +return result; +} + +public static class Serializer implements org.apache.nifi.distributed.cache.client.Serializer { + +@Override +public void serialize(TableInfo value, OutputStream output) throws SerializationException, IOException { +StringBuilder sb = new StringBuilder(value.getDatabaseName()); +sb.append(DB_TABLE_NAME_DELIMITER); +sb.append(value.getTableName()); +sb.append(DB_TABLE_NAME_DELIMITER); +sb.append(value.getTableId()); +List columnDefinitions = value.getColumns(); +if (columnDefinitions != null && !columnDefinitions.isEmpty()) { +sb.append(DB_TABLE_NAME_DELIMITER); +sb.append(columnDefinitions.stream().map((col) -> col.getName() + DB_TABLE_NAME_DELIMITER + col.getType()).collect(Collectors.joining(DB_TABLE_NAME_DELIMITER))); +} +output.write(sb.toString().getBytes()); +} +} + +public static class Deserializer implements org.apache.nifi.distributed.cache.client.Deserializer { + +@Override +public TableInfo deserialize(byte[] input) throws DeserializationException, IOException { +// Don
[jira] [Commented] (NIFI-3413) Implement a GetChangeDataCapture processor
[ https://issues.apache.org/jira/browse/NIFI-3413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941185#comment-15941185 ] ASF GitHub Bot commented on NIFI-3413: -- Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi/pull/1618#discussion_r107922204 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetChangeDataCaptureMySQL.java --- @@ -0,0 +1,879 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard; + +import com.github.shyiko.mysql.binlog.BinaryLogClient; +import com.github.shyiko.mysql.binlog.event.Event; +import com.github.shyiko.mysql.binlog.event.EventHeaderV4; +import com.github.shyiko.mysql.binlog.event.EventType; +import com.github.shyiko.mysql.binlog.event.QueryEventData; +import com.github.shyiko.mysql.binlog.event.RotateEventData; +import com.github.shyiko.mysql.binlog.event.TableMapEventData; +import org.apache.nifi.annotation.behavior.DynamicProperties; +import org.apache.nifi.annotation.behavior.DynamicProperty; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.Stateful; +import org.apache.nifi.annotation.behavior.TriggerSerially; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.annotation.lifecycle.OnStopped; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.PropertyValue; +import org.apache.nifi.components.state.Scope; +import org.apache.nifi.components.state.StateManager; +import org.apache.nifi.components.state.StateMap; +import org.apache.nifi.distributed.cache.client.Deserializer; +import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient; +import org.apache.nifi.distributed.cache.client.Serializer; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.processor.AbstractSessionFactoryProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessSessionFactory; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.processors.standard.db.CDCException; +import org.apache.nifi.processors.standard.db.event.ColumnDefinition; +import org.apache.nifi.processors.standard.db.event.RowEventException; +import org.apache.nifi.processors.standard.db.event.TableInfo; +import org.apache.nifi.processors.standard.db.event.TableInfoCacheKey; +import org.apache.nifi.processors.standard.db.event.io.EventWriter; +import org.apache.nifi.processors.standard.db.impl.mysql.event.BeginTransactionEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.RawBinlogEvent; +import org.apache.nifi.processors.standard.db.impl.mysql.BinlogEventListener; +import org.apache.nifi.processors.standard.db.impl.mysql.event.BinlogEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.CommitTransactionEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.DeleteRowsEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.SchemaChangeEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.UpdateRowsEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.InsertRowsEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.io.BeginTransactionEventWriter; +import org.apache.nifi.processors.st
[GitHub] nifi pull request #1618: NIFI-3413: Add GetChangeDataCaptureMySQL processor
Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi/pull/1618#discussion_r107914632 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/impl/mysql/event/io/DeleteRowsWriter.java --- @@ -0,0 +1,90 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard.db.impl.mysql.event.io; + +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processors.standard.GetChangeDataCaptureMySQL; +import org.apache.nifi.processors.standard.db.event.ColumnDefinition; +import org.apache.nifi.processors.standard.db.impl.mysql.event.DeleteRowsEventInfo; + +import java.io.IOException; +import java.io.Serializable; +import java.util.BitSet; +import java.util.concurrent.atomic.AtomicLong; + +import static org.apache.nifi.processors.standard.db.impl.mysql.MySQLCDCUtils.getWritableObject; + +/** + * A writer class to output MySQL binlog "delete rows" events to flow file(s). + */ +public class DeleteRowsWriter extends AbstractBinlogTableEventWriter { + +/** + * Creates and transfers a new flow file whose contents are the JSON-serialized value of the specified event, and the sequence ID attribute set + * + * @param session A reference to a ProcessSession from which the flow file(s) will be created and transferred + * @param eventInfo An event whose value will become the contents of the flow file + * @return The next available CDC sequence ID for use by the CDC processor + */ +public long writeEvent(final ProcessSession session, final DeleteRowsEventInfo eventInfo, final long currentSequenceId) { +final AtomicLong seqId = new AtomicLong(currentSequenceId); --- End diff -- The correct answer is that I'm usually missing something... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1618: NIFI-3413: Add GetChangeDataCaptureMySQL processor
Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi/pull/1618#discussion_r107908219 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/event/ColumnDefinition.java --- @@ -0,0 +1,70 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard.db.event; + +/** + * A class that specifies a definition for a relational table column, including type, name, etc. + */ +public class ColumnDefinition { + +private byte type; +private String name = ""; + +public ColumnDefinition(byte type) { --- End diff -- Is there a way to avoid a byte and use an enumeration? I may answer this myself as I continue the review... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1618: NIFI-3413: Add GetChangeDataCaptureMySQL processor
Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi/pull/1618#discussion_r107922204 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetChangeDataCaptureMySQL.java --- @@ -0,0 +1,879 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard; + +import com.github.shyiko.mysql.binlog.BinaryLogClient; +import com.github.shyiko.mysql.binlog.event.Event; +import com.github.shyiko.mysql.binlog.event.EventHeaderV4; +import com.github.shyiko.mysql.binlog.event.EventType; +import com.github.shyiko.mysql.binlog.event.QueryEventData; +import com.github.shyiko.mysql.binlog.event.RotateEventData; +import com.github.shyiko.mysql.binlog.event.TableMapEventData; +import org.apache.nifi.annotation.behavior.DynamicProperties; +import org.apache.nifi.annotation.behavior.DynamicProperty; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.Stateful; +import org.apache.nifi.annotation.behavior.TriggerSerially; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.annotation.lifecycle.OnStopped; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.PropertyValue; +import org.apache.nifi.components.state.Scope; +import org.apache.nifi.components.state.StateManager; +import org.apache.nifi.components.state.StateMap; +import org.apache.nifi.distributed.cache.client.Deserializer; +import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient; +import org.apache.nifi.distributed.cache.client.Serializer; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.processor.AbstractSessionFactoryProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessSessionFactory; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.processors.standard.db.CDCException; +import org.apache.nifi.processors.standard.db.event.ColumnDefinition; +import org.apache.nifi.processors.standard.db.event.RowEventException; +import org.apache.nifi.processors.standard.db.event.TableInfo; +import org.apache.nifi.processors.standard.db.event.TableInfoCacheKey; +import org.apache.nifi.processors.standard.db.event.io.EventWriter; +import org.apache.nifi.processors.standard.db.impl.mysql.event.BeginTransactionEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.RawBinlogEvent; +import org.apache.nifi.processors.standard.db.impl.mysql.BinlogEventListener; +import org.apache.nifi.processors.standard.db.impl.mysql.event.BinlogEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.CommitTransactionEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.DeleteRowsEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.SchemaChangeEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.UpdateRowsEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.InsertRowsEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.io.BeginTransactionEventWriter; +import org.apache.nifi.processors.standard.db.impl.mysql.event.io.CommitTransactionEventWriter; +import org.apache.nifi.processors.standard.db.impl.mysql.event.io.DeleteRowsWriter; +import org.apache.nifi.processors.standard.db.impl.mysql.event.io.InsertRowsWriter; +import
[GitHub] nifi pull request #1618: NIFI-3413: Add GetChangeDataCaptureMySQL processor
Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi/pull/1618#discussion_r107921926 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetChangeDataCaptureMySQL.java --- @@ -0,0 +1,879 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard; + +import com.github.shyiko.mysql.binlog.BinaryLogClient; +import com.github.shyiko.mysql.binlog.event.Event; +import com.github.shyiko.mysql.binlog.event.EventHeaderV4; +import com.github.shyiko.mysql.binlog.event.EventType; +import com.github.shyiko.mysql.binlog.event.QueryEventData; +import com.github.shyiko.mysql.binlog.event.RotateEventData; +import com.github.shyiko.mysql.binlog.event.TableMapEventData; +import org.apache.nifi.annotation.behavior.DynamicProperties; +import org.apache.nifi.annotation.behavior.DynamicProperty; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.Stateful; +import org.apache.nifi.annotation.behavior.TriggerSerially; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.annotation.lifecycle.OnStopped; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.PropertyValue; +import org.apache.nifi.components.state.Scope; +import org.apache.nifi.components.state.StateManager; +import org.apache.nifi.components.state.StateMap; +import org.apache.nifi.distributed.cache.client.Deserializer; +import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient; +import org.apache.nifi.distributed.cache.client.Serializer; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.processor.AbstractSessionFactoryProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessSessionFactory; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.processors.standard.db.CDCException; +import org.apache.nifi.processors.standard.db.event.ColumnDefinition; +import org.apache.nifi.processors.standard.db.event.RowEventException; +import org.apache.nifi.processors.standard.db.event.TableInfo; +import org.apache.nifi.processors.standard.db.event.TableInfoCacheKey; +import org.apache.nifi.processors.standard.db.event.io.EventWriter; +import org.apache.nifi.processors.standard.db.impl.mysql.event.BeginTransactionEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.RawBinlogEvent; +import org.apache.nifi.processors.standard.db.impl.mysql.BinlogEventListener; +import org.apache.nifi.processors.standard.db.impl.mysql.event.BinlogEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.CommitTransactionEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.DeleteRowsEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.SchemaChangeEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.UpdateRowsEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.InsertRowsEventInfo; +import org.apache.nifi.processors.standard.db.impl.mysql.event.io.BeginTransactionEventWriter; +import org.apache.nifi.processors.standard.db.impl.mysql.event.io.CommitTransactionEventWriter; +import org.apache.nifi.processors.standard.db.impl.mysql.event.io.DeleteRowsWriter; +import org.apache.nifi.processors.standard.db.impl.mysql.event.io.InsertRowsWriter; +import
[GitHub] nifi pull request #1618: NIFI-3413: Add GetChangeDataCaptureMySQL processor
Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi/pull/1618#discussion_r107912643 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/impl/mysql/event/io/AbstractBinlogTableEventWriter.java --- @@ -0,0 +1,65 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard.db.impl.mysql.event.io; + +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processors.standard.GetChangeDataCaptureMySQL; +import org.apache.nifi.processors.standard.db.impl.mysql.event.BinlogTableEventInfo; + +import java.io.IOException; + +/** + * An abstract base class for writing MYSQL binlog events into flow file(s), e.g. + */ +public abstract class AbstractBinlogTableEventWriter extends AbstractBinlogEventWriter { + +protected void writeJson(T event) throws IOException { +super.writeJson(event); +if (event.getDatabaseName() != null) { +jsonGenerator.writeStringField("database", event.getDatabaseName()); +} else { +jsonGenerator.writeNullField("database"); +} +if (event.getTableName() != null) { +jsonGenerator.writeStringField("table_name", event.getTableName()); +} else { +jsonGenerator.writeNullField("table_name"); +} +if (event.getTableId() != null) { +jsonGenerator.writeNumberField("table_id", event.getTableId()); +} else { +jsonGenerator.writeNullField("table_id"); +} +} + +// Default implementation for binlog events +@Override +public long writeEvent(ProcessSession session, T eventInfo, long currentSequenceId) { +FlowFile flowFile = session.create(); +flowFile = session.write(flowFile, (outputStream) -> { +super.startJson(outputStream, eventInfo); +super.writeJson(eventInfo); --- End diff -- Isn't the super.writeJson(event) encapsulaed by AbstractBinlogEventWriter.writeEvent(...) when it calls writeJson(T)? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1618: NIFI-3413: Add GetChangeDataCaptureMySQL processor
Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi/pull/1618#discussion_r107910775 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/event/TableInfoCacheKey.java --- @@ -0,0 +1,95 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard.db.event; + +import org.apache.nifi.distributed.cache.client.exception.SerializationException; + +import java.io.IOException; +import java.io.OutputStream; + +import static org.apache.nifi.processors.standard.db.event.TableInfo.DB_TABLE_NAME_DELIMITER; + +/** + * This class represents a key in a cache that contains information (column definitions, e.g.) for a database table + */ +public class TableInfoCacheKey { + +private final String databaseName; +private final String tableName; +private final long tableId; +private final String uuidPrefix; + +public TableInfoCacheKey(String uuidPrefix, String databaseName, String tableName, long tableId) { +this.uuidPrefix = uuidPrefix; +this.databaseName = databaseName; +this.tableName = tableName; +this.tableId = tableId; +} + +@Override +public boolean equals(Object o) { +if (this == o) return true; +if (o == null || getClass() != o.getClass()) return false; + +TableInfoCacheKey that = (TableInfoCacheKey) o; + +if (tableId != that.tableId) return false; +if (databaseName != null ? !databaseName.equals(that.databaseName) : that.databaseName != null) return false; --- End diff -- what if that.databaseName is null? I don't believe java.lang.String checks null. Could use EqualsBuilder from commons-lang, which does the null check since commons-lang3 is in the pom. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3596) GenerateTableFetch - Add attributes to generated flow files to ease SQL query overwrite
[ https://issues.apache.org/jira/browse/NIFI-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941144#comment-15941144 ] ASF GitHub Bot commented on NIFI-3596: -- Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/1596 Thanks for the review @mattyb149. I updated a test to check the selected column names. > GenerateTableFetch - Add attributes to generated flow files to ease SQL query > overwrite > --- > > Key: NIFI-3596 > URL: https://issues.apache.org/jira/browse/NIFI-3596 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Trivial > Fix For: 1.2.0 > > > The GenerateTableFetch processor will generate a SQL query based on the > provided parameters but, if the specific DB adapter is not available, it > might be necessary to overwrite the SQL query in the ExecuteSQL processor. To > do that it would be nice to have each part of the query as attributes to take > advantage of expression language. > Current workaround (when using GenerateTableFetch against a DB2 database for > example) is to have an intermediary ExtractText processor and have a regex > extracting each part of the generated query. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi issue #1596: NIFI-3596 - added attributes to GenerateTableFetch process...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/1596 Thanks for the review @mattyb149. I updated a test to check the selected column names. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Comment Edited] (NIFI-3304) In the Administration Guide, there is an incorrect JAVA 8 parameter for bootstrap.conf file
[ https://issues.apache.org/jira/browse/NIFI-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941121#comment-15941121 ] Michael Moser edited comment on NIFI-3304 at 3/24/17 9:02 PM: -- According to Oracle's Java 8 documentation \[1\] the defaults are currently -XX:ReservedCodeCacheSize=240m -XX:+UseCodeCacheFlushing Since NiFi 1.x requires Java 8, and these defaults already align with our Admin Guide recommendations, I think we can remove this section of the Admin Guide. Also, these settings have already been removed from the bootstrap.conf. With Java 7, the default ReservedCodeCacheSize was 48m and codecache flushing was buggy \[2\] so our older Admin Guide recommendations made more sense in the NiFi 0.x days. \[1\] - http://docs.oracle.com/javase/8/docs/technotes/tools/windows/java.html \[2\] - https://blogs.oracle.com/poonam/entry/why_do_i_get_message was (Author: mosermw): According to Oracle's Java 8 documentation \[1\] the defaults are currently -XX:ReservedCodeCacheSize=240m -XX:+UseCodeCacheFlushing These seem to already align with our Admin Guide recommendations, so I think we can remove this section of the Admin Guide. With Java 7, the default ReservedCodeCacheSize was 48m and codecache flushing was buggy \[2\] so our older Admin Guide recommendations made more sense. \[1\] - http://docs.oracle.com/javase/8/docs/technotes/tools/windows/java.html \[2\] - https://blogs.oracle.com/poonam/entry/why_do_i_get_message > In the Administration Guide, there is an incorrect JAVA 8 parameter for > bootstrap.conf file > --- > > Key: NIFI-3304 > URL: https://issues.apache.org/jira/browse/NIFI-3304 > Project: Apache NiFi > Issue Type: Bug > Components: Documentation & Website >Reporter: David A. Wynne >Assignee: Michael Moser >Priority: Minor > > In the Administration guide, section Bootstrap properties, there is a > paragraph about Java 8 handling of codecache, the listed properties are > accurate for Java version 7, but for Java version 8, one has changed > This is correct for Java version 7: > java.arg.8=-XX:CodeCacheFlushingMinimumFreeSpace=10m > But for Java version 8, it should be > java.arg.8=-XX:CodeCacheMinimumFreeSpace=10m -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3304) In the Administration Guide, there is an incorrect JAVA 8 parameter for bootstrap.conf file
[ https://issues.apache.org/jira/browse/NIFI-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941121#comment-15941121 ] Michael Moser commented on NIFI-3304: - According to Oracle's Java 8 documentation \[1\] the defaults are currently -XX:ReservedCodeCacheSize=240m -XX:+UseCodeCacheFlushing These seem to already align with our Admin Guide recommendations, so I think we can remove this section of the Admin Guide. With Java 7, the default ReservedCodeCacheSize was 48m and codecache flushing was buggy \[2\] so our older Admin Guide recommendations made more sense. \[1\] - http://docs.oracle.com/javase/8/docs/technotes/tools/windows/java.html \[2\] - https://blogs.oracle.com/poonam/entry/why_do_i_get_message > In the Administration Guide, there is an incorrect JAVA 8 parameter for > bootstrap.conf file > --- > > Key: NIFI-3304 > URL: https://issues.apache.org/jira/browse/NIFI-3304 > Project: Apache NiFi > Issue Type: Bug > Components: Documentation & Website >Reporter: David A. Wynne >Assignee: Michael Moser >Priority: Minor > > In the Administration guide, section Bootstrap properties, there is a > paragraph about Java 8 handling of codecache, the listed properties are > accurate for Java version 7, but for Java version 8, one has changed > This is correct for Java version 7: > java.arg.8=-XX:CodeCacheFlushingMinimumFreeSpace=10m > But for Java version 8, it should be > java.arg.8=-XX:CodeCacheMinimumFreeSpace=10m -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (NIFI-3304) In the Administration Guide, there is an incorrect JAVA 8 parameter for bootstrap.conf file
[ https://issues.apache.org/jira/browse/NIFI-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Moser reassigned NIFI-3304: --- Assignee: Michael Moser > In the Administration Guide, there is an incorrect JAVA 8 parameter for > bootstrap.conf file > --- > > Key: NIFI-3304 > URL: https://issues.apache.org/jira/browse/NIFI-3304 > Project: Apache NiFi > Issue Type: Bug > Components: Documentation & Website >Reporter: David A. Wynne >Assignee: Michael Moser >Priority: Minor > > In the Administration guide, section Bootstrap properties, there is a > paragraph about Java 8 handling of codecache, the listed properties are > accurate for Java version 7, but for Java version 8, one has changed > This is correct for Java version 7: > java.arg.8=-XX:CodeCacheFlushingMinimumFreeSpace=10m > But for Java version 8, it should be > java.arg.8=-XX:CodeCacheMinimumFreeSpace=10m -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3421) Start up failure - java.nio.channels.OverlappingFileLockException: null
[ https://issues.apache.org/jira/browse/NIFI-3421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941086#comment-15941086 ] ASF GitHub Bot commented on NIFI-3421: -- GitHub user mcgilman opened a pull request: https://github.com/apache/nifi/pull/1620 NIFI-3421: Preventing successive attempts to create a FlowController NIFI-3421: - On contextDestroyed, referencing beans created during contextInitialized to prevent successive attempts to create a bean if that bean failed to be created initially. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mcgilman/nifi NIFI-3421 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1620.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1620 commit 2ac22e78381cf0dfda6c2d981857865214a4cc41 Author: Matt Gilman Date: 2017-03-24T20:30:48Z NIFI-3421: - On contextDestroyed, referencing beans created during contextInitialized to prevent successive attempts to create a bean if that bean failed to be created initially. > Start up failure - java.nio.channels.OverlappingFileLockException: null > --- > > Key: NIFI-3421 > URL: https://issues.apache.org/jira/browse/NIFI-3421 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Matt Gilman >Assignee: Matt Gilman >Priority: Critical > Fix For: 1.2.0 > > > If NiFi fails to start up successfully, the act of attempting to shut down > appears to result in a duplicate FlowController instantiation. A symptom of > this is initializing the flow file repo twice which fails with an overlapping > file lock exception. > The actual reason that start up failed is logged further back in the app log. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (NIFI-3421) Start up failure - java.nio.channels.OverlappingFileLockException: null
[ https://issues.apache.org/jira/browse/NIFI-3421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman updated NIFI-3421: -- Status: Patch Available (was: In Progress) > Start up failure - java.nio.channels.OverlappingFileLockException: null > --- > > Key: NIFI-3421 > URL: https://issues.apache.org/jira/browse/NIFI-3421 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Matt Gilman >Assignee: Matt Gilman >Priority: Critical > Fix For: 1.2.0 > > > If NiFi fails to start up successfully, the act of attempting to shut down > appears to result in a duplicate FlowController instantiation. A symptom of > this is initializing the flow file repo twice which fails with an overlapping > file lock exception. > The actual reason that start up failed is logged further back in the app log. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi pull request #1620: NIFI-3421: Preventing successive attempts to create...
GitHub user mcgilman opened a pull request: https://github.com/apache/nifi/pull/1620 NIFI-3421: Preventing successive attempts to create a FlowController NIFI-3421: - On contextDestroyed, referencing beans created during contextInitialized to prevent successive attempts to create a bean if that bean failed to be created initially. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mcgilman/nifi NIFI-3421 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1620.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1620 commit 2ac22e78381cf0dfda6c2d981857865214a4cc41 Author: Matt Gilman Date: 2017-03-24T20:30:48Z NIFI-3421: - On contextDestroyed, referencing beans created during contextInitialized to prevent successive attempts to create a bean if that bean failed to be created initially. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3257) Cluster stability issues during high throughput
[ https://issues.apache.org/jira/browse/NIFI-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941078#comment-15941078 ] Mark Payne commented on NIFI-3257: -- Using the -XX:+PrintGC JVM option and using YourKit to profile garbage collection, along with significant DEBUG logging that I've added, I am seeing that the problem largely is due to excessive GC runs. I'm seeing up to 25% of my JVM time spent performing Garbage Collection. I've marked this ticket as being related to NIFI-3636 and NIFI-3648 because these tickets are intended to address the heavy garbage collection. I've also found that why we use multiple threads to replicate REST API calls, we do not read or merge node responses in parallel. This is done serially after all "Response" objects have been obtained. This is very inefficient and can result in very long request replication times. It could even result in one slow node causing other nodes' responses to timeout meaning that if Node 1 is slow to respond (due to GC or whatever), then the responses from Nodes 4, 5, and 6, for instance, could time out. As a result, nodes 4, 5, and 6 could be kicked out of the cluster as a result of Node 1 being slow. > Cluster stability issues during high throughput > --- > > Key: NIFI-3257 > URL: https://issues.apache.org/jira/browse/NIFI-3257 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.0.0, 1.1.0, 1.1.1, 1.0.1 >Reporter: Jeff Storck > > During high throughput of data in a cluster (135MB/s), nodes experience > frequent disconnects (every few minutes) and role switching (Primary and > Cluster Coordinator). This makes API requests difficult since the requests > can not be replicated to all nodes while reconnecting. The cluster can > recover for a time (as mentioned above, for a few minutes) before going > through another round of disconnects and role switching. > The cluster is able to continue to process data during these connection and > role-switching issues. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (NIFI-3649) Parallelize Consumption of Cluster Replication Responses
Mark Payne created NIFI-3649: Summary: Parallelize Consumption of Cluster Replication Responses Key: NIFI-3649 URL: https://issues.apache.org/jira/browse/NIFI-3649 Project: Apache NiFi Issue Type: Improvement Components: Core Framework Reporter: Mark Payne Assignee: Mark Payne Currently, when we replicate a REST API call to all nodes in the cluster, we wait until we get back a Response object for each node. We then go to the Response Mapper and merge those responses into 1 NodeResponse. This merging is done serially, and this is where we actually do the reading of the response. We need to instead do this merging in parallel. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-1705) AttributesToCSV
[ https://issues.apache.org/jira/browse/NIFI-1705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941068#comment-15941068 ] ASF GitHub Bot commented on NIFI-1705: -- Github user mattyb149 commented on a diff in the pull request: https://github.com/apache/nifi/pull/1589#discussion_r107990719 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AttributesToCSV.java --- @@ -0,0 +1,280 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.processors.standard; + +import org.apache.commons.lang3.StringEscapeUtils; +import org.apache.commons.lang3.StringUtils; +import org.apache.nifi.annotation.behavior.EventDriven; +import org.apache.nifi.annotation.behavior.SideEffectFree; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.flowfile.attributes.CoreAttributes; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessorInitializationContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; + + +import java.io.BufferedOutputStream; +import java.io.OutputStream; +import java.util.Arrays; +import java.util.HashMap; +import java.util.List; +import java.util.ArrayList; +import java.util.Set; +import java.util.HashSet; +import java.util.Map; +import java.util.Collections; +import java.util.stream.Collectors; + +@EventDriven +@SideEffectFree +@SupportsBatching +@Tags({"csv", "attributes", "flowfile"}) +@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED) +@CapabilityDescription("Generates a CSV representation of the input FlowFile Attributes. The resulting CSV " + +"can be written to either a newly generated attribute named 'CSVAttributes' or written to the FlowFile as content. " + +"If the attribute value contains a comma, newline or double quote, then the attribute value will be " + +"escaped with double quotes. Any double quote characters in the attribute value are escaped with " + +"another double quote. If the attribute value does not contain a comma, newline or double quote, then the " + +"attribute value is returned unchanged.") +@WritesAttribute(attribute = "CSVAttributes", description = "CSV representation of Attributes") +public class AttributesToCSV extends AbstractProcessor { + +private static final String OUTPUT_NEW_ATTRIBUTE = "flowfile-attribute"; +private static final String OUTPUT_OVERWRITE_CONTENT = "flowfile-content"; +private static final String OUTPUT_ATTRIBUTE_NAME = "CSVAttributes"; +private static final String OUTPUT_SEPARATOR = ","; +private static final String OUTPUT_MIME_TYPE = "text/csv"; + + +public static final PropertyDescriptor ATTRIBUTES_LIST = new PropertyDescriptor.Builder() +.name("attribute-list") +.displayName("Attribute List") +.description("Comma separated list of attributes to be included in the resulting CSV. If this value " + +"is left empty then all existing Attributes will be included. This list of attributes is " + +"case sensitive. If an attribute specified in the l
[jira] [Created] (NIFI-3648) Address Excessive Garbage Collection
Mark Payne created NIFI-3648: Summary: Address Excessive Garbage Collection Key: NIFI-3648 URL: https://issues.apache.org/jira/browse/NIFI-3648 Project: Apache NiFi Issue Type: Improvement Components: Core Framework, Extensions Reporter: Mark Payne Assignee: Mark Payne We have a lot of places in the codebase where we generate lots of unnecessary garbage - especially byte arrays. We need to clean this up to in order to relieve stress on the garbage collector. Specific points that I've found create unnecessary garbage: Provenance CompressableRecordWriter creates a new BufferedOutputStream for each 'compression block' that it creates. Each one has a 64 KB byte[]. This is very wasteful. We should instead subclass BufferedOutputStream so that we are able to provide a byte[] to use instead of an int that indicates the size. This way, we can just keep re-using the same byte[] that we create for each writer. This saves about 32,000 of these 64 KB byte[] for each writer that we create. And we create more than 1 of these per minute. EvaluateJsonPath uses a BufferedInputStream but it is not necessary, because the underlying library will also buffer data. So we are unnecessarily creating a lot of byte[]'s CompressContent uses Buffered Input AND Output. And uses 64 KB byte[]. And doesn't need them at all, because it reads and writes with its own byte[] buffer via StreamUtils.copy Site-to-site uses CompressionInputStream. This stream creates a new byte[] in the readChunkHeader() method continually. We should instead only create a new byte[] if we need a bigger buffer and otherwise just use an offset & length variable. Right now, SplitText uses TextLineDemarcator. The fill() method increases the size of the internal byte[] by 8 KB each time. When dealing with a large chunk of data, this is VERY expensive on GC because we continually create a byte[] and then discard it to create a new one. Take for example an 800 KB chunk. We would do this 100,000 times. If we instead double the size we would only have to create 8 of these. Other Processors that use Buffered streams unnecessarily: ConvertJSONToSQL ExecuteProcess ExecuteStreamCommand AttributesToJSON EvaluateJsonPath (when writing to content) ExtractGrok JmsConsumer -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi pull request #1589: NIFI-1705 Adding AttributesToCSV processor
Github user mattyb149 commented on a diff in the pull request: https://github.com/apache/nifi/pull/1589#discussion_r107990719 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AttributesToCSV.java --- @@ -0,0 +1,280 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.processors.standard; + +import org.apache.commons.lang3.StringEscapeUtils; +import org.apache.commons.lang3.StringUtils; +import org.apache.nifi.annotation.behavior.EventDriven; +import org.apache.nifi.annotation.behavior.SideEffectFree; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.flowfile.attributes.CoreAttributes; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessorInitializationContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; + + +import java.io.BufferedOutputStream; +import java.io.OutputStream; +import java.util.Arrays; +import java.util.HashMap; +import java.util.List; +import java.util.ArrayList; +import java.util.Set; +import java.util.HashSet; +import java.util.Map; +import java.util.Collections; +import java.util.stream.Collectors; + +@EventDriven +@SideEffectFree +@SupportsBatching +@Tags({"csv", "attributes", "flowfile"}) +@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED) +@CapabilityDescription("Generates a CSV representation of the input FlowFile Attributes. The resulting CSV " + +"can be written to either a newly generated attribute named 'CSVAttributes' or written to the FlowFile as content. " + +"If the attribute value contains a comma, newline or double quote, then the attribute value will be " + +"escaped with double quotes. Any double quote characters in the attribute value are escaped with " + +"another double quote. If the attribute value does not contain a comma, newline or double quote, then the " + +"attribute value is returned unchanged.") +@WritesAttribute(attribute = "CSVAttributes", description = "CSV representation of Attributes") +public class AttributesToCSV extends AbstractProcessor { + +private static final String OUTPUT_NEW_ATTRIBUTE = "flowfile-attribute"; +private static final String OUTPUT_OVERWRITE_CONTENT = "flowfile-content"; +private static final String OUTPUT_ATTRIBUTE_NAME = "CSVAttributes"; +private static final String OUTPUT_SEPARATOR = ","; +private static final String OUTPUT_MIME_TYPE = "text/csv"; + + +public static final PropertyDescriptor ATTRIBUTES_LIST = new PropertyDescriptor.Builder() +.name("attribute-list") +.displayName("Attribute List") +.description("Comma separated list of attributes to be included in the resulting CSV. If this value " + +"is left empty then all existing Attributes will be included. This list of attributes is " + +"case sensitive. If an attribute specified in the list is not found it will be emitted " + +"to the resulting CSV with an empty string or null depending on the 'Null Value' property. " + +"If a core attribute is specified in this list " + +
[jira] [Commented] (NIFI-1705) AttributesToCSV
[ https://issues.apache.org/jira/browse/NIFI-1705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941059#comment-15941059 ] ASF GitHub Bot commented on NIFI-1705: -- Github user mattyb149 commented on a diff in the pull request: https://github.com/apache/nifi/pull/1589#discussion_r107989450 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AttributesToCSV.java --- @@ -0,0 +1,280 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.processors.standard; + +import org.apache.commons.lang3.StringEscapeUtils; +import org.apache.commons.lang3.StringUtils; +import org.apache.nifi.annotation.behavior.EventDriven; +import org.apache.nifi.annotation.behavior.SideEffectFree; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.flowfile.attributes.CoreAttributes; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessorInitializationContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; + + +import java.io.BufferedOutputStream; +import java.io.OutputStream; +import java.util.Arrays; +import java.util.HashMap; +import java.util.List; +import java.util.ArrayList; +import java.util.Set; +import java.util.HashSet; +import java.util.Map; +import java.util.Collections; +import java.util.stream.Collectors; + +@EventDriven +@SideEffectFree +@SupportsBatching +@Tags({"csv", "attributes", "flowfile"}) +@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED) +@CapabilityDescription("Generates a CSV representation of the input FlowFile Attributes. The resulting CSV " + +"can be written to either a newly generated attribute named 'CSVAttributes' or written to the FlowFile as content. " + +"If the attribute value contains a comma, newline or double quote, then the attribute value will be " + +"escaped with double quotes. Any double quote characters in the attribute value are escaped with " + +"another double quote. If the attribute value does not contain a comma, newline or double quote, then the " + +"attribute value is returned unchanged.") +@WritesAttribute(attribute = "CSVAttributes", description = "CSV representation of Attributes") +public class AttributesToCSV extends AbstractProcessor { + +private static final String OUTPUT_NEW_ATTRIBUTE = "flowfile-attribute"; +private static final String OUTPUT_OVERWRITE_CONTENT = "flowfile-content"; +private static final String OUTPUT_ATTRIBUTE_NAME = "CSVAttributes"; +private static final String OUTPUT_SEPARATOR = ","; +private static final String OUTPUT_MIME_TYPE = "text/csv"; + + +public static final PropertyDescriptor ATTRIBUTES_LIST = new PropertyDescriptor.Builder() --- End diff -- Is there any value in supporting a regular expression or (alternatively) supporting Expression Language for the Attribute List property? For the former, I'm thinking of the use case where there might be an arbitrary number of attributes available, often with the same prefix. For the latter (I doubt we'd want to support both), I'm thinking of the case (if it exists) where
[GitHub] nifi pull request #1589: NIFI-1705 Adding AttributesToCSV processor
Github user mattyb149 commented on a diff in the pull request: https://github.com/apache/nifi/pull/1589#discussion_r107989450 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AttributesToCSV.java --- @@ -0,0 +1,280 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.processors.standard; + +import org.apache.commons.lang3.StringEscapeUtils; +import org.apache.commons.lang3.StringUtils; +import org.apache.nifi.annotation.behavior.EventDriven; +import org.apache.nifi.annotation.behavior.SideEffectFree; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.flowfile.attributes.CoreAttributes; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessorInitializationContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; + + +import java.io.BufferedOutputStream; +import java.io.OutputStream; +import java.util.Arrays; +import java.util.HashMap; +import java.util.List; +import java.util.ArrayList; +import java.util.Set; +import java.util.HashSet; +import java.util.Map; +import java.util.Collections; +import java.util.stream.Collectors; + +@EventDriven +@SideEffectFree +@SupportsBatching +@Tags({"csv", "attributes", "flowfile"}) +@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED) +@CapabilityDescription("Generates a CSV representation of the input FlowFile Attributes. The resulting CSV " + +"can be written to either a newly generated attribute named 'CSVAttributes' or written to the FlowFile as content. " + +"If the attribute value contains a comma, newline or double quote, then the attribute value will be " + +"escaped with double quotes. Any double quote characters in the attribute value are escaped with " + +"another double quote. If the attribute value does not contain a comma, newline or double quote, then the " + +"attribute value is returned unchanged.") +@WritesAttribute(attribute = "CSVAttributes", description = "CSV representation of Attributes") +public class AttributesToCSV extends AbstractProcessor { + +private static final String OUTPUT_NEW_ATTRIBUTE = "flowfile-attribute"; +private static final String OUTPUT_OVERWRITE_CONTENT = "flowfile-content"; +private static final String OUTPUT_ATTRIBUTE_NAME = "CSVAttributes"; +private static final String OUTPUT_SEPARATOR = ","; +private static final String OUTPUT_MIME_TYPE = "text/csv"; + + +public static final PropertyDescriptor ATTRIBUTES_LIST = new PropertyDescriptor.Builder() --- End diff -- Is there any value in supporting a regular expression or (alternatively) supporting Expression Language for the Attribute List property? For the former, I'm thinking of the use case where there might be an arbitrary number of attributes available, often with the same prefix. For the latter (I doubt we'd want to support both), I'm thinking of the case (if it exists) where the CSV column names might themselves be in an attribute, where their values might be in the corresponding attribute(s). That second one seems pretty much an edge case, I just wanted to bring these up for discussion. If neither seems particularly u
[jira] [Commented] (NIFI-3596) GenerateTableFetch - Add attributes to generated flow files to ease SQL query overwrite
[ https://issues.apache.org/jira/browse/NIFI-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941011#comment-15941011 ] ASF GitHub Bot commented on NIFI-3596: -- Github user mattyb149 commented on a diff in the pull request: https://github.com/apache/nifi/pull/1596#discussion_r107984019 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestGenerateTableFetch.java --- @@ -207,6 +207,12 @@ public void testAddedRows() throws ClassNotFoundException, SQLException, Initial assertEquals("SELECT * FROM TEST_QUERY_DB_TABLE ORDER BY name OFFSET 4 ROWS FETCH NEXT 2 ROWS ONLY", new String(flowFile.toByteArray())); flowFile = runner.getFlowFilesForRelationship(REL_SUCCESS).get(3); assertEquals("SELECT * FROM TEST_QUERY_DB_TABLE ORDER BY name OFFSET 6 ROWS FETCH NEXT 2 ROWS ONLY", new String(flowFile.toByteArray())); +assertEquals("TEST_QUERY_DB_TABLE", flowFile.getAttribute("generatetablefetch.tableName")); +assertEquals(null, flowFile.getAttribute("generatetablefetch.columnNames")); --- End diff -- All added asserts for columns are checking for null, can you update a test (or add one) that sets the Columns To Return property? Otherwise, looks good to me, will build and run on a real NiFi in the meantime > GenerateTableFetch - Add attributes to generated flow files to ease SQL query > overwrite > --- > > Key: NIFI-3596 > URL: https://issues.apache.org/jira/browse/NIFI-3596 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Trivial > Fix For: 1.2.0 > > > The GenerateTableFetch processor will generate a SQL query based on the > provided parameters but, if the specific DB adapter is not available, it > might be necessary to overwrite the SQL query in the ExecuteSQL processor. To > do that it would be nice to have each part of the query as attributes to take > advantage of expression language. > Current workaround (when using GenerateTableFetch against a DB2 database for > example) is to have an intermediary ExtractText processor and have a regex > extracting each part of the generated query. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi pull request #1596: NIFI-3596 - added attributes to GenerateTableFetch ...
Github user mattyb149 commented on a diff in the pull request: https://github.com/apache/nifi/pull/1596#discussion_r107984019 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestGenerateTableFetch.java --- @@ -207,6 +207,12 @@ public void testAddedRows() throws ClassNotFoundException, SQLException, Initial assertEquals("SELECT * FROM TEST_QUERY_DB_TABLE ORDER BY name OFFSET 4 ROWS FETCH NEXT 2 ROWS ONLY", new String(flowFile.toByteArray())); flowFile = runner.getFlowFilesForRelationship(REL_SUCCESS).get(3); assertEquals("SELECT * FROM TEST_QUERY_DB_TABLE ORDER BY name OFFSET 6 ROWS FETCH NEXT 2 ROWS ONLY", new String(flowFile.toByteArray())); +assertEquals("TEST_QUERY_DB_TABLE", flowFile.getAttribute("generatetablefetch.tableName")); +assertEquals(null, flowFile.getAttribute("generatetablefetch.columnNames")); --- End diff -- All added asserts for columns are checking for null, can you update a test (or add one) that sets the Columns To Return property? Otherwise, looks good to me, will build and run on a real NiFi in the meantime --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3646) Show named relationship icons on processor hover
[ https://issues.apache.org/jira/browse/NIFI-3646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940803#comment-15940803 ] Joseph Witt commented on NIFI-3646: --- [~ncarenza] can you talk more about how you envision this working compared to the current approach? The current approach lets you connect a processor to another processor and then allows you to specify a range of important details about that connection such as the source relationship name, back pressure settings, etc.. This seems like the right model. Can you compare your proposal to that? It looks like you're suggesting we list off one more more possible relationship names then grab that one to create an actual connection. Or, are you simply asking for a way to see the relationship names of all connections leaving a processor whenever you hoever over that processor? This would be not because you want to add more connections but simply so you could more easily understand the existing ones? Keep in mind a connection can represent one or more relationships. > Show named relationship icons on processor hover > > > Key: NIFI-3646 > URL: https://issues.apache.org/jira/browse/NIFI-3646 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Reporter: Nicholas Carenza >Priority: Minor > Labels: connection, ui > Attachments: nifi-connect-rel.png > > > When hovering over a processor, instead of showing a single add-connection > element in the middle, show an array of connections. I attached an image > example. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (NIFI-3647) Jolt processor - advanced UI - window size
[ https://issues.apache.org/jira/browse/NIFI-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-3647: - Description: When using the Advanced UI of the Jolt processor, the window is not correctly sized and some parts can become unreachable. Observed on both Chrome and FF. Steps to reproduce: - Go to Advanced part of the Jolt processor !Screen Shot 2017-03-24 at 6.17.07 PM.png|height=200! - Part at the bottom is truncated: !Screen Shot 2017-03-24 at 6.19.10 PM.png|height=200! - If I enter a large JSON, the top part "Jolt Transformation DSL" is not reachable anymore (unless I play with zoom in/out of the browser): !Screen Shot 2017-03-24 at 6.21.50 PM.png|height=200! was: When using the Advanced UI of the Jolt processor, the window is not correctly sized and some parts can become unreachable. Observed on both Chrome and FF. Steps to reproduce: - Go to Advanced part of the Jolt processor !Screen Shot 2017-03-24 at 6.17.07 PM.png|thumbnail! - Part at the bottom is truncated: !Screen Shot 2017-03-24 at 6.19.10 PM.png|thumbnail! - If I enter a large JSON, the top part "Jolt Transformation DSL" is not reachable anymore (unless I play with zoom in/out of the browser): !Screen Shot 2017-03-24 at 6.21.50 PM.png|thumbnail! > Jolt processor - advanced UI - window size > -- > > Key: NIFI-3647 > URL: https://issues.apache.org/jira/browse/NIFI-3647 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Pierre Villard >Priority: Minor > Attachments: Screen Shot 2017-03-24 at 6.17.07 PM.png, Screen Shot > 2017-03-24 at 6.19.10 PM.png, Screen Shot 2017-03-24 at 6.21.50 PM.png > > > When using the Advanced UI of the Jolt processor, the window is not correctly > sized and some parts can become unreachable. Observed on both Chrome and FF. > Steps to reproduce: > - Go to Advanced part of the Jolt processor > !Screen Shot 2017-03-24 at 6.17.07 PM.png|height=200! > - Part at the bottom is truncated: > !Screen Shot 2017-03-24 at 6.19.10 PM.png|height=200! > - If I enter a large JSON, the top part "Jolt Transformation DSL" is not > reachable anymore (unless I play with zoom in/out of the browser): > !Screen Shot 2017-03-24 at 6.21.50 PM.png|height=200! -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (NIFI-3647) Jolt processor - advanced UI - window size
Pierre Villard created NIFI-3647: Summary: Jolt processor - advanced UI - window size Key: NIFI-3647 URL: https://issues.apache.org/jira/browse/NIFI-3647 Project: Apache NiFi Issue Type: Bug Components: Extensions Reporter: Pierre Villard Priority: Minor Attachments: Screen Shot 2017-03-24 at 6.17.07 PM.png, Screen Shot 2017-03-24 at 6.19.10 PM.png, Screen Shot 2017-03-24 at 6.21.50 PM.png When using the Advanced UI of the Jolt processor, the window is not correctly sized and some parts can become unreachable. Observed on both Chrome and FF. Steps to reproduce: - Go to Advanced part of the Jolt processor !Screen Shot 2017-03-24 at 6.17.07 PM.png|thumbnail! - Part at the bottom is truncated: !Screen Shot 2017-03-24 at 6.19.10 PM.png|thumbnail! - If I enter a large JSON, the top part "Jolt Transformation DSL" is not reachable anymore (unless I play with zoom in/out of the browser): !Screen Shot 2017-03-24 at 6.21.50 PM.png|thumbnail! -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (NIFI-3646) Show named relationship icons on processor hover
Nicholas Carenza created NIFI-3646: -- Summary: Show named relationship icons on processor hover Key: NIFI-3646 URL: https://issues.apache.org/jira/browse/NIFI-3646 Project: Apache NiFi Issue Type: Improvement Components: Core UI Reporter: Nicholas Carenza Priority: Minor Attachments: nifi-connect-rel.png When hovering over a processor, instead of showing a single add-connection element in the middle, show an array of connections. I attached an image example. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3260) Official Docker Image
[ https://issues.apache.org/jira/browse/NIFI-3260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940778#comment-15940778 ] Aldrin Piri commented on NIFI-3260: --- We should also ensure we update the release guide to reflect the process given the very specific nature of steps in which things need to occur. > Official Docker Image > - > > Key: NIFI-3260 > URL: https://issues.apache.org/jira/browse/NIFI-3260 > Project: Apache NiFi > Issue Type: Improvement > Components: Tools and Build >Reporter: Jeremy Dyer >Assignee: Aldrin Piri > Labels: docker > Fix For: 1.2.0 > > > This JIRA is for setting up a Docker folder structure within the NiFi source > code as discussed in the dev mailing list at > https://lists.apache.org/thread.html/e905a559cb01b30f1a7032cec5c9605685f27a65bdf7fee41b735089@%3Cdev.nifi.apache.org%3E -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Reopened] (NIFI-3260) Official Docker Image
[ https://issues.apache.org/jira/browse/NIFI-3260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri reopened NIFI-3260: --- Assignee: Aldrin Piri (was: Jeremy Dyer) Reopening to ensure we are tracking INFRA changes before a release occurs. > Official Docker Image > - > > Key: NIFI-3260 > URL: https://issues.apache.org/jira/browse/NIFI-3260 > Project: Apache NiFi > Issue Type: Improvement > Components: Tools and Build >Reporter: Jeremy Dyer >Assignee: Aldrin Piri > Labels: docker > Fix For: 1.2.0 > > > This JIRA is for setting up a Docker folder structure within the NiFi source > code as discussed in the dev mailing list at > https://lists.apache.org/thread.html/e905a559cb01b30f1a7032cec5c9605685f27a65bdf7fee41b735089@%3Cdev.nifi.apache.org%3E -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi-minifi-cpp pull request #68: MINIFI-193: Ensure safe UTF encoding
Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/68 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi-cpp issue #68: MINIFI-193: Ensure safe UTF encoding
Github user apiri commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/68 Code changes look good and was able to verify execution of the binary as expected with patch changes. Will get this merged, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi-cpp issue #68: MINIFI-193: Ensure safe UTF encoding
Github user apiri commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/68 reviewing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (NIFI-3645) Add aws:kms, sse-c, sse-c-key, and sse-kms-key-id capability to server side encryption in the PutS3Object processor.
Matthew Clarke created NIFI-3645: Summary: Add aws:kms, sse-c, sse-c-key, and sse-kms-key-id capability to server side encryption in the PutS3Object processor. Key: NIFI-3645 URL: https://issues.apache.org/jira/browse/NIFI-3645 Project: Apache NiFi Issue Type: Improvement Affects Versions: 1.1.0 Reporter: Matthew Clarke Server Side Encryption (sse) currently exists in the PutS3Object processor but don’t see sse-c, sse-c-key or sse-kms-key-id options. --sse (string) Specifies server-side encryption of the object in S3. Valid values are AES256 and aws:kms. If the parameter is specified but no value is provided, AES256 is used. --sse-c (string) Specifies server-side encryption using customer provided keys of the the object in S3. AES256 is the only valid value. If the parameter is specified but no value is provided, AES256 is used. If you provide this value, --sse-c-key must be specified as well. --sse-c-key (string) The customer-provided encryption key to use to server-side encrypt the object in S3. If you provide this value, --sse-c must be specified as well. The key provided should not be base64 encoded. --sse-kms-key-id (string) The AWS KMS key ID that should be used to server-side encrypt the object in S3. Note that you should only provide this parameter if KMS key ID is different the default S3 master KMS key. Would like to see full support of the various server side encryption capabilities added to our S3 processors. This is related partially to another Apache Jira: NIFI-1769 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3518) Create a Morphlines processor
[ https://issues.apache.org/jira/browse/NIFI-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940584#comment-15940584 ] ASF GitHub Bot commented on NIFI-3518: -- Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/1576 Hey @WilliamNouet, sorry for not giving an update. Been too busy with other obligations. I'll try to get back to this when I have more free time. > Create a Morphlines processor > - > > Key: NIFI-3518 > URL: https://issues.apache.org/jira/browse/NIFI-3518 > Project: Apache NiFi > Issue Type: New Feature >Reporter: William Nouet >Priority: Minor > > Create a dedicate processor to run Morphlines transformations > (http://kitesdk.org/docs/1.1.0/morphlines/morphlines-reference-guide.html) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi issue #1576: NIFI-3518 Create a Morphlines processor
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/1576 Hey @WilliamNouet, sorry for not giving an update. Been too busy with other obligations. I'll try to get back to this when I have more free time. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi-cpp pull request #69: MINIFI-225: Add Linter for Google style gu...
Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/69#discussion_r107930194 --- Diff: thirdparty/google-styleguide/cppguide.html --- @@ -0,0 +1,6170 @@ + --- End diff -- Agreed. Will change this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi-cpp pull request #69: MINIFI-225: Add Linter for Google style gu...
Github user apiri commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/69#discussion_r107926105 --- Diff: thirdparty/google-styleguide/cppguide.html --- @@ -0,0 +1,6170 @@ + --- End diff -- Should likely just link to this in docs some place. As best as I can tell, this file is CC-By 3.0 as per https://github.com/google/styleguide/blob/gh-pages/README.md. This falls under a Category B (http://www.apache.org/legal/resolved.html#category-b) which seems to have some caveats best avoided unless there is a compelling reason to include --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi-cpp pull request #69: MINIFI-225: Add Linter for Google style gu...
Github user apiri commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/69#discussion_r107926443 --- Diff: thirdparty/google-styleguide/eclipse-cpp-google-style.xml --- @@ -0,0 +1,167 @@ + --- End diff -- Might also want to just link to this as per the LICENSE concerns above. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi-cpp pull request #69: MINIFI-225: Add Linter for Google style gu...
Github user apiri commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/69#discussion_r107925513 --- Diff: thirdparty/google-styleguide/cpplint.py --- @@ -0,0 +1,6154 @@ +#!/usr/bin/env python --- End diff -- We will need to capture this in our LICENSE file --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3380) Multiple Versions of the Same Component
[ https://issues.apache.org/jira/browse/NIFI-3380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940513#comment-15940513 ] ASF subversion and git services commented on NIFI-3380: --- Commit d90cf846b9dcd49acdeac7bb5fcd13ee80dcdba6 in nifi's branch refs/heads/master from [~bbende] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=d90cf84 ] NIFI-3380 Bumping NAR plugin to 1.2.0-SNAPSHOT development to leverage changes from master, adding buildnumber-maven-plugin to nifi-nar-bundles to properly set build info in MANIFEST of NARs - Refactoring NarDetails to include all info from MANIFEST - Adding the concept of a Bundle and refactoring NarClassLoaders to pass Bundles to ExtensionManager - Adding logic to fail start-up when multiple NARs with same coordinates exist, moving Bundle classes to framework API - Refactoring bundle API to classes and creating BundleCoordinate - Updating FlowController to use BundleCoordinate - Updating the UI and DTO model to support showing bundle details that loaded an extension type. - Adding bundle details for processor canvas node, processor dialogs, controller service dialogs, and reporting task dialogs. - Updating the formating of the bundle coordinates. - Addressing text overflow in the configuration/details dialog. - Fixing self referencing functions. - Updating extension UI mapping to incorporate bundle coordinates. - Discovering custom UIs through the supplied bundles. - Adding verification methods for creating extensions through the rest api. - Only returning extensions that are common amongst all nodes. - Rendering the ghost processors using a dotted border. - Adding bundle details to the flow.xml. - Loading NiFi build and version details from the framework NAR. - Removing properties for build and version details. - Wiring together front end and back end changes. - Including bundle coordinates in the component data model. - Wiring together component data model and flow.xml. - Addressing issue when resolve unvesioned dependent NARs. Updating unit tests to pass based on framework changes - Fixing logging of extension types during start up - Allowing the application to start if there is a compatible bundle found. - Reporting missing bundle when the a compatible bundle is not found. - Fixing table height in new component dialogs. Fixing chechstyle error and increasing test timeout for TestStandardControllerServiceProvider - Adding ability to change processor type at runtime - Adding backend code to change type for controller services - Cleaning up instance classloaders for temp components. - Creating a dialog for changing the version of a component. - Updating the formatting of the component type and bundle throughout. - Updating the new component dialogs to support selecting source group. - Cleaning up new component dialogs. - Cleaning up documentation in the cluster node endpoint. Adding missing include in nifi-web-ui pom compressor plugin - Refactoring so ConfigurableComponent provides getLogger() and so the nodes provide the ConfigurableComponent - Creating LoggableComponent to pass around the component, logger, and coordinate with in the framework - Finishing clean up following rebase. Calling lifecycle methods for add and remove when changing versions of a component - Introducing verifyCanUpdateBundle(coordinate) to ConfiguredComponent, and adding unit tests - Ensuring documentation is available for all components. Including those of the same type that are loaded from different bundles. Adding lookup from ClassLoader to Bundle, adding fix for instance class loading to include all parent NARs, and adding additional unit tests for FlowController - Adding validation to ensure referenced controller services implement the required API - Fixing template instantiation to look up compatible bundle - Requiring services/reporting tasks to be disabled/stopped. - Only supporting a change version option when the item has multiple versions available. - Limiting the possible new controller services to the applicable API version. - Showing the implemented API versions for Controller Services. - Updating the property descriptor tooltip to indicate the required service requirements. - Introducing version based sorting in the new component dialog, change version dialog, and new controller service dialog. - Addressing remainder of the issues from recent rebase. Ensuring bundles have been added to the flow before proposing a flow, and incorporating bundle information into flow fingerprinting - Refactoring the way missing bundles work to retain the desired bundle if available - Fixing logger.isDebugEnabled to be logger.isTraceEnabled - Auditing when user changes the bundle. - Ensuring bundle details are present in templates. Moving standard prioritizers to framework NAR and refactoring ExtensionManager logic to handle cases where an extension is
[jira] [Commented] (NIFI-3380) Multiple Versions of the Same Component
[ https://issues.apache.org/jira/browse/NIFI-3380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940515#comment-15940515 ] ASF GitHub Bot commented on NIFI-3380: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/1585 > Multiple Versions of the Same Component > --- > > Key: NIFI-3380 > URL: https://issues.apache.org/jira/browse/NIFI-3380 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Bryan Bende >Assignee: Bryan Bende > Fix For: 1.2.0 > > Attachments: nifi-example-processors-nar-1.0.nar, > nifi-example-processors-nar-2.0.nar, nifi-example-service-api-nar-1.0.nar, > nifi-example-service-api-nar-2.0.nar, nifi-example-service-nar-1.0.nar, > nifi-example-service-nar-1.1.nar, nifi-example-service-nar-2.0.nar > > > This ticket is to track the work for supporting multiple versions of the same > component within NiFi. The overall design for this feature is described in > detail at the following wiki page: > https://cwiki.apache.org/confluence/display/NIFI/Multiple+Versions+of+the+Same+Extension > This ticket will track only the core NiFi work, and a separate ticket will be > created to track enhancements for the NAR Maven Plugin. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (NIFI-3380) Multiple Versions of the Same Component
[ https://issues.apache.org/jira/browse/NIFI-3380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-3380: - Resolution: Fixed Status: Resolved (was: Patch Available) > Multiple Versions of the Same Component > --- > > Key: NIFI-3380 > URL: https://issues.apache.org/jira/browse/NIFI-3380 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Bryan Bende >Assignee: Bryan Bende > Fix For: 1.2.0 > > Attachments: nifi-example-processors-nar-1.0.nar, > nifi-example-processors-nar-2.0.nar, nifi-example-service-api-nar-1.0.nar, > nifi-example-service-api-nar-2.0.nar, nifi-example-service-nar-1.0.nar, > nifi-example-service-nar-1.1.nar, nifi-example-service-nar-2.0.nar > > > This ticket is to track the work for supporting multiple versions of the same > component within NiFi. The overall design for this feature is described in > detail at the following wiki page: > https://cwiki.apache.org/confluence/display/NIFI/Multiple+Versions+of+the+Same+Extension > This ticket will track only the core NiFi work, and a separate ticket will be > created to track enhancements for the NAR Maven Plugin. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3380) Multiple Versions of the Same Component
[ https://issues.apache.org/jira/browse/NIFI-3380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940517#comment-15940517 ] ASF GitHub Bot commented on NIFI-3380: -- Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/1585 This all looks good. I've run through the standard cases of standalone and clustered instances. Ran through some crazy cases that I could think of. Didn't run into any unexpected problems. Code looks good. I especially paid attention to the *Node classes, StandardFlowService, and Fingerprint classes when reviewing code but found nothing concerning. +1 merged to master! > Multiple Versions of the Same Component > --- > > Key: NIFI-3380 > URL: https://issues.apache.org/jira/browse/NIFI-3380 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Bryan Bende >Assignee: Bryan Bende > Fix For: 1.2.0 > > Attachments: nifi-example-processors-nar-1.0.nar, > nifi-example-processors-nar-2.0.nar, nifi-example-service-api-nar-1.0.nar, > nifi-example-service-api-nar-2.0.nar, nifi-example-service-nar-1.0.nar, > nifi-example-service-nar-1.1.nar, nifi-example-service-nar-2.0.nar > > > This ticket is to track the work for supporting multiple versions of the same > component within NiFi. The overall design for this feature is described in > detail at the following wiki page: > https://cwiki.apache.org/confluence/display/NIFI/Multiple+Versions+of+the+Same+Extension > This ticket will track only the core NiFi work, and a separate ticket will be > created to track enhancements for the NAR Maven Plugin. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi issue #1585: NIFI-3380 Support Multiple Versions of the Same Component
Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/1585 This all looks good. I've run through the standard cases of standalone and clustered instances. Ran through some crazy cases that I could think of. Didn't run into any unexpected problems. Code looks good. I especially paid attention to the *Node classes, StandardFlowService, and Fingerprint classes when reviewing code but found nothing concerning. +1 merged to master! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3520) HDFS processors experiencing Kerberos "impersonate" errors
[ https://issues.apache.org/jira/browse/NIFI-3520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940514#comment-15940514 ] ASF subversion and git services commented on NIFI-3520: --- Commit d90cf846b9dcd49acdeac7bb5fcd13ee80dcdba6 in nifi's branch refs/heads/master from [~bbende] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=d90cf84 ] NIFI-3380 Bumping NAR plugin to 1.2.0-SNAPSHOT development to leverage changes from master, adding buildnumber-maven-plugin to nifi-nar-bundles to properly set build info in MANIFEST of NARs - Refactoring NarDetails to include all info from MANIFEST - Adding the concept of a Bundle and refactoring NarClassLoaders to pass Bundles to ExtensionManager - Adding logic to fail start-up when multiple NARs with same coordinates exist, moving Bundle classes to framework API - Refactoring bundle API to classes and creating BundleCoordinate - Updating FlowController to use BundleCoordinate - Updating the UI and DTO model to support showing bundle details that loaded an extension type. - Adding bundle details for processor canvas node, processor dialogs, controller service dialogs, and reporting task dialogs. - Updating the formating of the bundle coordinates. - Addressing text overflow in the configuration/details dialog. - Fixing self referencing functions. - Updating extension UI mapping to incorporate bundle coordinates. - Discovering custom UIs through the supplied bundles. - Adding verification methods for creating extensions through the rest api. - Only returning extensions that are common amongst all nodes. - Rendering the ghost processors using a dotted border. - Adding bundle details to the flow.xml. - Loading NiFi build and version details from the framework NAR. - Removing properties for build and version details. - Wiring together front end and back end changes. - Including bundle coordinates in the component data model. - Wiring together component data model and flow.xml. - Addressing issue when resolve unvesioned dependent NARs. Updating unit tests to pass based on framework changes - Fixing logging of extension types during start up - Allowing the application to start if there is a compatible bundle found. - Reporting missing bundle when the a compatible bundle is not found. - Fixing table height in new component dialogs. Fixing chechstyle error and increasing test timeout for TestStandardControllerServiceProvider - Adding ability to change processor type at runtime - Adding backend code to change type for controller services - Cleaning up instance classloaders for temp components. - Creating a dialog for changing the version of a component. - Updating the formatting of the component type and bundle throughout. - Updating the new component dialogs to support selecting source group. - Cleaning up new component dialogs. - Cleaning up documentation in the cluster node endpoint. Adding missing include in nifi-web-ui pom compressor plugin - Refactoring so ConfigurableComponent provides getLogger() and so the nodes provide the ConfigurableComponent - Creating LoggableComponent to pass around the component, logger, and coordinate with in the framework - Finishing clean up following rebase. Calling lifecycle methods for add and remove when changing versions of a component - Introducing verifyCanUpdateBundle(coordinate) to ConfiguredComponent, and adding unit tests - Ensuring documentation is available for all components. Including those of the same type that are loaded from different bundles. Adding lookup from ClassLoader to Bundle, adding fix for instance class loading to include all parent NARs, and adding additional unit tests for FlowController - Adding validation to ensure referenced controller services implement the required API - Fixing template instantiation to look up compatible bundle - Requiring services/reporting tasks to be disabled/stopped. - Only supporting a change version option when the item has multiple versions available. - Limiting the possible new controller services to the applicable API version. - Showing the implemented API versions for Controller Services. - Updating the property descriptor tooltip to indicate the required service requirements. - Introducing version based sorting in the new component dialog, change version dialog, and new controller service dialog. - Addressing remainder of the issues from recent rebase. Ensuring bundles have been added to the flow before proposing a flow, and incorporating bundle information into flow fingerprinting - Refactoring the way missing bundles work to retain the desired bundle if available - Fixing logger.isDebugEnabled to be logger.isTraceEnabled - Auditing when user changes the bundle. - Ensuring bundle details are present in templates. Moving standard prioritizers to framework NAR and refactoring ExtensionManager logic to handle cases where an extension is
[jira] [Commented] (NIFI-3380) Multiple Versions of the Same Component
[ https://issues.apache.org/jira/browse/NIFI-3380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940512#comment-15940512 ] ASF subversion and git services commented on NIFI-3380: --- Commit d90cf846b9dcd49acdeac7bb5fcd13ee80dcdba6 in nifi's branch refs/heads/master from [~bbende] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=d90cf84 ] NIFI-3380 Bumping NAR plugin to 1.2.0-SNAPSHOT development to leverage changes from master, adding buildnumber-maven-plugin to nifi-nar-bundles to properly set build info in MANIFEST of NARs - Refactoring NarDetails to include all info from MANIFEST - Adding the concept of a Bundle and refactoring NarClassLoaders to pass Bundles to ExtensionManager - Adding logic to fail start-up when multiple NARs with same coordinates exist, moving Bundle classes to framework API - Refactoring bundle API to classes and creating BundleCoordinate - Updating FlowController to use BundleCoordinate - Updating the UI and DTO model to support showing bundle details that loaded an extension type. - Adding bundle details for processor canvas node, processor dialogs, controller service dialogs, and reporting task dialogs. - Updating the formating of the bundle coordinates. - Addressing text overflow in the configuration/details dialog. - Fixing self referencing functions. - Updating extension UI mapping to incorporate bundle coordinates. - Discovering custom UIs through the supplied bundles. - Adding verification methods for creating extensions through the rest api. - Only returning extensions that are common amongst all nodes. - Rendering the ghost processors using a dotted border. - Adding bundle details to the flow.xml. - Loading NiFi build and version details from the framework NAR. - Removing properties for build and version details. - Wiring together front end and back end changes. - Including bundle coordinates in the component data model. - Wiring together component data model and flow.xml. - Addressing issue when resolve unvesioned dependent NARs. Updating unit tests to pass based on framework changes - Fixing logging of extension types during start up - Allowing the application to start if there is a compatible bundle found. - Reporting missing bundle when the a compatible bundle is not found. - Fixing table height in new component dialogs. Fixing chechstyle error and increasing test timeout for TestStandardControllerServiceProvider - Adding ability to change processor type at runtime - Adding backend code to change type for controller services - Cleaning up instance classloaders for temp components. - Creating a dialog for changing the version of a component. - Updating the formatting of the component type and bundle throughout. - Updating the new component dialogs to support selecting source group. - Cleaning up new component dialogs. - Cleaning up documentation in the cluster node endpoint. Adding missing include in nifi-web-ui pom compressor plugin - Refactoring so ConfigurableComponent provides getLogger() and so the nodes provide the ConfigurableComponent - Creating LoggableComponent to pass around the component, logger, and coordinate with in the framework - Finishing clean up following rebase. Calling lifecycle methods for add and remove when changing versions of a component - Introducing verifyCanUpdateBundle(coordinate) to ConfiguredComponent, and adding unit tests - Ensuring documentation is available for all components. Including those of the same type that are loaded from different bundles. Adding lookup from ClassLoader to Bundle, adding fix for instance class loading to include all parent NARs, and adding additional unit tests for FlowController - Adding validation to ensure referenced controller services implement the required API - Fixing template instantiation to look up compatible bundle - Requiring services/reporting tasks to be disabled/stopped. - Only supporting a change version option when the item has multiple versions available. - Limiting the possible new controller services to the applicable API version. - Showing the implemented API versions for Controller Services. - Updating the property descriptor tooltip to indicate the required service requirements. - Introducing version based sorting in the new component dialog, change version dialog, and new controller service dialog. - Addressing remainder of the issues from recent rebase. Ensuring bundles have been added to the flow before proposing a flow, and incorporating bundle information into flow fingerprinting - Refactoring the way missing bundles work to retain the desired bundle if available - Fixing logger.isDebugEnabled to be logger.isTraceEnabled - Auditing when user changes the bundle. - Ensuring bundle details are present in templates. Moving standard prioritizers to framework NAR and refactoring ExtensionManager logic to handle cases where an extension is
[GitHub] nifi pull request #1585: NIFI-3380 Support Multiple Versions of the Same Com...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/1585 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi-cpp issue #69: MINIFI-225: Add Linter for Google style guide
Github user apiri commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/69 reviewing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi-cpp pull request #67: Ios
Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/67 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3413) Implement a GetChangeDataCapture processor
[ https://issues.apache.org/jira/browse/NIFI-3413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940497#comment-15940497 ] ASF GitHub Bot commented on NIFI-3413: -- Github user phrocker commented on the issue: https://github.com/apache/nifi/pull/1618 @mattyb149 I've left some minor comments. I'm going to pull this down and test this. I'm super excited. Looks cool. > Implement a GetChangeDataCapture processor > -- > > Key: NIFI-3413 > URL: https://issues.apache.org/jira/browse/NIFI-3413 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess > > Database systems such as MySQL, Oracle, and SQL Server allow access to their > transactional logs and such, in order for external clients to have a "change > data capture" (CDC) capability. I propose a GetChangeDataCapture processor to > enable this in NiFi. > The processor would be configured with a DBCPConnectionPool controller > service, as well as a Database Type property (similar to the one in > QueryDatabaseTable) for database-specific handling. Additional properties > might include the CDC table name, etc. Additional database-specific > properties could be handled using dynamic properties (and the documentation > should reflect this). > The processor would accept no incoming connections (it is a "Get" or source > processor), would be intended to run on the primary node only as a single > threaded processor, and would generate a flow file for each operation > (INSERT, UPDATE, DELETE, e,g,) in one or some number of formats (JSON, e.g.). > The flow files would be transferred in time order (to enable a replication > solution, for example), perhaps with some auto-incrementing attribute to also > indicate order if need be. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi issue #1618: NIFI-3413: Add GetChangeDataCaptureMySQL processor
Github user phrocker commented on the issue: https://github.com/apache/nifi/pull/1618 @mattyb149 I've left some minor comments. I'm going to pull this down and test this. I'm super excited. Looks cool. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (NIFI-1352) Add ability to add a zip comment in MergeContent
[ https://issues.apache.org/jira/browse/NIFI-1352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Niemiec updated NIFI-1352: - Affects Version/s: 1.2.0 Status: Patch Available (was: Open) > Add ability to add a zip comment in MergeContent > > > Key: NIFI-1352 > URL: https://issues.apache.org/jira/browse/NIFI-1352 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Brandon DeVries >Assignee: Gray Gwizdz >Priority: Trivial > Attachments: > 0001-NIFI-1352-Add-ability-to-add-a-zip-comment-in-MergeC.patch > > > The Zip format includes a file comment, which can be written with a > ZipOutputStream \[1\]. Modify MergeContent to optionally include a comment > (possibly via a "zip.comment" attribute or something...). > \[1\] > https://docs.oracle.com/javase/7/docs/api/java/util/zip/ZipOutputStream.html#setComment%28java.lang.String%29 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (NIFI-3583) PutFile should attempt to replace files atomically
[ https://issues.apache.org/jira/browse/NIFI-3583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Niemiec updated NIFI-3583: - Status: Patch Available (was: Open) > PutFile should attempt to replace files atomically > -- > > Key: NIFI-3583 > URL: https://issues.apache.org/jira/browse/NIFI-3583 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.1.1, 0.7.1 > Environment: On Windows, NIO2's Files.move operation may not allow > atomic file replacement, in which case it would have to fall back to the > existing method. >Reporter: Eugene Hart >Assignee: Gray Gwizdz >Priority: Trivial > Attachments: > 0001-NIFI-3853-PutFile-should-attempt-to-replace-files-at.patch > > > A PutFile configured with the "replace" conflict resolution strategy will > first delete any existing file and then move the new temporary file into > place. This causes there to be a small period of time where the file does not > exist at all. This could cause problems with other processes relying on this > file to always exist. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (NIFI-3644) Add DetectDuplicateUsingHBase processor
[ https://issues.apache.org/jira/browse/NIFI-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940467#comment-15940467 ] Bjorn Olsen edited comment on NIFI-3644 at 3/24/17 2:41 PM: Hi Joe Thanks for the suggestion, I hadn't considered writing an HBase version of DistributedMapCache. I've already written my own DetectDuplicateUsingHBase processor today, as I needed something that was quick to develop. Working code here, uses HBaseClientService & much copy-pasta from DetectDuplicate: https://github.com/baolsen/nifi/blob/DetectDuplicateUsingHBase/nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DetectDuplicateUsingHBase.java It seems that implementing an HBase-based DistributedMapCache is more complex, but more reusable. Do you have any suggestions for documentation for this sort of thing? Lastly, do you think it is worth including DetectDuplicateUsingHBase or rather wait for a more reusable option? I'm a bit tight on time, and Java and NiFi are both new to me. Meanwhile I can keep DetectDuplicateUsingHBase for my own use, so no worries there. was (Author: bjorn.ols...@gmail.com): Hi Joe Thanks for the suggestion, I hadn't considered writing an HBase version of DistributedMapCache. I've already written my own DetectDuplicateUsingHBase processor today, as I needed something that was quick to develop. Working code here, much copy-pasta from DetectDuplicate: https://github.com/baolsen/nifi/blob/DetectDuplicateUsingHBase/nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DetectDuplicateUsingHBase.java It seems that implementing an HBase-based DistributedMapCache is more complex, but more reusable. Do you have any suggestions for documentation for this sort of thing? Lastly, do you think it is worth including DetectDuplicateUsingHBase or rather wait for a more reusable option? I'm a bit tight on time, and Java and NiFi are both new to me. Meanwhile I can keep DetectDuplicateUsingHBase for my own use, so no worries there. > Add DetectDuplicateUsingHBase processor > --- > > Key: NIFI-3644 > URL: https://issues.apache.org/jira/browse/NIFI-3644 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Bjorn Olsen >Priority: Minor > > The DetectDuplicate processor makes use of a distributed map cache for > maintaining a list of unique file identifiers (such as hashes). > The distributed map cache functionality could be provided by an HBase table, > which then allows for reliably storing a huge volume of file identifiers and > auditing information. The downside of this approach is of course that HBase > is required. > Storing the unique file identifiers in a reliable, query-able manner along > with some audit information is of benefit to several use cases. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3644) Add DetectDuplicateUsingHBase processor
[ https://issues.apache.org/jira/browse/NIFI-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940467#comment-15940467 ] Bjorn Olsen commented on NIFI-3644: --- Hi Joe Thanks for the suggestion, I hadn't considered writing an HBase version of DistributedMapCache. I've already written my own DetectDuplicateUsingHBase processor today, as I needed something that was quick to develop. Code here, much copy-pasta from DetectDuplicate: https://github.com/baolsen/nifi/blob/DetectDuplicateUsingHBase/nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DetectDuplicateUsingHBase.java It seems that implementing an HBase-based DistributedMapCache is more complex, but more reusable. Do you have any suggestions for documentation for this sort of thing? Lastly, do you think it is worth including DetectDuplicateUsingHBase or rather wait for a more reusable option? I'm a bit tight on time, and Java and NiFi are both new to me. Meanwhile I can keep DetectDuplicateUsingHBase for my own use, so no worries there. > Add DetectDuplicateUsingHBase processor > --- > > Key: NIFI-3644 > URL: https://issues.apache.org/jira/browse/NIFI-3644 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Bjorn Olsen >Priority: Minor > > The DetectDuplicate processor makes use of a distributed map cache for > maintaining a list of unique file identifiers (such as hashes). > The distributed map cache functionality could be provided by an HBase table, > which then allows for reliably storing a huge volume of file identifiers and > auditing information. The downside of this approach is of course that HBase > is required. > Storing the unique file identifiers in a reliable, query-able manner along > with some audit information is of benefit to several use cases. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (NIFI-3644) Add DetectDuplicateUsingHBase processor
[ https://issues.apache.org/jira/browse/NIFI-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940467#comment-15940467 ] Bjorn Olsen edited comment on NIFI-3644 at 3/24/17 2:39 PM: Hi Joe Thanks for the suggestion, I hadn't considered writing an HBase version of DistributedMapCache. I've already written my own DetectDuplicateUsingHBase processor today, as I needed something that was quick to develop. Working code here, much copy-pasta from DetectDuplicate: https://github.com/baolsen/nifi/blob/DetectDuplicateUsingHBase/nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DetectDuplicateUsingHBase.java It seems that implementing an HBase-based DistributedMapCache is more complex, but more reusable. Do you have any suggestions for documentation for this sort of thing? Lastly, do you think it is worth including DetectDuplicateUsingHBase or rather wait for a more reusable option? I'm a bit tight on time, and Java and NiFi are both new to me. Meanwhile I can keep DetectDuplicateUsingHBase for my own use, so no worries there. was (Author: bjorn.ols...@gmail.com): Hi Joe Thanks for the suggestion, I hadn't considered writing an HBase version of DistributedMapCache. I've already written my own DetectDuplicateUsingHBase processor today, as I needed something that was quick to develop. Code here, much copy-pasta from DetectDuplicate: https://github.com/baolsen/nifi/blob/DetectDuplicateUsingHBase/nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DetectDuplicateUsingHBase.java It seems that implementing an HBase-based DistributedMapCache is more complex, but more reusable. Do you have any suggestions for documentation for this sort of thing? Lastly, do you think it is worth including DetectDuplicateUsingHBase or rather wait for a more reusable option? I'm a bit tight on time, and Java and NiFi are both new to me. Meanwhile I can keep DetectDuplicateUsingHBase for my own use, so no worries there. > Add DetectDuplicateUsingHBase processor > --- > > Key: NIFI-3644 > URL: https://issues.apache.org/jira/browse/NIFI-3644 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Bjorn Olsen >Priority: Minor > > The DetectDuplicate processor makes use of a distributed map cache for > maintaining a list of unique file identifiers (such as hashes). > The distributed map cache functionality could be provided by an HBase table, > which then allows for reliably storing a huge volume of file identifiers and > auditing information. The downside of this approach is of course that HBase > is required. > Storing the unique file identifiers in a reliable, query-able manner along > with some audit information is of benefit to several use cases. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3313) First deployment of NiFi can hang on VMs without sufficient entropy if using /dev/random
[ https://issues.apache.org/jira/browse/NIFI-3313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940402#comment-15940402 ] Michael Moser commented on NIFI-3313: - I was seeing an extra 30 to 120 seconds on startup, but WHOOPS I got my environments confused. I've been switching between 0.x (Java 7) and 1.x (java 8) a lot lately, and I used {{file:/dev/urandom}} in my 0.x bootstrap.conf. When I added the extra {{/.}} to my 0.x bootstrap.conf and the extra startup delay was gone, I came here and commented. Sorry! > First deployment of NiFi can hang on VMs without sufficient entropy if using > /dev/random > > > Key: NIFI-3313 > URL: https://issues.apache.org/jira/browse/NIFI-3313 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.1.1 >Reporter: Andy LoPresto >Assignee: Andy LoPresto >Priority: Critical > Labels: entropy, security, virtual-machine > Fix For: 1.2.0 > > > h1. Analysis of Issue > h2. Statement of Problem: > NiFi deployed on headless VM (little user interaction by way of keyboard and > mouse I/O) can take 5-10 minutes (reported) to start up. User reports this > occurs on a "secure" cluster. Further examination is required to determine > which specific process requires the large amount of random input (no steps to > reproduce, configuration files, logs, or VM environment information > provided). > h2. Context > The likely cause of this issue is that a process is attempting to read from > _/dev/random_, a \*nix "device" providing a pseudo-random number generator > (PRNG). Also available is _/dev/urandom_, a related PRNG. Despite common > misperceptions, _/dev/urandom_ is not "less-secure" than _/dev/random_ for > all general use cases. _/dev/random_ blocks if the entropy *estimate* (a > "guess" of the existing entropy introduced into the pool) is lower than the > amount of random data requested by the caller. In contrast, _/dev/urandom_ > does not block, but provides the output of the same cryptographically-secure > PRNG (CSPRNG) that _/dev/random_ reads from \[myths\]. After as little as 256 > bytes of initial seeding, accessing _/dev/random_ and _/dev/urandom_ are > functionally equivalent, as the long period of random data generated will not > require re-seeding before sufficient entropy can be provided again. > As mentioned earlier, further examination is required to determine if the > process requiring random input occurs at application boot or only at > "machine" (hardware or VM) boot. On the first deployment of the system with > certificates, the certificate generation process will require substantial > random input. However, on application launch and connection to a cluster, > even the TLS/SSL protocol requires some amount of random input. > h2. Proposed Solutions > h3. rngd > A software toolset for accessing dedicated hardware PRNG (*true* RNG, or > TRNG) called _rng-tools_ \[rngtools\] exists for Linux. Specialized hardware, > as well as Intel chips from IvyBridge and on (2012), can provide > hardware-generated random input to the kernel. Using the daemon _rngd_ to > seed the _/dev/random_ and _/dev/urandom_ entropy pool is the simplest > solution. > *Note: Do not use _/dev/urandom_ to seed _/dev/random_ using _rngd_. This is > like running a garden hose from a car's exhaust back into its gas tank and > trying to drive.* > h3. Instruct Java to use /dev/urandom > The Java Runtime Environment (JRE) can be instructed to use _/dev/urandom_ > for all invocations of {{SecureRandom}}, either on a per-Java process basis > \[jdk-urandom\] or in the JVM configuration \[oracle-urandom\], which means > it will not block on server startup. The NiFi {{bootstrap.conf}} file can be > modified to contain an additional Java argument directing the JVM to use > _/dev/urandom_. > h2. Other Solutions > h3. Entropy Gathering Tools > Tools to gather entropy from non-standard sources (audio card noise, video > capture from webcams, etc.) have been developed such as audio-entropyd > \[wagner\], but these tools are not verified or well-examined -- usually when > tested, they are only tested for the strength of their PRNG, not the ability > of the tool to capture entropy and generate sufficiently random data > unavailable to an attacker who may be able to determine the internal state. > h3. haveged > A solution has been proposed to use {{havaged}} \[haveged\], a user-space > daemon relying on the HAVEGE (HArdware Volatile Entropy Gathering and > Expansion) construct to continually increase the entropy on the system, > allowing _/dev/random_ to run without blocking. > However, on further investigation, multiple sources indica
[jira] [Commented] (NIFI-3639) Add HBase Get to HBase_1_1_2_ClientService
[ https://issues.apache.org/jira/browse/NIFI-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940374#comment-15940374 ] ASF GitHub Bot commented on NIFI-3639: -- Github user baolsen closed the pull request at: https://github.com/apache/nifi/pull/1615 > Add HBase Get to HBase_1_1_2_ClientService > -- > > Key: NIFI-3639 > URL: https://issues.apache.org/jira/browse/NIFI-3639 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Bjorn Olsen >Priority: Trivial > > Enhance HBase_1_1_2_ClientService and API to provide HBase Get functionality. > Currently only Put and Scan are supported. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi pull request #1615: NIFI-3639 - Implemented HBase Get functionality for...
Github user baolsen closed the pull request at: https://github.com/apache/nifi/pull/1615 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3639) Add HBase Get to HBase_1_1_2_ClientService
[ https://issues.apache.org/jira/browse/NIFI-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940373#comment-15940373 ] ASF GitHub Bot commented on NIFI-3639: -- Github user baolsen commented on the issue: https://github.com/apache/nifi/pull/1615 Great, thanks for all the info. Will close this PR meanwhile. > Add HBase Get to HBase_1_1_2_ClientService > -- > > Key: NIFI-3639 > URL: https://issues.apache.org/jira/browse/NIFI-3639 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Bjorn Olsen >Priority: Trivial > > Enhance HBase_1_1_2_ClientService and API to provide HBase Get functionality. > Currently only Put and Scan are supported. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi issue #1615: NIFI-3639 - Implemented HBase Get functionality for HBaseC...
Github user baolsen commented on the issue: https://github.com/apache/nifi/pull/1615 Great, thanks for all the info. Will close this PR meanwhile. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFIREG-2) Design logo for Registry
[ https://issues.apache.org/jira/browse/NIFIREG-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940341#comment-15940341 ] Rob Moran commented on NIFIREG-2: - Great feedback so far; all very valid points – thanks! I will continue to push the design and post updates. > Design logo for Registry > > > Key: NIFIREG-2 > URL: https://issues.apache.org/jira/browse/NIFIREG-2 > Project: NiFi Registry > Issue Type: Task >Reporter: Rob Moran >Assignee: Rob Moran >Priority: Minor > Attachments: registry-logo-concept.png > > > The attached image contains variations of the proposed logo design for > Registry. The points below describe some of the thinking behind it: > * Relationship to NiFi and MiNiFi through the use of the same typeface and > use of blocks representing bits of data > * For Registry these blocks also represent the storage/organization aspect > through their even distribution and arrangement > * The 3 gradated blocks across the top – forming the terminal part of a > lowercase *r* – represent movement (e.g., a versioned flow being saved to > NiFi or imported to NiFi from the registry) > * The solid gray (in color versions) and outlined block (in one-color > versions) help with idea of storage as previously mentioned, but also allude > to unused storage/free space > * The gray block also helps establish the strong diagonal slicing through it > and the lowest green block. Again this helps with the idea of movement, but > more so speaks to how Registry operates in the background, tucked away, > largely unseen by NiFi operators > * A departure from the NiFi color palette signifies how Registry functions > more as a standalone application -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFIREG-2) Design logo for Registry
[ https://issues.apache.org/jira/browse/NIFIREG-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940318#comment-15940318 ] Bryan Bende commented on NIFIREG-2: --- [~rmoran] thanks for putting this together! definitely cool to start seeing the registry take form. My only thought is that we might want the color scheme to be consistent with the existing NiFi/MiNiFi colors, but if others are strongly in favor of a different color scheme then I can get on-board. > Design logo for Registry > > > Key: NIFIREG-2 > URL: https://issues.apache.org/jira/browse/NIFIREG-2 > Project: NiFi Registry > Issue Type: Task >Reporter: Rob Moran >Assignee: Rob Moran >Priority: Minor > Attachments: registry-logo-concept.png > > > The attached image contains variations of the proposed logo design for > Registry. The points below describe some of the thinking behind it: > * Relationship to NiFi and MiNiFi through the use of the same typeface and > use of blocks representing bits of data > * For Registry these blocks also represent the storage/organization aspect > through their even distribution and arrangement > * The 3 gradated blocks across the top – forming the terminal part of a > lowercase *r* – represent movement (e.g., a versioned flow being saved to > NiFi or imported to NiFi from the registry) > * The solid gray (in color versions) and outlined block (in one-color > versions) help with idea of storage as previously mentioned, but also allude > to unused storage/free space > * The gray block also helps establish the strong diagonal slicing through it > and the lowest green block. Again this helps with the idea of movement, but > more so speaks to how Registry operates in the background, tucked away, > largely unseen by NiFi operators > * A departure from the NiFi color palette signifies how Registry functions > more as a standalone application -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (NIFI-2481) FileSystemRepository: explain maxArchiveBytes calculation better
[ https://issues.apache.org/jira/browse/NIFI-2481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Moser updated NIFI-2481: Resolution: Fixed Fix Version/s: 1.2.0 Status: Resolved (was: Patch Available) > FileSystemRepository: explain maxArchiveBytes calculation better > > > Key: NIFI-2481 > URL: https://issues.apache.org/jira/browse/NIFI-2481 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Brandon DeVries >Assignee: Michael Moser >Priority: Minor > Fix For: 1.2.0 > > > It appears as though the calculation of maxArchiveBytes in > FileSystemRepository.java might be backwards. The line in question\[1\] is: > {code} > final long maxArchiveBytes = (long) (capacity * (1D - (maxArchiveRatio - > 0.02))); > {code} > if you say capacity = 1000, then you get the following: > maxArchiveRatio = .1 ==> maxArchiveBytes = 920 > maxArchiveRatio = .5 ==> maxArchiveBytes = 520 > maxArchiveRatio = .9 ==> maxArchiveBytes = 120 > In other words, as you increase the > nifi.content.repository.archive.max.usage.percentage, you would be decreasing > the bytes allowed for archiving... > \[1\] > https://github.com/apache/nifi/blob/rel/nifi-0.7.0/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/FileSystemRepository.java#L189 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (NIFI-2481) FileSystemRepository: explain maxArchiveBytes calculation better
[ https://issues.apache.org/jira/browse/NIFI-2481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Moser updated NIFI-2481: Summary: FileSystemRepository: explain maxArchiveBytes calculation better (was: FileSystemRepository: maxArchiveBytes calculated incorrectly) > FileSystemRepository: explain maxArchiveBytes calculation better > > > Key: NIFI-2481 > URL: https://issues.apache.org/jira/browse/NIFI-2481 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Brandon DeVries >Assignee: Michael Moser >Priority: Minor > > It appears as though the calculation of maxArchiveBytes in > FileSystemRepository.java might be backwards. The line in question\[1\] is: > {code} > final long maxArchiveBytes = (long) (capacity * (1D - (maxArchiveRatio - > 0.02))); > {code} > if you say capacity = 1000, then you get the following: > maxArchiveRatio = .1 ==> maxArchiveBytes = 920 > maxArchiveRatio = .5 ==> maxArchiveBytes = 520 > maxArchiveRatio = .9 ==> maxArchiveBytes = 120 > In other words, as you increase the > nifi.content.repository.archive.max.usage.percentage, you would be decreasing > the bytes allowed for archiving... > \[1\] > https://github.com/apache/nifi/blob/rel/nifi-0.7.0/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/FileSystemRepository.java#L189 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFIREG-2) Design logo for Registry
[ https://issues.apache.org/jira/browse/NIFIREG-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940265#comment-15940265 ] Joseph Witt commented on NIFIREG-2: --- hey rob thanks for going after this and for sharing your thoughts behind it. It looks nice and I am overall favorable to it. Some initial thoughts I had: - I'd like to see some connection to the nifi/minifi theming. Whether that is via color or inclusion of some 'flow/water/fluid' like visuals is ok. - the registry concept while it is a standalone app it is entirely centered around providing central storage of nifi/minifi specific artifacts such as extensions, flows, and variables. If there was some way to capture that ... With the registry we're not talking about the flow so much as we're talking about what controls/defines the flow so perhaps pipes in the shape of an R? Anyway all that said my brain totally does not afford the same awesomeness yours does when it comes to visualizing concepts. I definitely defer to your thoughts on this just wanted to share some of my old school nifi thinking. > Design logo for Registry > > > Key: NIFIREG-2 > URL: https://issues.apache.org/jira/browse/NIFIREG-2 > Project: NiFi Registry > Issue Type: Task >Reporter: Rob Moran >Assignee: Rob Moran >Priority: Minor > Attachments: registry-logo-concept.png > > > The attached image contains variations of the proposed logo design for > Registry. The points below describe some of the thinking behind it: > * Relationship to NiFi and MiNiFi through the use of the same typeface and > use of blocks representing bits of data > * For Registry these blocks also represent the storage/organization aspect > through their even distribution and arrangement > * The 3 gradated blocks across the top – forming the terminal part of a > lowercase *r* – represent movement (e.g., a versioned flow being saved to > NiFi or imported to NiFi from the registry) > * The solid gray (in color versions) and outlined block (in one-color > versions) help with idea of storage as previously mentioned, but also allude > to unused storage/free space > * The gray block also helps establish the strong diagonal slicing through it > and the lowest green block. Again this helps with the idea of movement, but > more so speaks to how Registry operates in the background, tucked away, > largely unseen by NiFi operators > * A departure from the NiFi color palette signifies how Registry functions > more as a standalone application -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3639) Add HBase Get to HBase_1_1_2_ClientService
[ https://issues.apache.org/jira/browse/NIFI-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940254#comment-15940254 ] ASF GitHub Bot commented on NIFI-3639: -- Github user bbende commented on the issue: https://github.com/apache/nifi/pull/1615 @baolsen Thanks for the info. I believe only you can close the pull request on your end. As far as the automated checks, you can ignore appveryor, but the travis build should be passing as of yesterday. Looking at the travis logs for your build it shows: [INFO] --- maven-checkstyle-plugin:2.15:check (check-style) @ nifi-hbase-client-service-api --- [WARNING] src/main/java/org/apache/nifi/hbase/HBaseClientService.java[131] (regexp) RegexpSinglelineJava: Line has trailing whitespace. You can detect this locally by running "mvn clean install -Pcontrib-check" before submitting anything. The contrib-check will fail if anything doesn't pass checkstyles or the RAT licensing check. > Add HBase Get to HBase_1_1_2_ClientService > -- > > Key: NIFI-3639 > URL: https://issues.apache.org/jira/browse/NIFI-3639 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Bjorn Olsen >Priority: Trivial > > Enhance HBase_1_1_2_ClientService and API to provide HBase Get functionality. > Currently only Put and Scan are supported. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi issue #1615: NIFI-3639 - Implemented HBase Get functionality for HBaseC...
Github user bbende commented on the issue: https://github.com/apache/nifi/pull/1615 @baolsen Thanks for the info. I believe only you can close the pull request on your end. As far as the automated checks, you can ignore appveryor, but the travis build should be passing as of yesterday. Looking at the travis logs for your build it shows: [INFO] --- maven-checkstyle-plugin:2.15:check (check-style) @ nifi-hbase-client-service-api --- [WARNING] src/main/java/org/apache/nifi/hbase/HBaseClientService.java[131] (regexp) RegexpSinglelineJava: Line has trailing whitespace. You can detect this locally by running "mvn clean install -Pcontrib-check" before submitting anything. The contrib-check will fail if anything doesn't pass checkstyles or the RAT licensing check. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3644) Add DetectDuplicateUsingHBase processor
[ https://issues.apache.org/jira/browse/NIFI-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940247#comment-15940247 ] Joseph Witt commented on NIFI-3644: --- Bjorn, We can add you to the contributors list in JIRA so that you can assign items to yourself. However, in the meantime you can definitely contribute and work on tasks without this. For this concept please note you should only need to create an implementation of the DistributedMapCache which is backed by HBase rather than a new processor. DetectDuplicate can use any implementation of that interface by design. Thanks Joe > Add DetectDuplicateUsingHBase processor > --- > > Key: NIFI-3644 > URL: https://issues.apache.org/jira/browse/NIFI-3644 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Bjorn Olsen >Priority: Minor > > The DetectDuplicate processor makes use of a distributed map cache for > maintaining a list of unique file identifiers (such as hashes). > The distributed map cache functionality could be provided by an HBase table, > which then allows for reliably storing a huge volume of file identifiers and > auditing information. The downside of this approach is of course that HBase > is required. > Storing the unique file identifiers in a reliable, query-able manner along > with some audit information is of benefit to several use cases. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-79) Add support for bulletin level 'NONE'
[ https://issues.apache.org/jira/browse/NIFI-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940217#comment-15940217 ] Andre F de Miranda commented on NIFI-79: I bet it is so simple... but yet so amazingly scary journey into the depths of NiFi... :-) > Add support for bulletin level 'NONE' > - > > Key: NIFI-79 > URL: https://issues.apache.org/jira/browse/NIFI-79 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework, Core UI >Reporter: Joseph Witt >Assignee: Andre F de Miranda >Priority: Trivial > > sometimes you just don't care what a processor has to say -- This message was sent by Atlassian JIRA (v6.3.15#6346)