date:20170324


[ 
https://issues.apache.org/jira/browse/NIFI-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940072#comment-15940072
 ] 

ASF GitHub Bot commented on NIFI-3639:
--

Github user wangrenlei commented on the issue:

https://github.com/apache/nifi/pull/1615
  
@baolsen  you should check your style of code .


> Add HBase Get to HBase_1_1_2_ClientService
> --
>
> Key: NIFI-3639
> URL: https://issues.apache.org/jira/browse/NIFI-3639
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Bjorn Olsen
>Priority: Trivial
>
> Enhance HBase_1_1_2_ClientService and API to provide HBase Get functionality. 
> Currently only Put and Scan are supported.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1619: NIFI-2747 - Introduce FuzzyHashContent processor

GitHub user trixpan opened a pull request:

https://github.com/apache/nifi/pull/1619

NIFI-2747 - Introduce FuzzyHashContent processor

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [X] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [X] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [X] ~~~Is your initial contribution a single, squashed commit?~~~

### For code changes:
- [X] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [X] Have you written or updated unit tests to verify your changes?
- [X] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [X] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [X] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [X] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [X] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/trixpan/nifi NIFI-2747

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1619.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1619


commit 0f810d513d57d799e976f797d4c2062ab3e59ee2
Author: Andre F de Miranda 
Date:   2017-03-24T06:53:58Z

NIFI-2747 - Introduce FuzzyHashContent processor

commit 6f47cb52fcae254782cb47364f51ffcd999bc22a
Author: Andre F de Miranda 
Date:   2017-03-24T10:22:23Z

NIFI-3466 - Minor typo missed during devel/review




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2747) Add Processor support SSDeep (SpamSum) hashes (and possibly others e.g. TLSH)


[ 
https://issues.apache.org/jira/browse/NIFI-2747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940117#comment-15940117
 ] 

ASF GitHub Bot commented on NIFI-2747:
--

GitHub user trixpan opened a pull request:

https://github.com/apache/nifi/pull/1619

NIFI-2747 - Introduce FuzzyHashContent processor

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [X] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [X] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [X] ~~~Is your initial contribution a single, squashed commit?~~~

### For code changes:
- [X] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [X] Have you written or updated unit tests to verify your changes?
- [X] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [X] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [X] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [X] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [X] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/trixpan/nifi NIFI-2747

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1619.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1619


commit 0f810d513d57d799e976f797d4c2062ab3e59ee2
Author: Andre F de Miranda 
Date:   2017-03-24T06:53:58Z

NIFI-2747 - Introduce FuzzyHashContent processor

commit 6f47cb52fcae254782cb47364f51ffcd999bc22a
Author: Andre F de Miranda 
Date:   2017-03-24T10:22:23Z

NIFI-3466 - Minor typo missed during devel/review




> Add Processor support SSDeep (SpamSum) hashes (and possibly others e.g. TLSH)
> -
>
> Key: NIFI-2747
> URL: https://issues.apache.org/jira/browse/NIFI-2747
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.0.0
>Reporter: Andre F de Miranda
>Assignee: Andre F de Miranda
>
> It would be great if we had a processor to support fuzzy hashing algorithms 
> such as SpamSum/SSDeep



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi issue #1619: NIFI-2747 - Introduce FuzzyHashContent processor

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/1619
  
Note: This PR also contain a minor typo correction to "logic independent" 
part of of ParseCEF processor.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2747) Add Processor support SSDeep (SpamSum) hashes (and possibly others e.g. TLSH)


[ 
https://issues.apache.org/jira/browse/NIFI-2747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940128#comment-15940128
 ] 

ASF GitHub Bot commented on NIFI-2747:
--

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/1619
  
Note: This PR also contain a minor typo correction to "logic independent" 
part of of ParseCEF processor.


> Add Processor support SSDeep (SpamSum) hashes (and possibly others e.g. TLSH)
> -
>
> Key: NIFI-2747
> URL: https://issues.apache.org/jira/browse/NIFI-2747
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.0.0
>Reporter: Andre F de Miranda
>Assignee: Andre F de Miranda
>
> It would be great if we had a processor to support fuzzy hashing algorithms 
> such as SpamSum/SSDeep



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-2747) Add Processor support SSDeep (SpamSum) hashes (and possibly others e.g. TLSH)

2017-03-24 Thread Andre F de Miranda (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940147#comment-15940147
 ] 

Andre F de Miranda commented on NIFI-2747:
--

Another possible hashing algo is TLSH

Two implementations seem to exist

https://github.com/pombredanne/TLSH-2

which seems to be a fork from a now unavailable port made early 2016

And 

https://github.com/idealista/tlsh

which seems to be an independent implementation also crafted early 2016

Both implementations are ASL 2

> Add Processor support SSDeep (SpamSum) hashes (and possibly others e.g. TLSH)
> -
>
> Key: NIFI-2747
> URL: https://issues.apache.org/jira/browse/NIFI-2747
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.0.0
>Reporter: Andre F de Miranda
>Assignee: Andre F de Miranda
>
> It would be great if we had a processor to support fuzzy hashing algorithms 
> such as SpamSum/SSDeep



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi issue #1603: NIFI-2481 improved content repo archive description in Adm...

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/1603
  
LGTM +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2481) FileSystemRepository: maxArchiveBytes calculated incorrectly

2017-03-24 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940186#comment-15940186
 ] 

ASF GitHub Bot commented on NIFI-2481:
--

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/1603
  
LGTM +1


> FileSystemRepository: maxArchiveBytes calculated incorrectly
> 
>
> Key: NIFI-2481
> URL: https://issues.apache.org/jira/browse/NIFI-2481
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Brandon DeVries
>Assignee: Michael Moser
>Priority: Minor
>
> It appears as though the calculation of maxArchiveBytes in 
> FileSystemRepository.java might be backwards.  The line in question\[1\] is:
> {code}
> final long maxArchiveBytes = (long) (capacity * (1D - (maxArchiveRatio - 
> 0.02)));
> {code}
> if you say capacity = 1000, then you get the following:
> maxArchiveRatio = .1 ==> maxArchiveBytes = 920
> maxArchiveRatio = .5 ==> maxArchiveBytes = 520 
> maxArchiveRatio = .9 ==> maxArchiveBytes = 120
> In other words, as you increase the 
> nifi.content.repository.archive.max.usage.percentage, you would be decreasing 
> the bytes allowed for archiving...
> \[1\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.7.0/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/FileSystemRepository.java#L189



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-2481) FileSystemRepository: maxArchiveBytes calculated incorrectly


[ 
https://issues.apache.org/jira/browse/NIFI-2481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940198#comment-15940198
 ] 

ASF subversion and git services commented on NIFI-2481:
---

Commit 2dd61125d13dd04798663cc8f1ef06436cf7bb16 in nifi's branch 
refs/heads/master from [~boardm26]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=2dd6112 ]

NIFI-2481 improved content repo archive description in Admin Guide

This closes: #1603

Signed-off-by: Andre F de Miranda 


> FileSystemRepository: maxArchiveBytes calculated incorrectly
> 
>
> Key: NIFI-2481
> URL: https://issues.apache.org/jira/browse/NIFI-2481
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Brandon DeVries
>Assignee: Michael Moser
>Priority: Minor
>
> It appears as though the calculation of maxArchiveBytes in 
> FileSystemRepository.java might be backwards.  The line in question\[1\] is:
> {code}
> final long maxArchiveBytes = (long) (capacity * (1D - (maxArchiveRatio - 
> 0.02)));
> {code}
> if you say capacity = 1000, then you get the following:
> maxArchiveRatio = .1 ==> maxArchiveBytes = 920
> maxArchiveRatio = .5 ==> maxArchiveBytes = 520 
> maxArchiveRatio = .9 ==> maxArchiveBytes = 120
> In other words, as you increase the 
> nifi.content.repository.archive.max.usage.percentage, you would be decreasing 
> the bytes allowed for archiving...
> \[1\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.7.0/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/FileSystemRepository.java#L189



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFIREG-2) Design logo for Registry

2017-03-24 Thread Rob Moran (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFIREG-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rob Moran updated NIFIREG-2:

Description: 
The attached image contains variations of the proposed logo design for 
Registry. The points below describe some of the thinking behind it:
* Relationship to NiFi and MiNiFi through the use of the same typeface and use 
of blocks representing bits of data
* For Registry these blocks also represent the storage/organization aspect 
through their even distribution and arrangement
* The 3 gradated blocks across the top – forming the terminal part of a 
lowercase *r* – represent movement (e.g., a versioned flow being saved to NiFi 
or imported to NiFi from the registry)
* The solid gray (in color versions) and outlined block (in one-color versions) 
help with idea of storage as previously mentioned, but also allude to unused 
storage/free space
* The gray block also helps establish the strong diagonal slicing through it 
and the lowest green block. Again this helps with the idea of movement, but 
more so speaks to how Registry operates in the background, tucked away, largely 
unseen by NiFi operators
* A departure from the NiFi color palette signifies how Registry functions more 
as a standalone application

> Design logo for Registry
> 
>
> Key: NIFIREG-2
> URL: https://issues.apache.org/jira/browse/NIFIREG-2
> Project: NiFi Registry
>  Issue Type: Task
>Reporter: Rob Moran
>Assignee: Rob Moran
>Priority: Minor
> Attachments: registry-logo-concept.png
>
>
> The attached image contains variations of the proposed logo design for 
> Registry. The points below describe some of the thinking behind it:
> * Relationship to NiFi and MiNiFi through the use of the same typeface and 
> use of blocks representing bits of data
> * For Registry these blocks also represent the storage/organization aspect 
> through their even distribution and arrangement
> * The 3 gradated blocks across the top – forming the terminal part of a 
> lowercase *r* – represent movement (e.g., a versioned flow being saved to 
> NiFi or imported to NiFi from the registry)
> * The solid gray (in color versions) and outlined block (in one-color 
> versions) help with idea of storage as previously mentioned, but also allude 
> to unused storage/free space
> * The gray block also helps establish the strong diagonal slicing through it 
> and the lowest green block. Again this helps with the idea of movement, but 
> more so speaks to how Registry operates in the background, tucked away, 
> largely unseen by NiFi operators
> * A departure from the NiFi color palette signifies how Registry functions 
> more as a standalone application



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFIREG-2) Design logo for Registry

2017-03-24 Thread Rob Moran (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFIREG-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rob Moran updated NIFIREG-2:

Attachment: registry-logo-concept.png

> Design logo for Registry
> 
>
> Key: NIFIREG-2
> URL: https://issues.apache.org/jira/browse/NIFIREG-2
> Project: NiFi Registry
>  Issue Type: Task
>Reporter: Rob Moran
>Assignee: Rob Moran
>Priority: Minor
> Attachments: registry-logo-concept.png
>
>
> The attached image contains variations of the proposed logo design for 
> Registry. The points below describe some of the thinking behind it:
> * Relationship to NiFi and MiNiFi through the use of the same typeface and 
> use of blocks representing bits of data
> * For Registry these blocks also represent the storage/organization aspect 
> through their even distribution and arrangement
> * The 3 gradated blocks across the top – forming the terminal part of a 
> lowercase *r* – represent movement (e.g., a versioned flow being saved to 
> NiFi or imported to NiFi from the registry)
> * The solid gray (in color versions) and outlined block (in one-color 
> versions) help with idea of storage as previously mentioned, but also allude 
> to unused storage/free space
> * The gray block also helps establish the strong diagonal slicing through it 
> and the lowest green block. Again this helps with the idea of movement, but 
> more so speaks to how Registry operates in the background, tucked away, 
> largely unseen by NiFi operators
> * A departure from the NiFi color palette signifies how Registry functions 
> more as a standalone application



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1603: NIFI-2481 improved content repo archive description...

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1603


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2481) FileSystemRepository: maxArchiveBytes calculated incorrectly


[ 
https://issues.apache.org/jira/browse/NIFI-2481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940200#comment-15940200
 ] 

ASF GitHub Bot commented on NIFI-2481:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1603


> FileSystemRepository: maxArchiveBytes calculated incorrectly
> 
>
> Key: NIFI-2481
> URL: https://issues.apache.org/jira/browse/NIFI-2481
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Brandon DeVries
>Assignee: Michael Moser
>Priority: Minor
>
> It appears as though the calculation of maxArchiveBytes in 
> FileSystemRepository.java might be backwards.  The line in question\[1\] is:
> {code}
> final long maxArchiveBytes = (long) (capacity * (1D - (maxArchiveRatio - 
> 0.02)));
> {code}
> if you say capacity = 1000, then you get the following:
> maxArchiveRatio = .1 ==> maxArchiveBytes = 920
> maxArchiveRatio = .5 ==> maxArchiveBytes = 520 
> maxArchiveRatio = .9 ==> maxArchiveBytes = 120
> In other words, as you increase the 
> nifi.content.repository.archive.max.usage.percentage, you would be decreasing 
> the bytes allowed for archiving...
> \[1\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.7.0/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/FileSystemRepository.java#L189



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi issue #1324: NIFI-3166: Fix SocketRemoteSiteListener NPE.

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/1324
  
@ijokarumawak merge conflict, mind rebasing?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3166) SocketRemoteSiteListener NPE when calling NifiProperties.getRemoteInputHttpPort


[ 
https://issues.apache.org/jira/browse/NIFI-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940205#comment-15940205
 ] 

ASF GitHub Bot commented on NIFI-3166:
--

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/1324
  
@ijokarumawak merge conflict, mind rebasing?


> SocketRemoteSiteListener NPE when calling 
> NifiProperties.getRemoteInputHttpPort
> ---
>
> Key: NIFI-3166
> URL: https://issues.apache.org/jira/browse/NIFI-3166
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.1.0
>Reporter: Andre F de Miranda
>Assignee: Koji Kawamura
>
> When using the following properties:
> nifi.remote.input.host=node1.textbed.internal
> nifi.remote.input.secure=true
> nifi.remote.input.socket.port=54321
> nifi.remote.input.http.enabled=false
> nifi.remote.input.http.transaction.ttl=30 sec
> I hit
> {code}
> 2016-12-09 00:15:26,456 ERROR [Site-to-Site Worker Thread-145] 
> o.a.nifi.remote.SocketRemoteSiteListener
> java.lang.NullPointerException: null
> at 
> org.apache.nifi.remote.SocketRemoteSiteListener$1$1.run(SocketRemoteSiteListener.java:280)
>  ~[nifi-site-to-site-1.1.0.jar:1.1.0]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1473: NIFI-940 - Exposes filename attribute to DFM on Put...

Github user trixpan closed the pull request at:

https://github.com/apache/nifi/pull/1473


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #1473: NIFI-940 - Exposes filename attribute to DFM on PutFile, P...

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/1473
  
closing following @joewitt JIRA comments.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-940) Expose 'filename' as a Property on PutFile, PutHDFS, PutFTP, PutSFTP, and other Put-esque processors


[ 
https://issues.apache.org/jira/browse/NIFI-940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940210#comment-15940210
 ] 

ASF GitHub Bot commented on NIFI-940:
-

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/1473
  
closing following @joewitt JIRA comments.


> Expose 'filename' as a Property on PutFile, PutHDFS, PutFTP, PutSFTP, and 
> other Put-esque processors
> 
>
> Key: NIFI-940
> URL: https://issues.apache.org/jira/browse/NIFI-940
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 0.3.0
>Reporter: Michael Kobit
>Assignee: Andre F de Miranda
>
> Allow a NFM to set the filename to save files as. Should support Expression 
> language, and default to '${filename}' 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-940) Expose 'filename' as a Property on PutFile, PutHDFS, PutFTP, PutSFTP, and other Put-esque processors


[ 
https://issues.apache.org/jira/browse/NIFI-940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940211#comment-15940211
 ] 

ASF GitHub Bot commented on NIFI-940:
-

Github user trixpan closed the pull request at:

https://github.com/apache/nifi/pull/1473


> Expose 'filename' as a Property on PutFile, PutHDFS, PutFTP, PutSFTP, and 
> other Put-esque processors
> 
>
> Key: NIFI-940
> URL: https://issues.apache.org/jira/browse/NIFI-940
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 0.3.0
>Reporter: Michael Kobit
>Assignee: Andre F de Miranda
>
> Allow a NFM to set the filename to save files as. Should support Expression 
> language, and default to '${filename}' 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (NIFI-940) Expose 'filename' as a Property on PutFile, PutHDFS, PutFTP, PutSFTP, and other Put-esque processors

2017-03-24 Thread Andre F de Miranda (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andre F de Miranda resolved NIFI-940.
-
Resolution: Workaround

workaround (use update attribute) was recommended.

> Expose 'filename' as a Property on PutFile, PutHDFS, PutFTP, PutSFTP, and 
> other Put-esque processors
> 
>
> Key: NIFI-940
> URL: https://issues.apache.org/jira/browse/NIFI-940
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 0.3.0
>Reporter: Michael Kobit
>Assignee: Andre F de Miranda
>
> Allow a NFM to set the filename to save files as. Should support Expression 
> language, and default to '${filename}' 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-79) Add support for bulletin level 'NONE'

2017-03-24 Thread Andre F de Miranda (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940217#comment-15940217
 ] 

Andre F de Miranda commented on NIFI-79:


I bet it is so simple... but yet so amazingly scary journey into the depths of 
NiFi... :-)

> Add support for bulletin level 'NONE'
> -
>
> Key: NIFI-79
> URL: https://issues.apache.org/jira/browse/NIFI-79
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework, Core UI
>Reporter: Joseph Witt
>Assignee: Andre F de Miranda
>Priority: Trivial
>
> sometimes you just don't care what a processor has to say



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3644) Add DetectDuplicateUsingHBase processor

2017-03-24 Thread Joseph Witt (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940247#comment-15940247
 ] 

Joseph Witt commented on NIFI-3644:
---

Bjorn,

We can add you to the contributors list in JIRA so that you can assign items to 
yourself.  However, in the meantime you can definitely contribute and work on 
tasks without this.  For this concept please note you should only need to 
create an implementation of the DistributedMapCache which is backed by HBase 
rather than a new processor.  DetectDuplicate can use any implementation of 
that interface by design.

Thanks
Joe

> Add DetectDuplicateUsingHBase processor
> ---
>
> Key: NIFI-3644
> URL: https://issues.apache.org/jira/browse/NIFI-3644
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Bjorn Olsen
>Priority: Minor
>
> The DetectDuplicate processor makes use of a distributed map cache for 
> maintaining a list of unique file identifiers (such as hashes).
> The distributed map cache functionality could be provided by an HBase table, 
> which then allows for reliably storing a huge volume of file identifiers and 
> auditing information. The downside of this approach is of course that HBase 
> is required.
> Storing the unique file identifiers in a reliable, query-able manner along 
> with some audit information is of benefit to several use cases.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi issue #1615: NIFI-3639 - Implemented HBase Get functionality for HBaseC...

2017-03-24 Thread bbende

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/1615
  
@baolsen Thanks for the info. I believe only you can close the pull request 
on your end.

As far as the automated checks, you can ignore appveryor, but the travis 
build should be passing as of yesterday. Looking at the travis logs for your 
build it shows:

[INFO] --- maven-checkstyle-plugin:2.15:check (check-style) @ 
nifi-hbase-client-service-api ---
[WARNING] src/main/java/org/apache/nifi/hbase/HBaseClientService.java[131] 
(regexp) RegexpSinglelineJava: Line has trailing whitespace.

You can detect this locally by running "mvn clean install -Pcontrib-check" 
before submitting anything. The contrib-check will fail if anything doesn't 
pass checkstyles or the RAT licensing check.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3639) Add HBase Get to HBase_1_1_2_ClientService


[ 
https://issues.apache.org/jira/browse/NIFI-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940254#comment-15940254
 ] 

ASF GitHub Bot commented on NIFI-3639:
--

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/1615
  
@baolsen Thanks for the info. I believe only you can close the pull request 
on your end.

As far as the automated checks, you can ignore appveryor, but the travis 
build should be passing as of yesterday. Looking at the travis logs for your 
build it shows:

[INFO] --- maven-checkstyle-plugin:2.15:check (check-style) @ 
nifi-hbase-client-service-api ---
[WARNING] src/main/java/org/apache/nifi/hbase/HBaseClientService.java[131] 
(regexp) RegexpSinglelineJava: Line has trailing whitespace.

You can detect this locally by running "mvn clean install -Pcontrib-check" 
before submitting anything. The contrib-check will fail if anything doesn't 
pass checkstyles or the RAT licensing check.



> Add HBase Get to HBase_1_1_2_ClientService
> --
>
> Key: NIFI-3639
> URL: https://issues.apache.org/jira/browse/NIFI-3639
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Bjorn Olsen
>Priority: Trivial
>
> Enhance HBase_1_1_2_ClientService and API to provide HBase Get functionality. 
> Currently only Put and Scan are supported.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFIREG-2) Design logo for Registry

2017-03-24 Thread Joseph Witt (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFIREG-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940265#comment-15940265
 ] 

Joseph Witt commented on NIFIREG-2:
---

hey rob thanks for going after this and for sharing your thoughts behind it.  
It looks nice and I am overall favorable to it.  Some initial thoughts I had:
- I'd like to see some connection to the nifi/minifi theming.  Whether that is 
via color or inclusion of some 'flow/water/fluid' like visuals is ok.
- the registry concept while it is a standalone app it is entirely centered 
around providing central storage of nifi/minifi specific artifacts such as 
extensions, flows, and variables.  If there was some way to capture that ...  
With the registry we're not talking about the flow so much as we're talking 
about what controls/defines the flow so perhaps pipes in the shape of an R?

Anyway all that said my brain totally does not afford the same awesomeness 
yours does when it comes to visualizing concepts.  I definitely defer to your 
thoughts on this just wanted to share some of my old school nifi thinking.

> Design logo for Registry
> 
>
> Key: NIFIREG-2
> URL: https://issues.apache.org/jira/browse/NIFIREG-2
> Project: NiFi Registry
>  Issue Type: Task
>Reporter: Rob Moran
>Assignee: Rob Moran
>Priority: Minor
> Attachments: registry-logo-concept.png
>
>
> The attached image contains variations of the proposed logo design for 
> Registry. The points below describe some of the thinking behind it:
> * Relationship to NiFi and MiNiFi through the use of the same typeface and 
> use of blocks representing bits of data
> * For Registry these blocks also represent the storage/organization aspect 
> through their even distribution and arrangement
> * The 3 gradated blocks across the top – forming the terminal part of a 
> lowercase *r* – represent movement (e.g., a versioned flow being saved to 
> NiFi or imported to NiFi from the registry)
> * The solid gray (in color versions) and outlined block (in one-color 
> versions) help with idea of storage as previously mentioned, but also allude 
> to unused storage/free space
> * The gray block also helps establish the strong diagonal slicing through it 
> and the lowest green block. Again this helps with the idea of movement, but 
> more so speaks to how Registry operates in the background, tucked away, 
> largely unseen by NiFi operators
> * A departure from the NiFi color palette signifies how Registry functions 
> more as a standalone application



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-2481) FileSystemRepository: explain maxArchiveBytes calculation better


 [ 
https://issues.apache.org/jira/browse/NIFI-2481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Moser updated NIFI-2481:

Summary: FileSystemRepository: explain maxArchiveBytes calculation better  
(was: FileSystemRepository: maxArchiveBytes calculated incorrectly)

> FileSystemRepository: explain maxArchiveBytes calculation better
> 
>
> Key: NIFI-2481
> URL: https://issues.apache.org/jira/browse/NIFI-2481
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Brandon DeVries
>Assignee: Michael Moser
>Priority: Minor
>
> It appears as though the calculation of maxArchiveBytes in 
> FileSystemRepository.java might be backwards.  The line in question\[1\] is:
> {code}
> final long maxArchiveBytes = (long) (capacity * (1D - (maxArchiveRatio - 
> 0.02)));
> {code}
> if you say capacity = 1000, then you get the following:
> maxArchiveRatio = .1 ==> maxArchiveBytes = 920
> maxArchiveRatio = .5 ==> maxArchiveBytes = 520 
> maxArchiveRatio = .9 ==> maxArchiveBytes = 120
> In other words, as you increase the 
> nifi.content.repository.archive.max.usage.percentage, you would be decreasing 
> the bytes allowed for archiving...
> \[1\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.7.0/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/FileSystemRepository.java#L189



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-2481) FileSystemRepository: explain maxArchiveBytes calculation better


 [ 
https://issues.apache.org/jira/browse/NIFI-2481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Moser updated NIFI-2481:

   Resolution: Fixed
Fix Version/s: 1.2.0
   Status: Resolved  (was: Patch Available)

> FileSystemRepository: explain maxArchiveBytes calculation better
> 
>
> Key: NIFI-2481
> URL: https://issues.apache.org/jira/browse/NIFI-2481
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Brandon DeVries
>Assignee: Michael Moser
>Priority: Minor
> Fix For: 1.2.0
>
>
> It appears as though the calculation of maxArchiveBytes in 
> FileSystemRepository.java might be backwards.  The line in question\[1\] is:
> {code}
> final long maxArchiveBytes = (long) (capacity * (1D - (maxArchiveRatio - 
> 0.02)));
> {code}
> if you say capacity = 1000, then you get the following:
> maxArchiveRatio = .1 ==> maxArchiveBytes = 920
> maxArchiveRatio = .5 ==> maxArchiveBytes = 520 
> maxArchiveRatio = .9 ==> maxArchiveBytes = 120
> In other words, as you increase the 
> nifi.content.repository.archive.max.usage.percentage, you would be decreasing 
> the bytes allowed for archiving...
> \[1\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.7.0/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/FileSystemRepository.java#L189



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFIREG-2) Design logo for Registry

2017-03-24 Thread Bryan Bende (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFIREG-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940318#comment-15940318
 ] 

Bryan Bende commented on NIFIREG-2:
---

[~rmoran] thanks for putting this together! definitely cool to start seeing the 
registry take form.

My only thought is that we might want the color scheme to be consistent with 
the existing NiFi/MiNiFi colors, but if others are strongly in favor of a 
different color scheme then I can get on-board. 

> Design logo for Registry
> 
>
> Key: NIFIREG-2
> URL: https://issues.apache.org/jira/browse/NIFIREG-2
> Project: NiFi Registry
>  Issue Type: Task
>Reporter: Rob Moran
>Assignee: Rob Moran
>Priority: Minor
> Attachments: registry-logo-concept.png
>
>
> The attached image contains variations of the proposed logo design for 
> Registry. The points below describe some of the thinking behind it:
> * Relationship to NiFi and MiNiFi through the use of the same typeface and 
> use of blocks representing bits of data
> * For Registry these blocks also represent the storage/organization aspect 
> through their even distribution and arrangement
> * The 3 gradated blocks across the top – forming the terminal part of a 
> lowercase *r* – represent movement (e.g., a versioned flow being saved to 
> NiFi or imported to NiFi from the registry)
> * The solid gray (in color versions) and outlined block (in one-color 
> versions) help with idea of storage as previously mentioned, but also allude 
> to unused storage/free space
> * The gray block also helps establish the strong diagonal slicing through it 
> and the lowest green block. Again this helps with the idea of movement, but 
> more so speaks to how Registry operates in the background, tucked away, 
> largely unseen by NiFi operators
> * A departure from the NiFi color palette signifies how Registry functions 
> more as a standalone application



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFIREG-2) Design logo for Registry

2017-03-24 Thread Rob Moran (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFIREG-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940341#comment-15940341
 ] 

Rob Moran commented on NIFIREG-2:
-

Great feedback so far; all very valid points – thanks! I will continue to push 
the design and post updates.

> Design logo for Registry
> 
>
> Key: NIFIREG-2
> URL: https://issues.apache.org/jira/browse/NIFIREG-2
> Project: NiFi Registry
>  Issue Type: Task
>Reporter: Rob Moran
>Assignee: Rob Moran
>Priority: Minor
> Attachments: registry-logo-concept.png
>
>
> The attached image contains variations of the proposed logo design for 
> Registry. The points below describe some of the thinking behind it:
> * Relationship to NiFi and MiNiFi through the use of the same typeface and 
> use of blocks representing bits of data
> * For Registry these blocks also represent the storage/organization aspect 
> through their even distribution and arrangement
> * The 3 gradated blocks across the top – forming the terminal part of a 
> lowercase *r* – represent movement (e.g., a versioned flow being saved to 
> NiFi or imported to NiFi from the registry)
> * The solid gray (in color versions) and outlined block (in one-color 
> versions) help with idea of storage as previously mentioned, but also allude 
> to unused storage/free space
> * The gray block also helps establish the strong diagonal slicing through it 
> and the lowest green block. Again this helps with the idea of movement, but 
> more so speaks to how Registry operates in the background, tucked away, 
> largely unseen by NiFi operators
> * A departure from the NiFi color palette signifies how Registry functions 
> more as a standalone application



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi issue #1615: NIFI-3639 - Implemented HBase Get functionality for HBaseC...

2017-03-24 Thread baolsen

Github user baolsen commented on the issue:

https://github.com/apache/nifi/pull/1615
  
Great, thanks for all the info. 
Will close this PR meanwhile.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3639) Add HBase Get to HBase_1_1_2_ClientService


[ 
https://issues.apache.org/jira/browse/NIFI-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940373#comment-15940373
 ] 

ASF GitHub Bot commented on NIFI-3639:
--

Github user baolsen commented on the issue:

https://github.com/apache/nifi/pull/1615
  
Great, thanks for all the info. 
Will close this PR meanwhile.


> Add HBase Get to HBase_1_1_2_ClientService
> --
>
> Key: NIFI-3639
> URL: https://issues.apache.org/jira/browse/NIFI-3639
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Bjorn Olsen
>Priority: Trivial
>
> Enhance HBase_1_1_2_ClientService and API to provide HBase Get functionality. 
> Currently only Put and Scan are supported.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1615: NIFI-3639 - Implemented HBase Get functionality for...

2017-03-24 Thread baolsen

Github user baolsen closed the pull request at:

https://github.com/apache/nifi/pull/1615


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3639) Add HBase Get to HBase_1_1_2_ClientService


[ 
https://issues.apache.org/jira/browse/NIFI-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940374#comment-15940374
 ] 

ASF GitHub Bot commented on NIFI-3639:
--

Github user baolsen closed the pull request at:

https://github.com/apache/nifi/pull/1615


> Add HBase Get to HBase_1_1_2_ClientService
> --
>
> Key: NIFI-3639
> URL: https://issues.apache.org/jira/browse/NIFI-3639
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Bjorn Olsen
>Priority: Trivial
>
> Enhance HBase_1_1_2_ClientService and API to provide HBase Get functionality. 
> Currently only Put and Scan are supported.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3313) First deployment of NiFi can hang on VMs without sufficient entropy if using /dev/random


[ 
https://issues.apache.org/jira/browse/NIFI-3313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940402#comment-15940402
 ] 

Michael Moser commented on NIFI-3313:
-

I was seeing an extra 30 to 120 seconds on startup, but WHOOPS I got my 
environments confused.  I've been switching between 0.x (Java 7) and 1.x (java 
8) a lot lately, and I used {{file:/dev/urandom}} in my 0.x bootstrap.conf.  
When I added the extra {{/.}} to my 0.x bootstrap.conf and the extra startup 
delay was gone, I came here and commented.  Sorry!

> First deployment of NiFi can hang on VMs without sufficient entropy if using 
> /dev/random
> 
>
> Key: NIFI-3313
> URL: https://issues.apache.org/jira/browse/NIFI-3313
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.1.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Critical
>  Labels: entropy, security, virtual-machine
> Fix For: 1.2.0
>
>
> h1. Analysis of Issue
> h2. Statement of Problem:
> NiFi deployed on headless VM (little user interaction by way of keyboard and 
> mouse I/O) can take 5-10 minutes (reported) to start up. User reports this 
> occurs on a "secure" cluster. Further examination is required to determine 
> which specific process requires the large amount of random input (no steps to 
> reproduce, configuration files, logs, or VM environment information 
> provided). 
> h2. Context
> The likely cause of this issue is that a process is attempting to read from 
> _/dev/random_, a \*nix "device" providing a pseudo-random number generator 
> (PRNG). Also available is _/dev/urandom_, a related PRNG. Despite common 
> misperceptions, _/dev/urandom_ is not "less-secure" than _/dev/random_ for 
> all general use cases. _/dev/random_ blocks if the entropy *estimate* (a 
> "guess" of the existing entropy introduced into the pool) is lower than the 
> amount of random data requested by the caller. In contrast, _/dev/urandom_ 
> does not block, but provides the output of the same cryptographically-secure 
> PRNG (CSPRNG) that _/dev/random_ reads from \[myths\]. After as little as 256 
> bytes of initial seeding, accessing _/dev/random_ and _/dev/urandom_ are 
> functionally equivalent, as the long period of random data generated will not 
> require re-seeding before sufficient entropy can be provided again. 
> As mentioned earlier, further examination is required to determine if the 
> process requiring random input occurs at application boot or only at 
> "machine" (hardware or VM) boot. On the first deployment of the system with 
> certificates, the certificate generation process will require substantial 
> random input. However, on application launch and connection to a cluster, 
> even the TLS/SSL protocol requires some amount of random input. 
> h2. Proposed Solutions
> h3. rngd
> A software toolset for accessing dedicated hardware PRNG (*true* RNG, or 
> TRNG) called _rng-tools_ \[rngtools\] exists for Linux. Specialized hardware, 
> as well as Intel chips from IvyBridge and on (2012), can provide 
> hardware-generated random input to the kernel. Using the daemon _rngd_ to 
> seed the _/dev/random_ and _/dev/urandom_ entropy pool is the simplest 
> solution. 
> *Note: Do not use _/dev/urandom_ to seed _/dev/random_ using _rngd_. This is 
> like running a garden hose from a car's exhaust back into its gas tank and 
> trying to drive.*
> h3. Instruct Java to use /dev/urandom
> The Java Runtime Environment (JRE) can be instructed to use _/dev/urandom_ 
> for all invocations of {{SecureRandom}}, either on a per-Java process basis 
> \[jdk-urandom\] or in the JVM configuration \[oracle-urandom\], which means 
> it will not block on server startup. The NiFi {{bootstrap.conf}} file can be 
> modified to contain an additional Java argument directing the JVM to use 
> _/dev/urandom_. 
> h2. Other Solutions
> h3. Entropy Gathering Tools
> Tools to gather entropy from non-standard sources (audio card noise, video 
> capture from webcams, etc.) have been developed such as audio-entropyd 
> \[wagner\], but these tools are not verified or well-examined -- usually when 
> tested, they are only tested for the strength of their PRNG, not the ability 
> of the tool to capture entropy and generate sufficiently random data 
> unavailable to an attacker who may be able to determine the internal state. 
> h3. haveged
> A solution has been proposed to use {{havaged}} \[haveged\], a user-space 
> daemon relying on the HAVEGE (HArdware Volatile Entropy Gathering and 
> Expansion) construct to continually increase the entropy on the system, 
> allowing _/dev/random_ to run without blocking. 
> However, on further investigation, multiple sources indica

[jira] [Commented] (NIFI-3644) Add DetectDuplicateUsingHBase processor

2017-03-24 Thread Bjorn Olsen (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940467#comment-15940467
 ] 

Bjorn Olsen commented on NIFI-3644:
---

Hi Joe

Thanks for the suggestion, I hadn't considered writing an HBase version of 
DistributedMapCache. 

I've already written my own DetectDuplicateUsingHBase processor today, as I 
needed something that was quick to develop.

Code here, much copy-pasta from DetectDuplicate:
https://github.com/baolsen/nifi/blob/DetectDuplicateUsingHBase/nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DetectDuplicateUsingHBase.java

It seems that implementing an HBase-based DistributedMapCache is more complex, 
but more reusable. 
Do you have any suggestions for documentation for this sort of thing?

Lastly, do you think it is worth including DetectDuplicateUsingHBase or rather 
wait for a more reusable option?

I'm a bit tight on time, and Java and NiFi are both new to me.
Meanwhile I can keep DetectDuplicateUsingHBase for my own use, so no worries 
there.

> Add DetectDuplicateUsingHBase processor
> ---
>
> Key: NIFI-3644
> URL: https://issues.apache.org/jira/browse/NIFI-3644
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Bjorn Olsen
>Priority: Minor
>
> The DetectDuplicate processor makes use of a distributed map cache for 
> maintaining a list of unique file identifiers (such as hashes).
> The distributed map cache functionality could be provided by an HBase table, 
> which then allows for reliably storing a huge volume of file identifiers and 
> auditing information. The downside of this approach is of course that HBase 
> is required.
> Storing the unique file identifiers in a reliable, query-able manner along 
> with some audit information is of benefit to several use cases.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (NIFI-3644) Add DetectDuplicateUsingHBase processor

2017-03-24 Thread Bjorn Olsen (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940467#comment-15940467
 ] 

Bjorn Olsen edited comment on NIFI-3644 at 3/24/17 2:39 PM:


Hi Joe

Thanks for the suggestion, I hadn't considered writing an HBase version of 
DistributedMapCache. 

I've already written my own DetectDuplicateUsingHBase processor today, as I 
needed something that was quick to develop.

Working code here, much copy-pasta from DetectDuplicate:
https://github.com/baolsen/nifi/blob/DetectDuplicateUsingHBase/nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DetectDuplicateUsingHBase.java

It seems that implementing an HBase-based DistributedMapCache is more complex, 
but more reusable. 
Do you have any suggestions for documentation for this sort of thing?

Lastly, do you think it is worth including DetectDuplicateUsingHBase or rather 
wait for a more reusable option?

I'm a bit tight on time, and Java and NiFi are both new to me.
Meanwhile I can keep DetectDuplicateUsingHBase for my own use, so no worries 
there.


was (Author: bjorn.ols...@gmail.com):
Hi Joe

Thanks for the suggestion, I hadn't considered writing an HBase version of 
DistributedMapCache. 

I've already written my own DetectDuplicateUsingHBase processor today, as I 
needed something that was quick to develop.

Code here, much copy-pasta from DetectDuplicate:
https://github.com/baolsen/nifi/blob/DetectDuplicateUsingHBase/nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DetectDuplicateUsingHBase.java

It seems that implementing an HBase-based DistributedMapCache is more complex, 
but more reusable. 
Do you have any suggestions for documentation for this sort of thing?

Lastly, do you think it is worth including DetectDuplicateUsingHBase or rather 
wait for a more reusable option?

I'm a bit tight on time, and Java and NiFi are both new to me.
Meanwhile I can keep DetectDuplicateUsingHBase for my own use, so no worries 
there.

> Add DetectDuplicateUsingHBase processor
> ---
>
> Key: NIFI-3644
> URL: https://issues.apache.org/jira/browse/NIFI-3644
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Bjorn Olsen
>Priority: Minor
>
> The DetectDuplicate processor makes use of a distributed map cache for 
> maintaining a list of unique file identifiers (such as hashes).
> The distributed map cache functionality could be provided by an HBase table, 
> which then allows for reliably storing a huge volume of file identifiers and 
> auditing information. The downside of this approach is of course that HBase 
> is required.
> Storing the unique file identifiers in a reliable, query-able manner along 
> with some audit information is of benefit to several use cases.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (NIFI-3644) Add DetectDuplicateUsingHBase processor

2017-03-24 Thread Bjorn Olsen (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940467#comment-15940467
 ] 

Bjorn Olsen edited comment on NIFI-3644 at 3/24/17 2:41 PM:


Hi Joe

Thanks for the suggestion, I hadn't considered writing an HBase version of 
DistributedMapCache. 

I've already written my own DetectDuplicateUsingHBase processor today, as I 
needed something that was quick to develop.

Working code here, uses HBaseClientService & much copy-pasta from 
DetectDuplicate:
https://github.com/baolsen/nifi/blob/DetectDuplicateUsingHBase/nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DetectDuplicateUsingHBase.java

It seems that implementing an HBase-based DistributedMapCache is more complex, 
but more reusable. 
Do you have any suggestions for documentation for this sort of thing?

Lastly, do you think it is worth including DetectDuplicateUsingHBase or rather 
wait for a more reusable option?

I'm a bit tight on time, and Java and NiFi are both new to me.
Meanwhile I can keep DetectDuplicateUsingHBase for my own use, so no worries 
there.


was (Author: bjorn.ols...@gmail.com):
Hi Joe

Thanks for the suggestion, I hadn't considered writing an HBase version of 
DistributedMapCache. 

I've already written my own DetectDuplicateUsingHBase processor today, as I 
needed something that was quick to develop.

Working code here, much copy-pasta from DetectDuplicate:
https://github.com/baolsen/nifi/blob/DetectDuplicateUsingHBase/nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DetectDuplicateUsingHBase.java

It seems that implementing an HBase-based DistributedMapCache is more complex, 
but more reusable. 
Do you have any suggestions for documentation for this sort of thing?

Lastly, do you think it is worth including DetectDuplicateUsingHBase or rather 
wait for a more reusable option?

I'm a bit tight on time, and Java and NiFi are both new to me.
Meanwhile I can keep DetectDuplicateUsingHBase for my own use, so no worries 
there.

> Add DetectDuplicateUsingHBase processor
> ---
>
> Key: NIFI-3644
> URL: https://issues.apache.org/jira/browse/NIFI-3644
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Bjorn Olsen
>Priority: Minor
>
> The DetectDuplicate processor makes use of a distributed map cache for 
> maintaining a list of unique file identifiers (such as hashes).
> The distributed map cache functionality could be provided by an HBase table, 
> which then allows for reliably storing a huge volume of file identifiers and 
> auditing information. The downside of this approach is of course that HBase 
> is required.
> Storing the unique file identifiers in a reliable, query-able manner along 
> with some audit information is of benefit to several use cases.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-3583) PutFile should attempt to replace files atomically

2017-03-24 Thread Joseph Niemiec (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-3583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Niemiec updated NIFI-3583:
-
Status: Patch Available  (was: Open)

> PutFile should attempt to replace files atomically
> --
>
> Key: NIFI-3583
> URL: https://issues.apache.org/jira/browse/NIFI-3583
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.1.1, 0.7.1
> Environment: On Windows, NIO2's Files.move operation may not allow 
> atomic file replacement, in which case it would have to fall back to the 
> existing method.
>Reporter: Eugene Hart
>Assignee: Gray Gwizdz
>Priority: Trivial
> Attachments: 
> 0001-NIFI-3853-PutFile-should-attempt-to-replace-files-at.patch
>
>
> A PutFile configured with the "replace" conflict resolution strategy will 
> first delete any existing file and then move the new temporary file into 
> place. This causes there to be a small period of time where the file does not 
> exist at all. This could cause problems with other processes relying on this 
> file to always exist.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-1352) Add ability to add a zip comment in MergeContent

2017-03-24 Thread Joseph Niemiec (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-1352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Niemiec updated NIFI-1352:
-
Affects Version/s: 1.2.0
   Status: Patch Available  (was: Open)

> Add ability to add a zip comment in MergeContent
> 
>
> Key: NIFI-1352
> URL: https://issues.apache.org/jira/browse/NIFI-1352
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.2.0
>Reporter: Brandon DeVries
>Assignee: Gray Gwizdz
>Priority: Trivial
> Attachments: 
> 0001-NIFI-1352-Add-ability-to-add-a-zip-comment-in-MergeC.patch
>
>
> The Zip format includes a file comment, which can be written with a 
> ZipOutputStream \[1\].  Modify MergeContent to optionally include a comment 
> (possibly via a "zip.comment" attribute or something...).
> \[1\] 
> https://docs.oracle.com/javase/7/docs/api/java/util/zip/ZipOutputStream.html#setComment%28java.lang.String%29



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi issue #1618: NIFI-3413: Add GetChangeDataCaptureMySQL processor

Github user phrocker commented on the issue:

https://github.com/apache/nifi/pull/1618
  
@mattyb149  I've left some minor comments. I'm going to pull this down and 
test this. I'm super excited. Looks cool. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3413) Implement a GetChangeDataCapture processor


[ 
https://issues.apache.org/jira/browse/NIFI-3413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940497#comment-15940497
 ] 

ASF GitHub Bot commented on NIFI-3413:
--

Github user phrocker commented on the issue:

https://github.com/apache/nifi/pull/1618
  
@mattyb149  I've left some minor comments. I'm going to pull this down and 
test this. I'm super excited. Looks cool. 


> Implement a GetChangeDataCapture processor
> --
>
> Key: NIFI-3413
> URL: https://issues.apache.org/jira/browse/NIFI-3413
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Database systems such as MySQL, Oracle, and SQL Server allow access to their 
> transactional logs and such, in order for external clients to have a "change 
> data capture" (CDC) capability. I propose a GetChangeDataCapture processor to 
> enable this in NiFi.
> The processor would be configured with a DBCPConnectionPool controller 
> service, as well as a Database Type property (similar to the one in 
> QueryDatabaseTable) for database-specific handling. Additional properties 
> might include the CDC table name, etc.  Additional database-specific 
> properties could be handled using dynamic properties (and the documentation 
> should reflect this).
> The processor would accept no incoming connections (it is a "Get" or source 
> processor), would be intended to run on the primary node only as a single 
> threaded processor, and would generate a flow file for each operation 
> (INSERT, UPDATE, DELETE, e,g,) in one or some number of formats (JSON, e.g.). 
> The flow files would be transferred in time order (to enable a replication 
> solution, for example), perhaps with some auto-incrementing attribute to also 
> indicate order if need be.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi-minifi-cpp pull request #67: Ios

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi-cpp/pull/67


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp issue #69: MINIFI-225: Add Linter for Google style guide

Github user apiri commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/69
  
reviewing


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1585: NIFI-3380 Support Multiple Versions of the Same Com...

2017-03-24 Thread ASF subversion and git services (JIRA)

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1585


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3380) Multiple Versions of the Same Component

[
https://issues.apache.org/jira/browse/NIFI-3380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940512#comment-15940512
]

ASF subversion and git services commented on NIFI-3380:
---

Commit d90cf846b9dcd49acdeac7bb5fcd13ee80dcdba6 in nifi's branch
refs/heads/master from [~bbende]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=d90cf84 ]

NIFI-3380 Bumping NAR plugin to 1.2.0-SNAPSHOT development to leverage changes
from master, adding buildnumber-maven-plugin to nifi-nar-bundles to properly
set build info in MANIFEST of NARs
- Refactoring NarDetails to include all info from MANIFEST
- Adding the concept of a Bundle and refactoring NarClassLoaders to pass
Bundles to ExtensionManager
- Adding logic to fail start-up when multiple NARs with same coordinates exist,
moving Bundle classes to framework API
- Refactoring bundle API to classes and creating BundleCoordinate
- Updating FlowController to use BundleCoordinate

- Updating the UI and DTO model to support showing bundle details that loaded
an extension type.
- Adding bundle details for processor canvas node, processor dialogs,
controller service dialogs, and reporting task dialogs.
- Updating the formating of the bundle coordinates.
- Addressing text overflow in the configuration/details dialog.
- Fixing self referencing functions.
- Updating extension UI mapping to incorporate bundle coordinates.
- Discovering custom UIs through the supplied bundles.
- Adding verification methods for creating extensions through the rest api.
- Only returning extensions that are common amongst all nodes.
- Rendering the ghost processors using a dotted border.
- Adding bundle details to the flow.xml.
- Loading NiFi build and version details from the framework NAR.
- Removing properties for build and version details.
- Wiring together front end and back end changes.
- Including bundle coordinates in the component data model.
- Wiring together component data model and flow.xml.
- Addressing issue when resolve unvesioned dependent NARs.

Updating unit tests to pass based on framework changes
- Fixing logging of extension types during start up

- Allowing the application to start if there is a compatible bundle found. -
Reporting missing bundle when the a compatible bundle is not found. - Fixing
table height in new component dialogs.

Fixing chechstyle error and increasing test timeout for
TestStandardControllerServiceProvider
- Adding ability to change processor type at runtime
- Adding backend code to change type for controller services

- Cleaning up instance classloaders for temp components.
- Creating a dialog for changing the version of a component.
- Updating the formatting of the component type and bundle throughout.
- Updating the new component dialogs to support selecting source group.
- Cleaning up new component dialogs.
- Cleaning up documentation in the cluster node endpoint.

Adding missing include in nifi-web-ui pom compressor plugin
- Refactoring so ConfigurableComponent provides getLogger() and so the nodes
provide the ConfigurableComponent
- Creating LoggableComponent to pass around the component, logger, and
coordinate with in the framework

- Finishing clean up following rebase.

Calling lifecycle methods for add and remove when changing versions of a
component
- Introducing verifyCanUpdateBundle(coordinate) to ConfiguredComponent, and
adding unit tests

- Ensuring documentation is available for all components. Including those of
the same type that are loaded from different bundles.

Adding lookup from ClassLoader to Bundle, adding fix for instance class loading
to include all parent NARs, and adding additional unit tests for FlowController
- Adding validation to ensure referenced controller services implement the
required API
- Fixing template instantiation to look up compatible bundle

- Requiring services/reporting tasks to be disabled/stopped.
- Only supporting a change version option when the item has multiple versions
available.
- Limiting the possible new controller services to the applicable API version.
- Showing the implemented API versions for Controller Services.
- Updating the property descriptor tooltip to indicate the required service
requirements.
- Introducing version based sorting in the new component dialog, change version
dialog, and new controller service dialog.
- Addressing remainder of the issues from recent rebase.

Ensuring bundles have been added to the flow before proposing a flow, and
incorporating bundle information into flow fingerprinting
- Refactoring the way missing bundles work to retain the desired bundle if
available
- Fixing logger.isDebugEnabled to be logger.isTraceEnabled

- Auditing when user changes the bundle. - Ensuring bundle details are present
in templates.

Moving standard prioritizers to framework NAR and refactoring ExtensionManager
logic to handle cases where an extension is

[jira] [Commented] (NIFI-3520) HDFS processors experiencing Kerberos "impersonate" errors

2017-03-24 Thread ASF subversion and git services (JIRA)

[
https://issues.apache.org/jira/browse/NIFI-3520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940514#comment-15940514
]

ASF subversion and git services commented on NIFI-3520:
---

Commit d90cf846b9dcd49acdeac7bb5fcd13ee80dcdba6 in nifi's branch
refs/heads/master from [~bbende]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=d90cf84 ]

Updating unit tests to pass based on framework changes
- Fixing logging of extension types during start up

- Allowing the application to start if there is a compatible bundle found. -
Reporting missing bundle when the a compatible bundle is not found. - Fixing
table height in new component dialogs.

- Finishing clean up following rebase.

Calling lifecycle methods for add and remove when changing versions of a
component
- Introducing verifyCanUpdateBundle(coordinate) to ConfiguredComponent, and
adding unit tests

- Ensuring documentation is available for all components. Including those of
the same type that are loaded from different bundles.

- Auditing when user changes the bundle. - Ensuring bundle details are present
in templates.

Moving standard prioritizers to framework NAR and refactoring ExtensionManager
logic to handle cases where an extension is

[GitHub] nifi issue #1585: NIFI-3380 Support Multiple Versions of the Same Component

2017-03-24 Thread markap14

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/1585
  
This all looks good. I've run through the standard cases of standalone and 
clustered instances. Ran through some crazy cases that I could think of. Didn't 
run into any unexpected problems. Code looks good. I especially paid attention 
to the *Node classes, StandardFlowService, and Fingerprint classes when 
reviewing code but found nothing concerning. +1 merged to master!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-3380) Multiple Versions of the Same Component


 [ 
https://issues.apache.org/jira/browse/NIFI-3380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne updated NIFI-3380:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Multiple Versions of the Same Component
> ---
>
> Key: NIFI-3380
> URL: https://issues.apache.org/jira/browse/NIFI-3380
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Bryan Bende
>Assignee: Bryan Bende
> Fix For: 1.2.0
>
> Attachments: nifi-example-processors-nar-1.0.nar, 
> nifi-example-processors-nar-2.0.nar, nifi-example-service-api-nar-1.0.nar, 
> nifi-example-service-api-nar-2.0.nar, nifi-example-service-nar-1.0.nar, 
> nifi-example-service-nar-1.1.nar, nifi-example-service-nar-2.0.nar
>
>
> This ticket is to track the work for supporting multiple versions of the same 
> component within NiFi. The overall design for this feature is described in 
> detail at the following wiki page:
> https://cwiki.apache.org/confluence/display/NIFI/Multiple+Versions+of+the+Same+Extension
> This ticket will track only the core NiFi work, and a separate ticket will be 
> created to track enhancements for the NAR Maven Plugin.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3380) Multiple Versions of the Same Component

2017-03-24 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-3380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940517#comment-15940517
 ] 

ASF GitHub Bot commented on NIFI-3380:
--

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/1585
  
This all looks good. I've run through the standard cases of standalone and 
clustered instances. Ran through some crazy cases that I could think of. Didn't 
run into any unexpected problems. Code looks good. I especially paid attention 
to the *Node classes, StandardFlowService, and Fingerprint classes when 
reviewing code but found nothing concerning. +1 merged to master!


> Multiple Versions of the Same Component
> ---
>
> Key: NIFI-3380
> URL: https://issues.apache.org/jira/browse/NIFI-3380
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Bryan Bende
>Assignee: Bryan Bende
> Fix For: 1.2.0
>
> Attachments: nifi-example-processors-nar-1.0.nar, 
> nifi-example-processors-nar-2.0.nar, nifi-example-service-api-nar-1.0.nar, 
> nifi-example-service-api-nar-2.0.nar, nifi-example-service-nar-1.0.nar, 
> nifi-example-service-nar-1.1.nar, nifi-example-service-nar-2.0.nar
>
>
> This ticket is to track the work for supporting multiple versions of the same 
> component within NiFi. The overall design for this feature is described in 
> detail at the following wiki page:
> https://cwiki.apache.org/confluence/display/NIFI/Multiple+Versions+of+the+Same+Extension
> This ticket will track only the core NiFi work, and a separate ticket will be 
> created to track enhancements for the NAR Maven Plugin.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3380) Multiple Versions of the Same Component

[
https://issues.apache.org/jira/browse/NIFI-3380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940513#comment-15940513
]

ASF subversion and git services commented on NIFI-3380:
---

Commit d90cf846b9dcd49acdeac7bb5fcd13ee80dcdba6 in nifi's branch
refs/heads/master from [~bbende]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=d90cf84 ]

Updating unit tests to pass based on framework changes
- Fixing logging of extension types during start up

- Allowing the application to start if there is a compatible bundle found. -
Reporting missing bundle when the a compatible bundle is not found. - Fixing
table height in new component dialogs.

- Finishing clean up following rebase.

Calling lifecycle methods for add and remove when changing versions of a
component
- Introducing verifyCanUpdateBundle(coordinate) to ConfiguredComponent, and
adding unit tests

- Ensuring documentation is available for all components. Including those of
the same type that are loaded from different bundles.

- Auditing when user changes the bundle. - Ensuring bundle details are present
in templates.

Moving standard prioritizers to framework NAR and refactoring ExtensionManager
logic to handle cases where an extension is

[jira] [Commented] (NIFI-3380) Multiple Versions of the Same Component


[ 
https://issues.apache.org/jira/browse/NIFI-3380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940515#comment-15940515
 ] 

ASF GitHub Bot commented on NIFI-3380:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1585


> Multiple Versions of the Same Component
> ---
>
> Key: NIFI-3380
> URL: https://issues.apache.org/jira/browse/NIFI-3380
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Bryan Bende
>Assignee: Bryan Bende
> Fix For: 1.2.0
>
> Attachments: nifi-example-processors-nar-1.0.nar, 
> nifi-example-processors-nar-2.0.nar, nifi-example-service-api-nar-1.0.nar, 
> nifi-example-service-api-nar-2.0.nar, nifi-example-service-nar-1.0.nar, 
> nifi-example-service-nar-1.1.nar, nifi-example-service-nar-2.0.nar
>
>
> This ticket is to track the work for supporting multiple versions of the same 
> component within NiFi. The overall design for this feature is described in 
> detail at the following wiki page:
> https://cwiki.apache.org/confluence/display/NIFI/Multiple+Versions+of+the+Same+Extension
> This ticket will track only the core NiFi work, and a separate ticket will be 
> created to track enhancements for the NAR Maven Plugin.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi-minifi-cpp pull request #69: MINIFI-225: Add Linter for Google style gu...

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/69#discussion_r107925513
  
--- Diff: thirdparty/google-styleguide/cpplint.py ---
@@ -0,0 +1,6154 @@
+#!/usr/bin/env python
--- End diff --

We will need to capture this in our LICENSE file


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #69: MINIFI-225: Add Linter for Google style gu...

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/69#discussion_r107926105
  
--- Diff: thirdparty/google-styleguide/cppguide.html ---
@@ -0,0 +1,6170 @@
+
--- End diff --

Should likely just link to this in docs some place.  As best as I can tell, 
this file is CC-By 3.0 as per 
https://github.com/google/styleguide/blob/gh-pages/README.md.  This falls under 
a Category B (http://www.apache.org/legal/resolved.html#category-b) which seems 
to have some caveats best avoided unless there is a compelling reason to 
include 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #69: MINIFI-225: Add Linter for Google style gu...

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/69#discussion_r107926443
  
--- Diff: thirdparty/google-styleguide/eclipse-cpp-google-style.xml ---
@@ -0,0 +1,167 @@
+
--- End diff --

Might also want to just link to this as per the LICENSE concerns above.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #69: MINIFI-225: Add Linter for Google style gu...

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/69#discussion_r107930194
  
--- Diff: thirdparty/google-styleguide/cppguide.html ---
@@ -0,0 +1,6170 @@
+
--- End diff --

Agreed. Will change this. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #1576: NIFI-3518 Create a Morphlines processor

2017-03-24 Thread JPercivall

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/1576
  
Hey @WilliamNouet, sorry for not giving an update. Been too busy with other 
obligations. I'll try to get back to this when I have more free time.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3518) Create a Morphlines processor


[ 
https://issues.apache.org/jira/browse/NIFI-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940584#comment-15940584
 ] 

ASF GitHub Bot commented on NIFI-3518:
--

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/1576
  
Hey @WilliamNouet, sorry for not giving an update. Been too busy with other 
obligations. I'll try to get back to this when I have more free time.


> Create a Morphlines processor
> -
>
> Key: NIFI-3518
> URL: https://issues.apache.org/jira/browse/NIFI-3518
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: William Nouet
>Priority: Minor
>
> Create a dedicate processor to run Morphlines transformations 
> (http://kitesdk.org/docs/1.1.0/morphlines/morphlines-reference-guide.html) 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (NIFI-3645) Add aws:kms, sse-c, sse-c-key, and sse-kms-key-id capability to server side encryption in the PutS3Object processor.

2017-03-24 Thread Matthew Clarke (JIRA)

Matthew Clarke created NIFI-3645:


 Summary: Add aws:kms, sse-c, sse-c-key, and sse-kms-key-id 
capability to server side encryption in the PutS3Object processor.
 Key: NIFI-3645
 URL: https://issues.apache.org/jira/browse/NIFI-3645
 Project: Apache NiFi
  Issue Type: Improvement
Affects Versions: 1.1.0
Reporter: Matthew Clarke


Server Side Encryption (sse) currently exists in the PutS3Object processor but 
don’t see sse-c, sse-c-key or sse-kms-key-id options. 

--sse (string) Specifies server-side encryption of the object in S3. Valid 
values are AES256 and aws:kms. If the parameter is specified but no value is 
provided, AES256 is used. 
--sse-c (string) Specifies server-side encryption using customer provided keys 
of the the object in S3. AES256 is the only valid value. If the parameter is 
specified but no value is provided, AES256 is used. If you provide this value, 
--sse-c-key must be specified as well. 
--sse-c-key (string) The customer-provided encryption key to use to server-side 
encrypt the object in S3. If you provide this value, --sse-c must be specified 
as well. The key provided should not be base64 encoded. 
--sse-kms-key-id (string) The AWS KMS key ID that should be used to server-side 
encrypt the object in S3. Note that you should only provide this parameter if 
KMS key ID is different the default S3 master KMS key. 

Would like to see full support of the various server side encryption 
capabilities added to our S3 processors.

This is related partially to another Apache Jira: NIFI-1769



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi-minifi-cpp issue #68: MINIFI-193: Ensure safe UTF encoding

Github user apiri commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/68
  
reviewing


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp issue #68: MINIFI-193: Ensure safe UTF encoding

Github user apiri commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/68
  
Code changes look good and was able to verify execution of the binary as 
expected with patch changes.  Will get this merged, thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #68: MINIFI-193: Ensure safe UTF encoding

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi-cpp/pull/68


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Reopened] (NIFI-3260) Official Docker Image

2017-03-24 Thread Aldrin Piri (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-3260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aldrin Piri reopened NIFI-3260:
---
  Assignee: Aldrin Piri  (was: Jeremy Dyer)

Reopening to ensure we are tracking INFRA changes before a release occurs.

> Official Docker Image
> -
>
> Key: NIFI-3260
> URL: https://issues.apache.org/jira/browse/NIFI-3260
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Tools and Build
>Reporter: Jeremy Dyer
>Assignee: Aldrin Piri
>  Labels: docker
> Fix For: 1.2.0
>
>
> This JIRA is for setting up a Docker folder structure within the NiFi source 
> code as discussed in the dev mailing list at
> https://lists.apache.org/thread.html/e905a559cb01b30f1a7032cec5c9605685f27a65bdf7fee41b735089@%3Cdev.nifi.apache.org%3E



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3260) Official Docker Image

2017-03-24 Thread Aldrin Piri (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-3260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940778#comment-15940778
 ] 

Aldrin Piri commented on NIFI-3260:
---

We should also ensure we update the release guide to reflect the process given 
the very specific nature of steps in which things need to occur.

> Official Docker Image
> -
>
> Key: NIFI-3260
> URL: https://issues.apache.org/jira/browse/NIFI-3260
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Tools and Build
>Reporter: Jeremy Dyer
>Assignee: Aldrin Piri
>  Labels: docker
> Fix For: 1.2.0
>
>
> This JIRA is for setting up a Docker folder structure within the NiFi source 
> code as discussed in the dev mailing list at
> https://lists.apache.org/thread.html/e905a559cb01b30f1a7032cec5c9605685f27a65bdf7fee41b735089@%3Cdev.nifi.apache.org%3E



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (NIFI-3646) Show named relationship icons on processor hover

2017-03-24 Thread Nicholas Carenza (JIRA)

Nicholas Carenza created NIFI-3646:
--

 Summary: Show named relationship icons on processor hover
 Key: NIFI-3646
 URL: https://issues.apache.org/jira/browse/NIFI-3646
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Core UI
Reporter: Nicholas Carenza
Priority: Minor
 Attachments: nifi-connect-rel.png

When hovering over a processor, instead of showing a single add-connection 
element in the middle, show an array of connections. I attached an image 
example.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (NIFI-3647) Jolt processor - advanced UI - window size

2017-03-24 Thread Pierre Villard (JIRA)

Pierre Villard created NIFI-3647:


 Summary: Jolt processor - advanced UI - window size
 Key: NIFI-3647
 URL: https://issues.apache.org/jira/browse/NIFI-3647
 Project: Apache NiFi
  Issue Type: Bug
  Components: Extensions
Reporter: Pierre Villard
Priority: Minor
 Attachments: Screen Shot 2017-03-24 at 6.17.07 PM.png, Screen Shot 
2017-03-24 at 6.19.10 PM.png, Screen Shot 2017-03-24 at 6.21.50 PM.png

When using the Advanced UI of the Jolt processor, the window is not correctly 
sized and some parts can become unreachable. Observed on both Chrome and FF.

Steps to reproduce:
- Go to Advanced part of the Jolt processor
!Screen Shot 2017-03-24 at 6.17.07 PM.png|thumbnail!

- Part at the bottom is truncated:
!Screen Shot 2017-03-24 at 6.19.10 PM.png|thumbnail!

- If I enter a large JSON, the top part "Jolt Transformation DSL" is not 
reachable anymore (unless I play with zoom in/out of the browser):
!Screen Shot 2017-03-24 at 6.21.50 PM.png|thumbnail!




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-3647) Jolt processor - advanced UI - window size

2017-03-24 Thread Pierre Villard (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-3647:
-
Description: 
When using the Advanced UI of the Jolt processor, the window is not correctly 
sized and some parts can become unreachable. Observed on both Chrome and FF.

Steps to reproduce:
- Go to Advanced part of the Jolt processor
!Screen Shot 2017-03-24 at 6.17.07 PM.png|height=200!

- Part at the bottom is truncated:
!Screen Shot 2017-03-24 at 6.19.10 PM.png|height=200!

- If I enter a large JSON, the top part "Jolt Transformation DSL" is not 
reachable anymore (unless I play with zoom in/out of the browser):
!Screen Shot 2017-03-24 at 6.21.50 PM.png|height=200!


  was:
When using the Advanced UI of the Jolt processor, the window is not correctly 
sized and some parts can become unreachable. Observed on both Chrome and FF.

Steps to reproduce:
- Go to Advanced part of the Jolt processor
!Screen Shot 2017-03-24 at 6.17.07 PM.png|thumbnail!

- Part at the bottom is truncated:
!Screen Shot 2017-03-24 at 6.19.10 PM.png|thumbnail!

- If I enter a large JSON, the top part "Jolt Transformation DSL" is not 
reachable anymore (unless I play with zoom in/out of the browser):
!Screen Shot 2017-03-24 at 6.21.50 PM.png|thumbnail!



> Jolt processor - advanced UI - window size
> --
>
> Key: NIFI-3647
> URL: https://issues.apache.org/jira/browse/NIFI-3647
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Pierre Villard
>Priority: Minor
> Attachments: Screen Shot 2017-03-24 at 6.17.07 PM.png, Screen Shot 
> 2017-03-24 at 6.19.10 PM.png, Screen Shot 2017-03-24 at 6.21.50 PM.png
>
>
> When using the Advanced UI of the Jolt processor, the window is not correctly 
> sized and some parts can become unreachable. Observed on both Chrome and FF.
> Steps to reproduce:
> - Go to Advanced part of the Jolt processor
> !Screen Shot 2017-03-24 at 6.17.07 PM.png|height=200!
> - Part at the bottom is truncated:
> !Screen Shot 2017-03-24 at 6.19.10 PM.png|height=200!
> - If I enter a large JSON, the top part "Jolt Transformation DSL" is not 
> reachable anymore (unless I play with zoom in/out of the browser):
> !Screen Shot 2017-03-24 at 6.21.50 PM.png|height=200!



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3646) Show named relationship icons on processor hover

2017-03-24 Thread Joseph Witt (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-3646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940803#comment-15940803
 ] 

Joseph Witt commented on NIFI-3646:
---

[~ncarenza] can you talk more about how you envision this working compared to 
the current approach?

The current approach lets you connect a processor to another processor and then 
allows you to specify a range of important details about that connection such 
as the source relationship name, back pressure settings, etc..  This seems like 
the right model.  

Can you compare your proposal to that?  It looks like you're suggesting we list 
off one more more possible relationship names then grab that one to create an 
actual connection.

Or, are you simply asking for a way to see the relationship names of all 
connections leaving a processor whenever you hoever over that processor?  This 
would be not because you want to add more connections but simply so you could 
more easily understand the existing ones?  Keep in mind a connection can 
represent one or more relationships.

> Show named relationship icons on processor hover
> 
>
> Key: NIFI-3646
> URL: https://issues.apache.org/jira/browse/NIFI-3646
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Reporter: Nicholas Carenza
>Priority: Minor
>  Labels: connection, ui
> Attachments: nifi-connect-rel.png
>
>
> When hovering over a processor, instead of showing a single add-connection 
> element in the middle, show an array of connections. I attached an image 
> example.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1596: NIFI-3596 - added attributes to GenerateTableFetch ...

2017-03-24 Thread mattyb149

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1596#discussion_r107984019
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestGenerateTableFetch.java
 ---
@@ -207,6 +207,12 @@ public void testAddedRows() throws 
ClassNotFoundException, SQLException, Initial
 assertEquals("SELECT * FROM TEST_QUERY_DB_TABLE ORDER BY name 
OFFSET 4 ROWS FETCH NEXT 2 ROWS ONLY", new String(flowFile.toByteArray()));
 flowFile = runner.getFlowFilesForRelationship(REL_SUCCESS).get(3);
 assertEquals("SELECT * FROM TEST_QUERY_DB_TABLE ORDER BY name 
OFFSET 6 ROWS FETCH NEXT 2 ROWS ONLY", new String(flowFile.toByteArray()));
+assertEquals("TEST_QUERY_DB_TABLE", 
flowFile.getAttribute("generatetablefetch.tableName"));
+assertEquals(null, 
flowFile.getAttribute("generatetablefetch.columnNames"));
--- End diff --

All added asserts for columns are checking for null, can you update a test 
(or add one) that sets the Columns To Return property? Otherwise, looks good to 
me, will build and run on a real NiFi in the meantime


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3596) GenerateTableFetch - Add attributes to generated flow files to ease SQL query overwrite


[ 
https://issues.apache.org/jira/browse/NIFI-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941011#comment-15941011
 ] 

ASF GitHub Bot commented on NIFI-3596:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1596#discussion_r107984019
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestGenerateTableFetch.java
 ---
@@ -207,6 +207,12 @@ public void testAddedRows() throws 
ClassNotFoundException, SQLException, Initial
 assertEquals("SELECT * FROM TEST_QUERY_DB_TABLE ORDER BY name 
OFFSET 4 ROWS FETCH NEXT 2 ROWS ONLY", new String(flowFile.toByteArray()));
 flowFile = runner.getFlowFilesForRelationship(REL_SUCCESS).get(3);
 assertEquals("SELECT * FROM TEST_QUERY_DB_TABLE ORDER BY name 
OFFSET 6 ROWS FETCH NEXT 2 ROWS ONLY", new String(flowFile.toByteArray()));
+assertEquals("TEST_QUERY_DB_TABLE", 
flowFile.getAttribute("generatetablefetch.tableName"));
+assertEquals(null, 
flowFile.getAttribute("generatetablefetch.columnNames"));
--- End diff --

All added asserts for columns are checking for null, can you update a test 
(or add one) that sets the Columns To Return property? Otherwise, looks good to 
me, will build and run on a real NiFi in the meantime


> GenerateTableFetch - Add attributes to generated flow files to ease SQL query 
> overwrite
> ---
>
> Key: NIFI-3596
> URL: https://issues.apache.org/jira/browse/NIFI-3596
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Trivial
> Fix For: 1.2.0
>
>
> The GenerateTableFetch processor will generate a SQL query based on the 
> provided parameters but, if the specific DB adapter is not available, it 
> might be necessary to overwrite the SQL query in the ExecuteSQL processor. To 
> do that it would be nice to have each part of the query as attributes to take 
> advantage of expression language.
> Current workaround (when using GenerateTableFetch against a DB2 database for 
> example) is to have an intermediary ExtractText processor and have a regex 
> extracting each part of the generated query.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1589: NIFI-1705 Adding AttributesToCSV processor

2017-03-24 Thread mattyb149

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1589#discussion_r107989450
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AttributesToCSV.java
 ---
@@ -0,0 +1,280 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import org.apache.commons.lang3.StringEscapeUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+
+import java.io.BufferedOutputStream;
+import java.io.OutputStream;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Set;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Collections;
+import java.util.stream.Collectors;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"csv", "attributes", "flowfile"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Generates a CSV representation of the input 
FlowFile Attributes. The resulting CSV " +
+"can be written to either a newly generated attribute named 
'CSVAttributes' or written to the FlowFile as content.  " +
+"If the attribute value contains a comma, newline or double quote, 
then the attribute value will be " +
+"escaped with double quotes.  Any double quote characters in the 
attribute value are escaped with " +
+"another double quote.  If the attribute value does not contain a 
comma, newline or double quote, then the " +
+"attribute value is returned unchanged.")
+@WritesAttribute(attribute = "CSVAttributes", description = "CSV 
representation of Attributes")
+public class AttributesToCSV extends AbstractProcessor {
+
+private static final String OUTPUT_NEW_ATTRIBUTE = 
"flowfile-attribute";
+private static final String OUTPUT_OVERWRITE_CONTENT = 
"flowfile-content";
+private static final String OUTPUT_ATTRIBUTE_NAME = "CSVAttributes";
+private static final String OUTPUT_SEPARATOR = ",";
+private static final String OUTPUT_MIME_TYPE = "text/csv";
+
+
+public static final PropertyDescriptor ATTRIBUTES_LIST = new 
PropertyDescriptor.Builder()
--- End diff --

Is there any value in supporting a regular expression or (alternatively) 
supporting Expression Language for the Attribute List property? For the former, 
I'm thinking of the use case where there might be an arbitrary number of 
attributes available, often with the same prefix. For the latter (I doubt we'd 
want to support both), I'm thinking of the case (if it exists) where the CSV 
column names might themselves be in an attribute, where their values might be 
in the corresponding attribute(s). That second one seems pretty much an edge 
case, I just wanted to bring these up for discussion. If neither seems 
particularly u

[jira] [Commented] (NIFI-1705) AttributesToCSV


[ 
https://issues.apache.org/jira/browse/NIFI-1705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941059#comment-15941059
 ] 

ASF GitHub Bot commented on NIFI-1705:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1589#discussion_r107989450
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AttributesToCSV.java
 ---
@@ -0,0 +1,280 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import org.apache.commons.lang3.StringEscapeUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+
+import java.io.BufferedOutputStream;
+import java.io.OutputStream;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Set;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Collections;
+import java.util.stream.Collectors;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"csv", "attributes", "flowfile"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Generates a CSV representation of the input 
FlowFile Attributes. The resulting CSV " +
+"can be written to either a newly generated attribute named 
'CSVAttributes' or written to the FlowFile as content.  " +
+"If the attribute value contains a comma, newline or double quote, 
then the attribute value will be " +
+"escaped with double quotes.  Any double quote characters in the 
attribute value are escaped with " +
+"another double quote.  If the attribute value does not contain a 
comma, newline or double quote, then the " +
+"attribute value is returned unchanged.")
+@WritesAttribute(attribute = "CSVAttributes", description = "CSV 
representation of Attributes")
+public class AttributesToCSV extends AbstractProcessor {
+
+private static final String OUTPUT_NEW_ATTRIBUTE = 
"flowfile-attribute";
+private static final String OUTPUT_OVERWRITE_CONTENT = 
"flowfile-content";
+private static final String OUTPUT_ATTRIBUTE_NAME = "CSVAttributes";
+private static final String OUTPUT_SEPARATOR = ",";
+private static final String OUTPUT_MIME_TYPE = "text/csv";
+
+
+public static final PropertyDescriptor ATTRIBUTES_LIST = new 
PropertyDescriptor.Builder()
--- End diff --

Is there any value in supporting a regular expression or (alternatively) 
supporting Expression Language for the Attribute List property? For the former, 
I'm thinking of the use case where there might be an arbitrary number of 
attributes available, often with the same prefix. For the latter (I doubt we'd 
want to support both), I'm thinking of the case (if it exists) where

[GitHub] nifi pull request #1589: NIFI-1705 Adding AttributesToCSV processor

2017-03-24 Thread mattyb149

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1589#discussion_r107990719
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AttributesToCSV.java
 ---
@@ -0,0 +1,280 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import org.apache.commons.lang3.StringEscapeUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+
+import java.io.BufferedOutputStream;
+import java.io.OutputStream;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Set;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Collections;
+import java.util.stream.Collectors;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"csv", "attributes", "flowfile"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Generates a CSV representation of the input 
FlowFile Attributes. The resulting CSV " +
+"can be written to either a newly generated attribute named 
'CSVAttributes' or written to the FlowFile as content.  " +
+"If the attribute value contains a comma, newline or double quote, 
then the attribute value will be " +
+"escaped with double quotes.  Any double quote characters in the 
attribute value are escaped with " +
+"another double quote.  If the attribute value does not contain a 
comma, newline or double quote, then the " +
+"attribute value is returned unchanged.")
+@WritesAttribute(attribute = "CSVAttributes", description = "CSV 
representation of Attributes")
+public class AttributesToCSV extends AbstractProcessor {
+
+private static final String OUTPUT_NEW_ATTRIBUTE = 
"flowfile-attribute";
+private static final String OUTPUT_OVERWRITE_CONTENT = 
"flowfile-content";
+private static final String OUTPUT_ATTRIBUTE_NAME = "CSVAttributes";
+private static final String OUTPUT_SEPARATOR = ",";
+private static final String OUTPUT_MIME_TYPE = "text/csv";
+
+
+public static final PropertyDescriptor ATTRIBUTES_LIST = new 
PropertyDescriptor.Builder()
+.name("attribute-list")
+.displayName("Attribute List")
+.description("Comma separated list of attributes to be 
included in the resulting CSV. If this value " +
+"is left empty then all existing Attributes will be 
included. This list of attributes is " +
+"case sensitive. If an attribute specified in the list 
is not found it will be emitted " +
+"to the resulting CSV with an empty string or null 
depending on the 'Null Value' property. " +
+"If a core attribute is specified in this list " +
+

[jira] [Created] (NIFI-3648) Address Excessive Garbage Collection

Mark Payne created NIFI-3648:


 Summary: Address Excessive Garbage Collection
 Key: NIFI-3648
 URL: https://issues.apache.org/jira/browse/NIFI-3648
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Core Framework, Extensions
Reporter: Mark Payne
Assignee: Mark Payne


We have a lot of places in the codebase where we generate lots of unnecessary 
garbage - especially byte arrays. We need to clean this up to in order to 
relieve stress on the garbage collector.

Specific points that I've found create unnecessary garbage:

Provenance CompressableRecordWriter creates a new BufferedOutputStream for each 
'compression block' that it creates. Each one has a 64 KB byte[]. This is very 
wasteful. We should instead subclass BufferedOutputStream so that we are able 
to provide a byte[] to use instead of an int that indicates the size. This way, 
we can just keep re-using the same byte[] that we create for each writer. This 
saves about 32,000 of these 64 KB byte[] for each writer that we create. And we 
create more than 1 of these per minute.

EvaluateJsonPath uses a BufferedInputStream but it is not necessary, because 
the underlying library will also buffer data. So we are unnecessarily creating 
a lot of byte[]'s

CompressContent uses Buffered Input AND Output. And uses 64 KB byte[]. And 
doesn't need them at all, because it reads and writes with its own byte[] 
buffer via StreamUtils.copy

Site-to-site uses CompressionInputStream. This stream creates a new byte[] in 
the readChunkHeader() method continually. We should instead only create a new 
byte[] if we need a bigger buffer and otherwise just use an offset & length 
variable.

Right now, SplitText uses TextLineDemarcator. The fill() method increases the 
size of the internal byte[] by 8 KB each time. When dealing with a large chunk 
of data, this is VERY expensive on GC because we continually create a byte[] 
and then discard it to create a new one. Take for example an 800 KB chunk. We 
would do this 100,000 times. If we instead double the size we would only have 
to create 8 of these.

Other Processors that use Buffered streams unnecessarily:

ConvertJSONToSQL
ExecuteProcess
ExecuteStreamCommand
AttributesToJSON
EvaluateJsonPath (when writing to content)
ExtractGrok
JmsConsumer




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-1705) AttributesToCSV


[ 
https://issues.apache.org/jira/browse/NIFI-1705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941068#comment-15941068
 ] 

ASF GitHub Bot commented on NIFI-1705:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1589#discussion_r107990719
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AttributesToCSV.java
 ---
@@ -0,0 +1,280 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import org.apache.commons.lang3.StringEscapeUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+
+import java.io.BufferedOutputStream;
+import java.io.OutputStream;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Set;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Collections;
+import java.util.stream.Collectors;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"csv", "attributes", "flowfile"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Generates a CSV representation of the input 
FlowFile Attributes. The resulting CSV " +
+"can be written to either a newly generated attribute named 
'CSVAttributes' or written to the FlowFile as content.  " +
+"If the attribute value contains a comma, newline or double quote, 
then the attribute value will be " +
+"escaped with double quotes.  Any double quote characters in the 
attribute value are escaped with " +
+"another double quote.  If the attribute value does not contain a 
comma, newline or double quote, then the " +
+"attribute value is returned unchanged.")
+@WritesAttribute(attribute = "CSVAttributes", description = "CSV 
representation of Attributes")
+public class AttributesToCSV extends AbstractProcessor {
+
+private static final String OUTPUT_NEW_ATTRIBUTE = 
"flowfile-attribute";
+private static final String OUTPUT_OVERWRITE_CONTENT = 
"flowfile-content";
+private static final String OUTPUT_ATTRIBUTE_NAME = "CSVAttributes";
+private static final String OUTPUT_SEPARATOR = ",";
+private static final String OUTPUT_MIME_TYPE = "text/csv";
+
+
+public static final PropertyDescriptor ATTRIBUTES_LIST = new 
PropertyDescriptor.Builder()
+.name("attribute-list")
+.displayName("Attribute List")
+.description("Comma separated list of attributes to be 
included in the resulting CSV. If this value " +
+"is left empty then all existing Attributes will be 
included. This list of attributes is " +
+"case sensitive. If an attribute specified in the l

[jira] [Created] (NIFI-3649) Parallelize Consumption of Cluster Replication Responses

Mark Payne created NIFI-3649:


 Summary: Parallelize Consumption of Cluster Replication Responses
 Key: NIFI-3649
 URL: https://issues.apache.org/jira/browse/NIFI-3649
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Core Framework
Reporter: Mark Payne
Assignee: Mark Payne


Currently, when we replicate a REST API call to all nodes in the cluster, we 
wait until we get back a Response object for each node. We then go to the 
Response Mapper and merge those responses into 1 NodeResponse. This merging is 
done serially, and this is where we actually do the reading of the response. We 
need to instead do this merging in parallel.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3257) Cluster stability issues during high throughput


[ 
https://issues.apache.org/jira/browse/NIFI-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941078#comment-15941078
 ] 

Mark Payne commented on NIFI-3257:
--

Using the -XX:+PrintGC JVM option and using YourKit to profile garbage 
collection, along with significant DEBUG logging that I've added, I am seeing 
that the problem largely is due to excessive GC runs. I'm seeing up to 25% of 
my JVM time spent performing Garbage Collection. I've marked this ticket as 
being related to NIFI-3636 and NIFI-3648 because these tickets are intended to 
address the heavy garbage collection.

I've also found that why we use multiple threads to replicate REST API calls, 
we do not read or merge node responses in parallel. This is done serially after 
all "Response" objects have been obtained. This is very inefficient and can 
result in very long request replication times. It could even result in one slow 
node causing other nodes' responses to timeout meaning that if Node 1 is slow 
to respond (due to GC or whatever), then the responses from Nodes 4, 5, and 6, 
for instance, could time out. As a result, nodes 4, 5, and 6 could be kicked 
out of the cluster as a result of Node 1 being slow.

> Cluster stability issues during high throughput
> ---
>
> Key: NIFI-3257
> URL: https://issues.apache.org/jira/browse/NIFI-3257
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0, 1.1.0, 1.1.1, 1.0.1
>Reporter: Jeff Storck
>
> During high throughput of data in a cluster (135MB/s), nodes experience 
> frequent disconnects (every few minutes) and role switching (Primary and 
> Cluster Coordinator).  This makes API requests difficult since the requests 
> can not be replicated to all nodes while reconnecting.  The cluster can 
> recover for a time (as mentioned above, for a few minutes) before going 
> through another round of disconnects and role switching.
> The cluster is able to continue to process data during these connection and 
> role-switching issues.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1620: NIFI-3421: Preventing successive attempts to create...

2017-03-24 Thread mcgilman

GitHub user mcgilman opened a pull request:

https://github.com/apache/nifi/pull/1620

NIFI-3421: Preventing successive attempts to create a FlowController

NIFI-3421:
- On contextDestroyed, referencing beans created during contextInitialized 
to prevent successive attempts to create a bean if that bean failed to be 
created initially.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mcgilman/nifi NIFI-3421

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1620.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1620


commit 2ac22e78381cf0dfda6c2d981857865214a4cc41
Author: Matt Gilman 
Date:   2017-03-24T20:30:48Z

NIFI-3421:
- On contextDestroyed, referencing beans created during contextInitialized 
to prevent successive attempts to create a bean if that bean failed to be 
created initially.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-3421) Start up failure - java.nio.channels.OverlappingFileLockException: null

2017-03-24 Thread Matt Gilman (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-3421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman updated NIFI-3421:
--
Status: Patch Available  (was: In Progress)

> Start up failure - java.nio.channels.OverlappingFileLockException: null
> ---
>
> Key: NIFI-3421
> URL: https://issues.apache.org/jira/browse/NIFI-3421
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Critical
> Fix For: 1.2.0
>
>
> If NiFi fails to start up successfully, the act of attempting to shut down 
> appears to result in a duplicate FlowController instantiation. A symptom of 
> this is initializing the flow file repo twice which fails with an overlapping 
> file lock exception.
> The actual reason that start up failed is logged further back in the app log.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3421) Start up failure - java.nio.channels.OverlappingFileLockException: null


[ 
https://issues.apache.org/jira/browse/NIFI-3421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941086#comment-15941086
 ] 

ASF GitHub Bot commented on NIFI-3421:
--

GitHub user mcgilman opened a pull request:

https://github.com/apache/nifi/pull/1620

NIFI-3421: Preventing successive attempts to create a FlowController

NIFI-3421:
- On contextDestroyed, referencing beans created during contextInitialized 
to prevent successive attempts to create a bean if that bean failed to be 
created initially.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mcgilman/nifi NIFI-3421

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1620.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1620


commit 2ac22e78381cf0dfda6c2d981857865214a4cc41
Author: Matt Gilman 
Date:   2017-03-24T20:30:48Z

NIFI-3421:
- On contextDestroyed, referencing beans created during contextInitialized 
to prevent successive attempts to create a bean if that bean failed to be 
created initially.




> Start up failure - java.nio.channels.OverlappingFileLockException: null
> ---
>
> Key: NIFI-3421
> URL: https://issues.apache.org/jira/browse/NIFI-3421
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Critical
> Fix For: 1.2.0
>
>
> If NiFi fails to start up successfully, the act of attempting to shut down 
> appears to result in a duplicate FlowController instantiation. A symptom of 
> this is initializing the flow file repo twice which fails with an overlapping 
> file lock exception.
> The actual reason that start up failed is logged further back in the app log.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (NIFI-3304) In the Administration Guide, there is an incorrect JAVA 8 parameter for bootstrap.conf file


 [ 
https://issues.apache.org/jira/browse/NIFI-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Moser reassigned NIFI-3304:
---

Assignee: Michael Moser

> In the Administration Guide, there is an incorrect JAVA 8 parameter for 
> bootstrap.conf file
> ---
>
> Key: NIFI-3304
> URL: https://issues.apache.org/jira/browse/NIFI-3304
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Reporter: David A. Wynne
>Assignee: Michael Moser
>Priority: Minor
>
> In the Administration guide, section Bootstrap properties, there is a 
> paragraph about Java 8 handling of codecache, the listed properties are 
> accurate for Java version 7, but for Java version 8, one has changed
> This is correct for Java version 7:
> java.arg.8=-XX:CodeCacheFlushingMinimumFreeSpace=10m
> But for Java version 8, it should be
> java.arg.8=-XX:CodeCacheMinimumFreeSpace=10m



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3304) In the Administration Guide, there is an incorrect JAVA 8 parameter for bootstrap.conf file


[ 
https://issues.apache.org/jira/browse/NIFI-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941121#comment-15941121
 ] 

Michael Moser commented on NIFI-3304:
-

According to Oracle's Java 8 documentation \[1\] the defaults are currently

-XX:ReservedCodeCacheSize=240m
-XX:+UseCodeCacheFlushing

These seem to already align with our Admin Guide recommendations, so I think we 
can remove this section of the Admin Guide.

With Java 7, the default ReservedCodeCacheSize was 48m and codecache flushing 
was buggy \[2\] so our older Admin Guide recommendations made more sense.

\[1\] - http://docs.oracle.com/javase/8/docs/technotes/tools/windows/java.html
\[2\] - https://blogs.oracle.com/poonam/entry/why_do_i_get_message


> In the Administration Guide, there is an incorrect JAVA 8 parameter for 
> bootstrap.conf file
> ---
>
> Key: NIFI-3304
> URL: https://issues.apache.org/jira/browse/NIFI-3304
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Reporter: David A. Wynne
>Assignee: Michael Moser
>Priority: Minor
>
> In the Administration guide, section Bootstrap properties, there is a 
> paragraph about Java 8 handling of codecache, the listed properties are 
> accurate for Java version 7, but for Java version 8, one has changed
> This is correct for Java version 7:
> java.arg.8=-XX:CodeCacheFlushingMinimumFreeSpace=10m
> But for Java version 8, it should be
> java.arg.8=-XX:CodeCacheMinimumFreeSpace=10m



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (NIFI-3304) In the Administration Guide, there is an incorrect JAVA 8 parameter for bootstrap.conf file


[ 
https://issues.apache.org/jira/browse/NIFI-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941121#comment-15941121
 ] 

Michael Moser edited comment on NIFI-3304 at 3/24/17 9:02 PM:
--

According to Oracle's Java 8 documentation \[1\] the defaults are currently

-XX:ReservedCodeCacheSize=240m
-XX:+UseCodeCacheFlushing

Since NiFi 1.x requires Java 8, and these defaults already align with our Admin 
Guide recommendations, I think we can remove this section of the Admin Guide.  
Also, these settings have already been removed from the bootstrap.conf.

With Java 7, the default ReservedCodeCacheSize was 48m and codecache flushing 
was buggy \[2\] so our older Admin Guide recommendations made more sense in the 
NiFi 0.x days.

\[1\] - http://docs.oracle.com/javase/8/docs/technotes/tools/windows/java.html
\[2\] - https://blogs.oracle.com/poonam/entry/why_do_i_get_message



was (Author: mosermw):
According to Oracle's Java 8 documentation \[1\] the defaults are currently

-XX:ReservedCodeCacheSize=240m
-XX:+UseCodeCacheFlushing

These seem to already align with our Admin Guide recommendations, so I think we 
can remove this section of the Admin Guide.

With Java 7, the default ReservedCodeCacheSize was 48m and codecache flushing 
was buggy \[2\] so our older Admin Guide recommendations made more sense.

\[1\] - http://docs.oracle.com/javase/8/docs/technotes/tools/windows/java.html
\[2\] - https://blogs.oracle.com/poonam/entry/why_do_i_get_message


> In the Administration Guide, there is an incorrect JAVA 8 parameter for 
> bootstrap.conf file
> ---
>
> Key: NIFI-3304
> URL: https://issues.apache.org/jira/browse/NIFI-3304
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Reporter: David A. Wynne
>Assignee: Michael Moser
>Priority: Minor
>
> In the Administration guide, section Bootstrap properties, there is a 
> paragraph about Java 8 handling of codecache, the listed properties are 
> accurate for Java version 7, but for Java version 8, one has changed
> This is correct for Java version 7:
> java.arg.8=-XX:CodeCacheFlushingMinimumFreeSpace=10m
> But for Java version 8, it should be
> java.arg.8=-XX:CodeCacheMinimumFreeSpace=10m



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi issue #1596: NIFI-3596 - added attributes to GenerateTableFetch process...

2017-03-24 Thread pvillard31

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/1596
  
Thanks for the review @mattyb149.
I updated a test to check the selected column names.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3596) GenerateTableFetch - Add attributes to generated flow files to ease SQL query overwrite


[ 
https://issues.apache.org/jira/browse/NIFI-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941144#comment-15941144
 ] 

ASF GitHub Bot commented on NIFI-3596:
--

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/1596
  
Thanks for the review @mattyb149.
I updated a test to check the selected column names.


> GenerateTableFetch - Add attributes to generated flow files to ease SQL query 
> overwrite
> ---
>
> Key: NIFI-3596
> URL: https://issues.apache.org/jira/browse/NIFI-3596
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Trivial
> Fix For: 1.2.0
>
>
> The GenerateTableFetch processor will generate a SQL query based on the 
> provided parameters but, if the specific DB adapter is not available, it 
> might be necessary to overwrite the SQL query in the ExecuteSQL processor. To 
> do that it would be nice to have each part of the query as attributes to take 
> advantage of expression language.
> Current workaround (when using GenerateTableFetch against a DB2 database for 
> example) is to have an intermediary ExtractText processor and have a regex 
> extracting each part of the generated query.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1618: NIFI-3413: Add GetChangeDataCaptureMySQL processor

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1618#discussion_r107912643
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/impl/mysql/event/io/AbstractBinlogTableEventWriter.java
 ---
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.db.impl.mysql.event.io;
+
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processors.standard.GetChangeDataCaptureMySQL;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.BinlogTableEventInfo;
+
+import java.io.IOException;
+
+/**
+ * An abstract base class for writing MYSQL binlog events into flow 
file(s), e.g.
+ */
+public abstract class AbstractBinlogTableEventWriter extends AbstractBinlogEventWriter {
+
+protected void writeJson(T event) throws IOException {
+super.writeJson(event);
+if (event.getDatabaseName() != null) {
+jsonGenerator.writeStringField("database", 
event.getDatabaseName());
+} else {
+jsonGenerator.writeNullField("database");
+}
+if (event.getTableName() != null) {
+jsonGenerator.writeStringField("table_name", 
event.getTableName());
+} else {
+jsonGenerator.writeNullField("table_name");
+}
+if (event.getTableId() != null) {
+jsonGenerator.writeNumberField("table_id", event.getTableId());
+} else {
+jsonGenerator.writeNullField("table_id");
+}
+}
+
+// Default implementation for binlog events
+@Override
+public long writeEvent(ProcessSession session, T eventInfo, long 
currentSequenceId) {
+FlowFile flowFile = session.create();
+flowFile = session.write(flowFile, (outputStream) -> {
+super.startJson(outputStream, eventInfo);
+super.writeJson(eventInfo);
--- End diff --

Isn't the super.writeJson(event) encapsulaed by 
AbstractBinlogEventWriter.writeEvent(...) when it calls writeJson(T)?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1618: NIFI-3413: Add GetChangeDataCaptureMySQL processor

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1618#discussion_r107910775
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/event/TableInfoCacheKey.java
 ---
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.db.event;
+
+import 
org.apache.nifi.distributed.cache.client.exception.SerializationException;
+
+import java.io.IOException;
+import java.io.OutputStream;
+
+import static 
org.apache.nifi.processors.standard.db.event.TableInfo.DB_TABLE_NAME_DELIMITER;
+
+/**
+ * This class represents a key in a cache that contains information 
(column definitions, e.g.) for a database table
+ */
+public class TableInfoCacheKey {
+
+private final String databaseName;
+private final String tableName;
+private final long tableId;
+private final String uuidPrefix;
+
+public TableInfoCacheKey(String uuidPrefix, String databaseName, 
String tableName, long tableId) {
+this.uuidPrefix = uuidPrefix;
+this.databaseName = databaseName;
+this.tableName = tableName;
+this.tableId = tableId;
+}
+
+@Override
+public boolean equals(Object o) {
+if (this == o) return true;
+if (o == null || getClass() != o.getClass()) return false;
+
+TableInfoCacheKey that = (TableInfoCacheKey) o;
+
+if (tableId != that.tableId) return false;
+if (databaseName != null ? !databaseName.equals(that.databaseName) 
: that.databaseName != null) return false;
--- End diff --

what if that.databaseName is null? I don't believe java.lang.String checks 
null. Could use EqualsBuilder from commons-lang, which does the null check 
since commons-lang3 is in the pom. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1618: NIFI-3413: Add GetChangeDataCaptureMySQL processor

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1618#discussion_r107922204
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetChangeDataCaptureMySQL.java
 ---
@@ -0,0 +1,879 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import com.github.shyiko.mysql.binlog.BinaryLogClient;
+import com.github.shyiko.mysql.binlog.event.Event;
+import com.github.shyiko.mysql.binlog.event.EventHeaderV4;
+import com.github.shyiko.mysql.binlog.event.EventType;
+import com.github.shyiko.mysql.binlog.event.QueryEventData;
+import com.github.shyiko.mysql.binlog.event.RotateEventData;
+import com.github.shyiko.mysql.binlog.event.TableMapEventData;
+import org.apache.nifi.annotation.behavior.DynamicProperties;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateManager;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.standard.db.CDCException;
+import org.apache.nifi.processors.standard.db.event.ColumnDefinition;
+import org.apache.nifi.processors.standard.db.event.RowEventException;
+import org.apache.nifi.processors.standard.db.event.TableInfo;
+import org.apache.nifi.processors.standard.db.event.TableInfoCacheKey;
+import org.apache.nifi.processors.standard.db.event.io.EventWriter;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.BeginTransactionEventInfo;
+import org.apache.nifi.processors.standard.db.impl.mysql.RawBinlogEvent;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.BinlogEventListener;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.BinlogEventInfo;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.CommitTransactionEventInfo;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.DeleteRowsEventInfo;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.SchemaChangeEventInfo;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.UpdateRowsEventInfo;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.InsertRowsEventInfo;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.io.BeginTransactionEventWriter;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.io.CommitTransactionEventWriter;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.io.DeleteRowsWriter;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.io.InsertRowsWriter;
+import

[GitHub] nifi pull request #1618: NIFI-3413: Add GetChangeDataCaptureMySQL processor

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1618#discussion_r107921926
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetChangeDataCaptureMySQL.java
 ---
@@ -0,0 +1,879 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import com.github.shyiko.mysql.binlog.BinaryLogClient;
+import com.github.shyiko.mysql.binlog.event.Event;
+import com.github.shyiko.mysql.binlog.event.EventHeaderV4;
+import com.github.shyiko.mysql.binlog.event.EventType;
+import com.github.shyiko.mysql.binlog.event.QueryEventData;
+import com.github.shyiko.mysql.binlog.event.RotateEventData;
+import com.github.shyiko.mysql.binlog.event.TableMapEventData;
+import org.apache.nifi.annotation.behavior.DynamicProperties;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateManager;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.standard.db.CDCException;
+import org.apache.nifi.processors.standard.db.event.ColumnDefinition;
+import org.apache.nifi.processors.standard.db.event.RowEventException;
+import org.apache.nifi.processors.standard.db.event.TableInfo;
+import org.apache.nifi.processors.standard.db.event.TableInfoCacheKey;
+import org.apache.nifi.processors.standard.db.event.io.EventWriter;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.BeginTransactionEventInfo;
+import org.apache.nifi.processors.standard.db.impl.mysql.RawBinlogEvent;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.BinlogEventListener;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.BinlogEventInfo;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.CommitTransactionEventInfo;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.DeleteRowsEventInfo;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.SchemaChangeEventInfo;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.UpdateRowsEventInfo;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.InsertRowsEventInfo;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.io.BeginTransactionEventWriter;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.io.CommitTransactionEventWriter;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.io.DeleteRowsWriter;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.io.InsertRowsWriter;
+import

[GitHub] nifi pull request #1618: NIFI-3413: Add GetChangeDataCaptureMySQL processor

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1618#discussion_r107908219
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/event/ColumnDefinition.java
 ---
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.db.event;
+
+/**
+ * A class that specifies a definition for a relational table column, 
including type, name, etc.
+ */
+public class ColumnDefinition {
+
+private byte type;
+private String name = "";
+
+public ColumnDefinition(byte type) {
--- End diff --

Is there a way to avoid a byte and use an enumeration? I may answer this 
myself as I continue the review...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1618: NIFI-3413: Add GetChangeDataCaptureMySQL processor

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1618#discussion_r107914632
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/impl/mysql/event/io/DeleteRowsWriter.java
 ---
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.db.impl.mysql.event.io;
+
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processors.standard.GetChangeDataCaptureMySQL;
+import org.apache.nifi.processors.standard.db.event.ColumnDefinition;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.DeleteRowsEventInfo;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.BitSet;
+import java.util.concurrent.atomic.AtomicLong;
+
+import static 
org.apache.nifi.processors.standard.db.impl.mysql.MySQLCDCUtils.getWritableObject;
+
+/**
+ * A writer class to output MySQL binlog "delete rows" events to flow 
file(s).
+ */
+public class DeleteRowsWriter extends 
AbstractBinlogTableEventWriter {
+
+/**
+ * Creates and transfers a new flow file whose contents are the 
JSON-serialized value of the specified event, and the sequence ID attribute set
+ *
+ * @param session   A reference to a ProcessSession from which the 
flow file(s) will be created and transferred
+ * @param eventInfo An event whose value will become the contents of 
the flow file
+ * @return The next available CDC sequence ID for use by the CDC 
processor
+ */
+public long writeEvent(final ProcessSession session, final 
DeleteRowsEventInfo eventInfo, final long currentSequenceId) {
+final AtomicLong seqId = new AtomicLong(currentSequenceId);
--- End diff --

The correct answer is that I'm usually missing something...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3413) Implement a GetChangeDataCapture processor


[ 
https://issues.apache.org/jira/browse/NIFI-3413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941183#comment-15941183
 ] 

ASF GitHub Bot commented on NIFI-3413:
--

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1618#discussion_r107910775
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/event/TableInfoCacheKey.java
 ---
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.db.event;
+
+import 
org.apache.nifi.distributed.cache.client.exception.SerializationException;
+
+import java.io.IOException;
+import java.io.OutputStream;
+
+import static 
org.apache.nifi.processors.standard.db.event.TableInfo.DB_TABLE_NAME_DELIMITER;
+
+/**
+ * This class represents a key in a cache that contains information 
(column definitions, e.g.) for a database table
+ */
+public class TableInfoCacheKey {
+
+private final String databaseName;
+private final String tableName;
+private final long tableId;
+private final String uuidPrefix;
+
+public TableInfoCacheKey(String uuidPrefix, String databaseName, 
String tableName, long tableId) {
+this.uuidPrefix = uuidPrefix;
+this.databaseName = databaseName;
+this.tableName = tableName;
+this.tableId = tableId;
+}
+
+@Override
+public boolean equals(Object o) {
+if (this == o) return true;
+if (o == null || getClass() != o.getClass()) return false;
+
+TableInfoCacheKey that = (TableInfoCacheKey) o;
+
+if (tableId != that.tableId) return false;
+if (databaseName != null ? !databaseName.equals(that.databaseName) 
: that.databaseName != null) return false;
--- End diff --

what if that.databaseName is null? I don't believe java.lang.String checks 
null. Could use EqualsBuilder from commons-lang, which does the null check 
since commons-lang3 is in the pom. 


> Implement a GetChangeDataCapture processor
> --
>
> Key: NIFI-3413
> URL: https://issues.apache.org/jira/browse/NIFI-3413
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Database systems such as MySQL, Oracle, and SQL Server allow access to their 
> transactional logs and such, in order for external clients to have a "change 
> data capture" (CDC) capability. I propose a GetChangeDataCapture processor to 
> enable this in NiFi.
> The processor would be configured with a DBCPConnectionPool controller 
> service, as well as a Database Type property (similar to the one in 
> QueryDatabaseTable) for database-specific handling. Additional properties 
> might include the CDC table name, etc.  Additional database-specific 
> properties could be handled using dynamic properties (and the documentation 
> should reflect this).
> The processor would accept no incoming connections (it is a "Get" or source 
> processor), would be intended to run on the primary node only as a single 
> threaded processor, and would generate a flow file for each operation 
> (INSERT, UPDATE, DELETE, e,g,) in one or some number of formats (JSON, e.g.). 
> The flow files would be transferred in time order (to enable a replication 
> solution, for example), perhaps with some auto-incrementing attribute to also 
> indicate order if need be.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3413) Implement a GetChangeDataCapture processor


[ 
https://issues.apache.org/jira/browse/NIFI-3413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941190#comment-15941190
 ] 

ASF GitHub Bot commented on NIFI-3413:
--

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1618#discussion_r107920928
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetChangeDataCaptureMySQL.java
 ---
@@ -0,0 +1,879 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import com.github.shyiko.mysql.binlog.BinaryLogClient;
+import com.github.shyiko.mysql.binlog.event.Event;
+import com.github.shyiko.mysql.binlog.event.EventHeaderV4;
+import com.github.shyiko.mysql.binlog.event.EventType;
+import com.github.shyiko.mysql.binlog.event.QueryEventData;
+import com.github.shyiko.mysql.binlog.event.RotateEventData;
+import com.github.shyiko.mysql.binlog.event.TableMapEventData;
+import org.apache.nifi.annotation.behavior.DynamicProperties;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateManager;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.standard.db.CDCException;
+import org.apache.nifi.processors.standard.db.event.ColumnDefinition;
+import org.apache.nifi.processors.standard.db.event.RowEventException;
+import org.apache.nifi.processors.standard.db.event.TableInfo;
+import org.apache.nifi.processors.standard.db.event.TableInfoCacheKey;
+import org.apache.nifi.processors.standard.db.event.io.EventWriter;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.BeginTransactionEventInfo;
+import org.apache.nifi.processors.standard.db.impl.mysql.RawBinlogEvent;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.BinlogEventListener;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.BinlogEventInfo;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.CommitTransactionEventInfo;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.DeleteRowsEventInfo;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.SchemaChangeEventInfo;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.UpdateRowsEventInfo;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.InsertRowsEventInfo;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.io.BeginTransactionEventWriter;
+import 
org.apache.nifi.processors.st

[GitHub] nifi pull request #1618: NIFI-3413: Add GetChangeDataCaptureMySQL processor

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1618#discussion_r107921330
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetChangeDataCaptureMySQL.java
 ---
@@ -0,0 +1,879 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import com.github.shyiko.mysql.binlog.BinaryLogClient;
+import com.github.shyiko.mysql.binlog.event.Event;
+import com.github.shyiko.mysql.binlog.event.EventHeaderV4;
+import com.github.shyiko.mysql.binlog.event.EventType;
+import com.github.shyiko.mysql.binlog.event.QueryEventData;
+import com.github.shyiko.mysql.binlog.event.RotateEventData;
+import com.github.shyiko.mysql.binlog.event.TableMapEventData;
+import org.apache.nifi.annotation.behavior.DynamicProperties;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateManager;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.standard.db.CDCException;
+import org.apache.nifi.processors.standard.db.event.ColumnDefinition;
+import org.apache.nifi.processors.standard.db.event.RowEventException;
+import org.apache.nifi.processors.standard.db.event.TableInfo;
+import org.apache.nifi.processors.standard.db.event.TableInfoCacheKey;
+import org.apache.nifi.processors.standard.db.event.io.EventWriter;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.BeginTransactionEventInfo;
+import org.apache.nifi.processors.standard.db.impl.mysql.RawBinlogEvent;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.BinlogEventListener;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.BinlogEventInfo;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.CommitTransactionEventInfo;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.DeleteRowsEventInfo;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.SchemaChangeEventInfo;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.UpdateRowsEventInfo;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.InsertRowsEventInfo;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.io.BeginTransactionEventWriter;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.io.CommitTransactionEventWriter;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.io.DeleteRowsWriter;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.io.InsertRowsWriter;
+import

[GitHub] nifi pull request #1618: NIFI-3413: Add GetChangeDataCaptureMySQL processor

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1618#discussion_r107909853
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/event/TableInfo.java
 ---
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.db.event;
+
+import 
org.apache.nifi.distributed.cache.client.exception.DeserializationException;
+import 
org.apache.nifi.distributed.cache.client.exception.SerializationException;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * A POJO for holding table information related to update events.
+ */
+public class TableInfo {
+
+final static String DB_TABLE_NAME_DELIMITER = "@!@";
+
+private String databaseName;
+private String tableName;
+private Long tableId;
+private List columns;
+
+public TableInfo(String databaseName, String tableName, Long tableId, 
List columns) {
+this.databaseName = databaseName;
+this.tableName = tableName;
+this.tableId = tableId;
+this.columns = columns;
+}
+
+public String getDatabaseName() {
+return databaseName;
+}
+
+public String getTableName() {
+return tableName;
+}
+
+public void setTableName(String tableName) {
+this.tableName = tableName;
+}
+
+public Long getTableId() {
+return tableId;
+}
+
+public List getColumns() {
+return columns;
+}
+
+public void setColumns(List columns) {
+this.columns = columns;
+}
+
+@Override
+public boolean equals(Object o) {
+if (this == o) return true;
+if (o == null || getClass() != o.getClass()) return false;
+
+TableInfo tableInfo = (TableInfo) o;
+
+if (!databaseName.equals(tableInfo.databaseName)) return false;
+if (!tableName.equals(tableInfo.tableName)) return false;
+if (!tableId.equals(tableInfo.tableId)) return false;
+return columns != null ? columns.equals(tableInfo.columns) : 
tableInfo.columns == null;
+}
+
+@Override
+public int hashCode() {
+int result = databaseName.hashCode();
+result = 31 * result + tableName.hashCode();
+result = 31 * result + tableId.hashCode();
+result = 31 * result + (columns != null ? columns.hashCode() : 0);
+return result;
+}
+
+public static class Serializer implements 
org.apache.nifi.distributed.cache.client.Serializer {
+
+@Override
+public void serialize(TableInfo value, OutputStream output) throws 
SerializationException, IOException {
+StringBuilder sb = new StringBuilder(value.getDatabaseName());
+sb.append(DB_TABLE_NAME_DELIMITER);
+sb.append(value.getTableName());
+sb.append(DB_TABLE_NAME_DELIMITER);
+sb.append(value.getTableId());
+List columnDefinitions = value.getColumns();
+if (columnDefinitions != null && !columnDefinitions.isEmpty()) 
{
+sb.append(DB_TABLE_NAME_DELIMITER);
+sb.append(columnDefinitions.stream().map((col) -> 
col.getName() + DB_TABLE_NAME_DELIMITER + 
col.getType()).collect(Collectors.joining(DB_TABLE_NAME_DELIMITER)));
+}
+output.write(sb.toString().getBytes());
+}
+}
+
+public static class Deserializer implements 
org.apache.nifi.distributed.cache.client.Deserializer {
+
+@Override
+public TableInfo deserialize(byte[] input) throws 
DeserializationException, IOException {
+// Don

[jira] [Commented] (NIFI-3413) Implement a GetChangeDataCapture processor


[ 
https://issues.apache.org/jira/browse/NIFI-3413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941186#comment-15941186
 ] 

ASF GitHub Bot commented on NIFI-3413:
--

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1618#discussion_r107912643
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/impl/mysql/event/io/AbstractBinlogTableEventWriter.java
 ---
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.db.impl.mysql.event.io;
+
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processors.standard.GetChangeDataCaptureMySQL;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.BinlogTableEventInfo;
+
+import java.io.IOException;
+
+/**
+ * An abstract base class for writing MYSQL binlog events into flow 
file(s), e.g.
+ */
+public abstract class AbstractBinlogTableEventWriter extends AbstractBinlogEventWriter {
+
+protected void writeJson(T event) throws IOException {
+super.writeJson(event);
+if (event.getDatabaseName() != null) {
+jsonGenerator.writeStringField("database", 
event.getDatabaseName());
+} else {
+jsonGenerator.writeNullField("database");
+}
+if (event.getTableName() != null) {
+jsonGenerator.writeStringField("table_name", 
event.getTableName());
+} else {
+jsonGenerator.writeNullField("table_name");
+}
+if (event.getTableId() != null) {
+jsonGenerator.writeNumberField("table_id", event.getTableId());
+} else {
+jsonGenerator.writeNullField("table_id");
+}
+}
+
+// Default implementation for binlog events
+@Override
+public long writeEvent(ProcessSession session, T eventInfo, long 
currentSequenceId) {
+FlowFile flowFile = session.create();
+flowFile = session.write(flowFile, (outputStream) -> {
+super.startJson(outputStream, eventInfo);
+super.writeJson(eventInfo);
--- End diff --

Isn't the super.writeJson(event) encapsulaed by 
AbstractBinlogEventWriter.writeEvent(...) when it calls writeJson(T)?


> Implement a GetChangeDataCapture processor
> --
>
> Key: NIFI-3413
> URL: https://issues.apache.org/jira/browse/NIFI-3413
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Database systems such as MySQL, Oracle, and SQL Server allow access to their 
> transactional logs and such, in order for external clients to have a "change 
> data capture" (CDC) capability. I propose a GetChangeDataCapture processor to 
> enable this in NiFi.
> The processor would be configured with a DBCPConnectionPool controller 
> service, as well as a Database Type property (similar to the one in 
> QueryDatabaseTable) for database-specific handling. Additional properties 
> might include the CDC table name, etc.  Additional database-specific 
> properties could be handled using dynamic properties (and the documentation 
> should reflect this).
> The processor would accept no incoming connections (it is a "Get" or source 
> processor), would be intended to run on the primary node only as a single 
> threaded processor, and would generate a flow file for each operation 
> (INSERT, UPDATE, DELETE, e,g,) in one or some number of formats (JSON, e.g.). 
> The flow files would be transferred in time order (to enable a replication 
> solution, for example), perhaps with some auto-incrementing attribute to also 
> indicate order if need be.



--
This message was sent by Atlass

[GitHub] nifi pull request #1618: NIFI-3413: Add GetChangeDataCaptureMySQL processor

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1618#discussion_r107914548
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/impl/mysql/event/io/DeleteRowsWriter.java
 ---
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.db.impl.mysql.event.io;
+
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processors.standard.GetChangeDataCaptureMySQL;
+import org.apache.nifi.processors.standard.db.event.ColumnDefinition;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.DeleteRowsEventInfo;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.BitSet;
+import java.util.concurrent.atomic.AtomicLong;
+
+import static 
org.apache.nifi.processors.standard.db.impl.mysql.MySQLCDCUtils.getWritableObject;
+
+/**
+ * A writer class to output MySQL binlog "delete rows" events to flow 
file(s).
+ */
+public class DeleteRowsWriter extends 
AbstractBinlogTableEventWriter {
+
+/**
+ * Creates and transfers a new flow file whose contents are the 
JSON-serialized value of the specified event, and the sequence ID attribute set
+ *
+ * @param session   A reference to a ProcessSession from which the 
flow file(s) will be created and transferred
+ * @param eventInfo An event whose value will become the contents of 
the flow file
+ * @return The next available CDC sequence ID for use by the CDC 
processor
+ */
+public long writeEvent(final ProcessSession session, final 
DeleteRowsEventInfo eventInfo, final long currentSequenceId) {
+final AtomicLong seqId = new AtomicLong(currentSequenceId);
--- End diff --

Why use atomic long? Seems to me that CAS is unnecessary unless I'm missing 
something. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3413) Implement a GetChangeDataCapture processor


[ 
https://issues.apache.org/jira/browse/NIFI-3413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941185#comment-15941185
 ] 

ASF GitHub Bot commented on NIFI-3413:
--

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1618#discussion_r107922204
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetChangeDataCaptureMySQL.java
 ---
@@ -0,0 +1,879 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import com.github.shyiko.mysql.binlog.BinaryLogClient;
+import com.github.shyiko.mysql.binlog.event.Event;
+import com.github.shyiko.mysql.binlog.event.EventHeaderV4;
+import com.github.shyiko.mysql.binlog.event.EventType;
+import com.github.shyiko.mysql.binlog.event.QueryEventData;
+import com.github.shyiko.mysql.binlog.event.RotateEventData;
+import com.github.shyiko.mysql.binlog.event.TableMapEventData;
+import org.apache.nifi.annotation.behavior.DynamicProperties;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateManager;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.standard.db.CDCException;
+import org.apache.nifi.processors.standard.db.event.ColumnDefinition;
+import org.apache.nifi.processors.standard.db.event.RowEventException;
+import org.apache.nifi.processors.standard.db.event.TableInfo;
+import org.apache.nifi.processors.standard.db.event.TableInfoCacheKey;
+import org.apache.nifi.processors.standard.db.event.io.EventWriter;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.BeginTransactionEventInfo;
+import org.apache.nifi.processors.standard.db.impl.mysql.RawBinlogEvent;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.BinlogEventListener;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.BinlogEventInfo;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.CommitTransactionEventInfo;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.DeleteRowsEventInfo;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.SchemaChangeEventInfo;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.UpdateRowsEventInfo;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.InsertRowsEventInfo;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.io.BeginTransactionEventWriter;
+import 
org.apache.nifi.processors.st

[jira] [Commented] (NIFI-3413) Implement a GetChangeDataCapture processor


[ 
https://issues.apache.org/jira/browse/NIFI-3413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941191#comment-15941191
 ] 

ASF GitHub Bot commented on NIFI-3413:
--

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1618#discussion_r107914548
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/impl/mysql/event/io/DeleteRowsWriter.java
 ---
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.db.impl.mysql.event.io;
+
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processors.standard.GetChangeDataCaptureMySQL;
+import org.apache.nifi.processors.standard.db.event.ColumnDefinition;
+import 
org.apache.nifi.processors.standard.db.impl.mysql.event.DeleteRowsEventInfo;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.BitSet;
+import java.util.concurrent.atomic.AtomicLong;
+
+import static 
org.apache.nifi.processors.standard.db.impl.mysql.MySQLCDCUtils.getWritableObject;
+
+/**
+ * A writer class to output MySQL binlog "delete rows" events to flow 
file(s).
+ */
+public class DeleteRowsWriter extends 
AbstractBinlogTableEventWriter {
+
+/**
+ * Creates and transfers a new flow file whose contents are the 
JSON-serialized value of the specified event, and the sequence ID attribute set
+ *
+ * @param session   A reference to a ProcessSession from which the 
flow file(s) will be created and transferred
+ * @param eventInfo An event whose value will become the contents of 
the flow file
+ * @return The next available CDC sequence ID for use by the CDC 
processor
+ */
+public long writeEvent(final ProcessSession session, final 
DeleteRowsEventInfo eventInfo, final long currentSequenceId) {
+final AtomicLong seqId = new AtomicLong(currentSequenceId);
--- End diff --

Why use atomic long? Seems to me that CAS is unnecessary unless I'm missing 
something. 


> Implement a GetChangeDataCapture processor
> --
>
> Key: NIFI-3413
> URL: https://issues.apache.org/jira/browse/NIFI-3413
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Database systems such as MySQL, Oracle, and SQL Server allow access to their 
> transactional logs and such, in order for external clients to have a "change 
> data capture" (CDC) capability. I propose a GetChangeDataCapture processor to 
> enable this in NiFi.
> The processor would be configured with a DBCPConnectionPool controller 
> service, as well as a Database Type property (similar to the one in 
> QueryDatabaseTable) for database-specific handling. Additional properties 
> might include the CDC table name, etc.  Additional database-specific 
> properties could be handled using dynamic properties (and the documentation 
> should reflect this).
> The processor would accept no incoming connections (it is a "Get" or source 
> processor), would be intended to run on the primary node only as a single 
> threaded processor, and would generate a flow file for each operation 
> (INSERT, UPDATE, DELETE, e,g,) in one or some number of formats (JSON, e.g.). 
> The flow files would be transferred in time order (to enable a replication 
> solution, for example), perhaps with some auto-incrementing attribute to also 
> indicate order if need be.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1618: NIFI-3413: Add GetChangeDataCaptureMySQL processor