[GitHub] nifi pull request #3179: NIFI-5834: Restore default PutHiveQL error handling...
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3179#discussion_r236571657
--- Diff:
nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors/src/main/java/org/apache/nifi/processors/hive/PutHive3QL.java
---
@@ -148,7 +148,23 @@ public void constructProcess() {
if (e instanceof SQLNonTransientException) {
return ErrorTypes.InvalidInput;
} else if (e instanceof SQLException) {
-return ErrorTypes.TemporalFailure;
+// Use the SQLException's vendor code for guidance -- see
Hive's ErrorMsg class for details on error codes
+int errorCode = ((SQLException) e).getErrorCode();
--- End diff --
I think that's a good idea :)
---
[jira] [Commented] (NIFI-5834) Restore default PutHiveQL error handling behavior
[
https://issues.apache.org/jira/browse/NIFI-5834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700095#comment-16700095
]
ASF GitHub Bot commented on NIFI-5834:
--
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3179#discussion_r236571657
--- Diff:
nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors/src/main/java/org/apache/nifi/processors/hive/PutHive3QL.java
---
@@ -148,7 +148,23 @@ public void constructProcess() {
if (e instanceof SQLNonTransientException) {
return ErrorTypes.InvalidInput;
} else if (e instanceof SQLException) {
-return ErrorTypes.TemporalFailure;
+// Use the SQLException's vendor code for guidance -- see
Hive's ErrorMsg class for details on error codes
+int errorCode = ((SQLException) e).getErrorCode();
--- End diff --
I think that's a good idea :)
> Restore default PutHiveQL error handling behavior
> -
>
> Key: NIFI-5834
> URL: https://issues.apache.org/jira/browse/NIFI-5834
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> As part of adding Rollback On Failure to PutHiveQL (via NIFI-3415), the code
> was refactored to allow failures to be rolled back rather than transferred to
> the failure relationship (if Rollback On Failure is set). As part of that,
> all transient SQLExceptions were declared to be of type "Temporal Failure".
> This (along with the refactor) allowed the failures to be handled as
> rollbacks or transfers as specified.
> Hive returns all exceptions as transient SQLExceptions, with an error code
> that better infers the behavior of the operation. This, via the discovery of
> NIFI-5045, resulted in the handling of error codes within the Hive error code
> range. However the default behavior when the error code is not in the
> Hive-valid range is to rollback regardless of whether Rollback On Failure is
> true or not. This was done as a "better safe than sorry" approach, but it
> made the behavior inconsistent with earlier versions of the processor, where
> failures were simply routed to failure rather than rolling back.
> This case proposes to return the default behavior for unknown SQLExceptions
> to "TemporalFailure", which will make the behavior consistent with previous
> versions of the processor, where unknown errors will be transferred to
> failure unless Rollback on Failure is true.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5834) Restore default PutHiveQL error handling behavior
[ https://issues.apache.org/jira/browse/NIFI-5834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700098#comment-16700098 ] ASF subversion and git services commented on NIFI-5834: --- Commit 455e3c1bc8d4e12ea75d4e6ac2e4c58cbb535e5d in nifi's branch refs/heads/master from [~ca9mbu] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=455e3c1 ] NIFI-5834: Restore default PutHiveQL error handling behavior NIFI-5834: Incorporated review comments This closes #3179. Signed-off-by: Koji Kawamura > Restore default PutHiveQL error handling behavior > - > > Key: NIFI-5834 > URL: https://issues.apache.org/jira/browse/NIFI-5834 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > As part of adding Rollback On Failure to PutHiveQL (via NIFI-3415), the code > was refactored to allow failures to be rolled back rather than transferred to > the failure relationship (if Rollback On Failure is set). As part of that, > all transient SQLExceptions were declared to be of type "Temporal Failure". > This (along with the refactor) allowed the failures to be handled as > rollbacks or transfers as specified. > Hive returns all exceptions as transient SQLExceptions, with an error code > that better infers the behavior of the operation. This, via the discovery of > NIFI-5045, resulted in the handling of error codes within the Hive error code > range. However the default behavior when the error code is not in the > Hive-valid range is to rollback regardless of whether Rollback On Failure is > true or not. This was done as a "better safe than sorry" approach, but it > made the behavior inconsistent with earlier versions of the processor, where > failures were simply routed to failure rather than rolling back. > This case proposes to return the default behavior for unknown SQLExceptions > to "TemporalFailure", which will make the behavior consistent with previous > versions of the processor, where unknown errors will be transferred to > failure unless Rollback on Failure is true. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3179: NIFI-5834: Restore default PutHiveQL error handling...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3179 ---
[jira] [Commented] (NIFI-5834) Restore default PutHiveQL error handling behavior
[ https://issues.apache.org/jira/browse/NIFI-5834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700099#comment-16700099 ] ASF subversion and git services commented on NIFI-5834: --- Commit 455e3c1bc8d4e12ea75d4e6ac2e4c58cbb535e5d in nifi's branch refs/heads/master from [~ca9mbu] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=455e3c1 ] NIFI-5834: Restore default PutHiveQL error handling behavior NIFI-5834: Incorporated review comments This closes #3179. Signed-off-by: Koji Kawamura > Restore default PutHiveQL error handling behavior > - > > Key: NIFI-5834 > URL: https://issues.apache.org/jira/browse/NIFI-5834 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > As part of adding Rollback On Failure to PutHiveQL (via NIFI-3415), the code > was refactored to allow failures to be rolled back rather than transferred to > the failure relationship (if Rollback On Failure is set). As part of that, > all transient SQLExceptions were declared to be of type "Temporal Failure". > This (along with the refactor) allowed the failures to be handled as > rollbacks or transfers as specified. > Hive returns all exceptions as transient SQLExceptions, with an error code > that better infers the behavior of the operation. This, via the discovery of > NIFI-5045, resulted in the handling of error codes within the Hive error code > range. However the default behavior when the error code is not in the > Hive-valid range is to rollback regardless of whether Rollback On Failure is > true or not. This was done as a "better safe than sorry" approach, but it > made the behavior inconsistent with earlier versions of the processor, where > failures were simply routed to failure rather than rolling back. > This case proposes to return the default behavior for unknown SQLExceptions > to "TemporalFailure", which will make the behavior consistent with previous > versions of the processor, where unknown errors will be transferred to > failure unless Rollback on Failure is true. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5834) Restore default PutHiveQL error handling behavior
[ https://issues.apache.org/jira/browse/NIFI-5834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700101#comment-16700101 ] ASF GitHub Bot commented on NIFI-5834: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3179 > Restore default PutHiveQL error handling behavior > - > > Key: NIFI-5834 > URL: https://issues.apache.org/jira/browse/NIFI-5834 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.9.0 > > > As part of adding Rollback On Failure to PutHiveQL (via NIFI-3415), the code > was refactored to allow failures to be rolled back rather than transferred to > the failure relationship (if Rollback On Failure is set). As part of that, > all transient SQLExceptions were declared to be of type "Temporal Failure". > This (along with the refactor) allowed the failures to be handled as > rollbacks or transfers as specified. > Hive returns all exceptions as transient SQLExceptions, with an error code > that better infers the behavior of the operation. This, via the discovery of > NIFI-5045, resulted in the handling of error codes within the Hive error code > range. However the default behavior when the error code is not in the > Hive-valid range is to rollback regardless of whether Rollback On Failure is > true or not. This was done as a "better safe than sorry" approach, but it > made the behavior inconsistent with earlier versions of the processor, where > failures were simply routed to failure rather than rolling back. > This case proposes to return the default behavior for unknown SQLExceptions > to "TemporalFailure", which will make the behavior consistent with previous > versions of the processor, where unknown errors will be transferred to > failure unless Rollback on Failure is true. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5834) Restore default PutHiveQL error handling behavior
[ https://issues.apache.org/jira/browse/NIFI-5834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Kawamura updated NIFI-5834: Resolution: Fixed Fix Version/s: 1.9.0 Status: Resolved (was: Patch Available) > Restore default PutHiveQL error handling behavior > - > > Key: NIFI-5834 > URL: https://issues.apache.org/jira/browse/NIFI-5834 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.9.0 > > > As part of adding Rollback On Failure to PutHiveQL (via NIFI-3415), the code > was refactored to allow failures to be rolled back rather than transferred to > the failure relationship (if Rollback On Failure is set). As part of that, > all transient SQLExceptions were declared to be of type "Temporal Failure". > This (along with the refactor) allowed the failures to be handled as > rollbacks or transfers as specified. > Hive returns all exceptions as transient SQLExceptions, with an error code > that better infers the behavior of the operation. This, via the discovery of > NIFI-5045, resulted in the handling of error codes within the Hive error code > range. However the default behavior when the error code is not in the > Hive-valid range is to rollback regardless of whether Rollback On Failure is > true or not. This was done as a "better safe than sorry" approach, but it > made the behavior inconsistent with earlier versions of the processor, where > failures were simply routed to failure rather than rolling back. > This case proposes to return the default behavior for unknown SQLExceptions > to "TemporalFailure", which will make the behavior consistent with previous > versions of the processor, where unknown errors will be transferred to > failure unless Rollback on Failure is true. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (MINIFICPP-686) Move static property reads to onSchedule function of processors.
Arpad Boda created MINIFICPP-686: Summary: Move static property reads to onSchedule function of processors. Key: MINIFICPP-686 URL: https://issues.apache.org/jira/browse/MINIFICPP-686 Project: NiFi MiNiFi C++ Issue Type: Improvement Reporter: Arpad Boda Assignee: Arpad Boda Fix For: 0.6.0 As [~phrocker] pointed out in scope of [https://github.com/apache/nifi-minifi-cpp/pull/445] , read and process of static property values should be done in onSchedule function of processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #450: MINIFICPP-686 - Move static property read...
GitHub user arpadboda opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/450 MINIFICPP-686 - Move static property reads to onSchedule function of ⦠â¦processors. Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/arpadboda/nifi-minifi-cpp MINIFICPP-686 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/450.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #450 commit 851e1f2a3c9bcd4fd7c8fd7db5cbbe720258dc46 Author: Arpad Boda Date: 2018-11-27T12:29:34Z MINIFICPP-686 - Move static property reads to onSchedule function of processors. ---
[jira] [Commented] (MINIFICPP-686) Move static property reads to onSchedule function of processors.
[ https://issues.apache.org/jira/browse/MINIFICPP-686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700343#comment-16700343 ] ASF GitHub Bot commented on MINIFICPP-686: -- GitHub user arpadboda opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/450 MINIFICPP-686 - Move static property reads to onSchedule function of … …processors. Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/arpadboda/nifi-minifi-cpp MINIFICPP-686 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/450.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #450 commit 851e1f2a3c9bcd4fd7c8fd7db5cbbe720258dc46 Author: Arpad Boda Date: 2018-11-27T12:29:34Z MINIFICPP-686 - Move static property reads to onSchedule function of processors. > Move static property reads to onSchedule function of processors. > > > Key: MINIFICPP-686 > URL: https://issues.apache.org/jira/browse/MINIFICPP-686 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Arpad Boda >Assignee: Arpad Boda >Priority: Minor > Fix For: 0.6.0 > > > As [~phrocker] pointed out in scope of > [https://github.com/apache/nifi-minifi-cpp/pull/445] , read and process of > static property values should be done in onSchedule function of processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #450: MINIFICPP-686 - Move static property read...
Github user arpadboda closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/450 ---
[jira] [Commented] (MINIFICPP-686) Move static property reads to onSchedule function of processors.
[ https://issues.apache.org/jira/browse/MINIFICPP-686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700379#comment-16700379 ] ASF GitHub Bot commented on MINIFICPP-686: -- Github user arpadboda closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/450 > Move static property reads to onSchedule function of processors. > > > Key: MINIFICPP-686 > URL: https://issues.apache.org/jira/browse/MINIFICPP-686 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Arpad Boda >Assignee: Arpad Boda >Priority: Minor > Fix For: 0.6.0 > > > As [~phrocker] pointed out in scope of > [https://github.com/apache/nifi-minifi-cpp/pull/445] , read and process of > static property values should be done in onSchedule function of processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #449: MINIFICPP-685 - Improve const correctness...
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/449#discussion_r236664502
--- Diff: libminifi/include/core/ConfigurableComponent.h ---
@@ -153,14 +153,14 @@ class ConfigurableComponent {
*
* @return vector of property keys
*/
- std::vector getDynamicPropertyKeys();
+ std::vector getDynamicPropertyKeys() const;
/**
* Returns a vector all properties
*
* @return map of property keys to Property instances.
*/
- std::map getProperties();
+ std::map getProperties() const;
--- End diff --
Did you not change to const ref map because current implementations expect
to modify this map?
---
[jira] [Commented] (MINIFICPP-685) Improve const correctness of property getters
[
https://issues.apache.org/jira/browse/MINIFICPP-685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700427#comment-16700427
]
ASF GitHub Bot commented on MINIFICPP-685:
--
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/449#discussion_r236664502
--- Diff: libminifi/include/core/ConfigurableComponent.h ---
@@ -153,14 +153,14 @@ class ConfigurableComponent {
*
* @return vector of property keys
*/
- std::vector getDynamicPropertyKeys();
+ std::vector getDynamicPropertyKeys() const;
/**
* Returns a vector all properties
*
* @return map of property keys to Property instances.
*/
- std::map getProperties();
+ std::map getProperties() const;
--- End diff --
Did you not change to const ref map because current implementations expect
to modify this map?
> Improve const correctness of property getters
> -
>
> Key: MINIFICPP-685
> URL: https://issues.apache.org/jira/browse/MINIFICPP-685
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Arpad Boda
>Assignee: Arpad Boda
>Priority: Minor
> Fix For: 0.6.0
>
>
> Most of property getters are non-const, but they should be.
> Virtual functions shouldn't be modified in minor release, the rest can be
> fixed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #449: MINIFICPP-685 - Improve const correctness...
Github user arpadboda commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/449#discussion_r236665802
--- Diff: libminifi/include/core/ConfigurableComponent.h ---
@@ -153,14 +153,14 @@ class ConfigurableComponent {
*
* @return vector of property keys
*/
- std::vector getDynamicPropertyKeys();
+ std::vector getDynamicPropertyKeys() const;
/**
* Returns a vector all properties
*
* @return map of property keys to Property instances.
*/
- std::map getProperties();
+ std::map getProperties() const;
--- End diff --
Nope, the map is created upon execution, (a merge of dynamic and static
properties), so there is no real point in returning const ref.
---
[jira] [Commented] (MINIFICPP-685) Improve const correctness of property getters
[
https://issues.apache.org/jira/browse/MINIFICPP-685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700433#comment-16700433
]
ASF GitHub Bot commented on MINIFICPP-685:
--
Github user arpadboda commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/449#discussion_r236665802
--- Diff: libminifi/include/core/ConfigurableComponent.h ---
@@ -153,14 +153,14 @@ class ConfigurableComponent {
*
* @return vector of property keys
*/
- std::vector getDynamicPropertyKeys();
+ std::vector getDynamicPropertyKeys() const;
/**
* Returns a vector all properties
*
* @return map of property keys to Property instances.
*/
- std::map getProperties();
+ std::map getProperties() const;
--- End diff --
Nope, the map is created upon execution, (a merge of dynamic and static
properties), so there is no real point in returning const ref.
> Improve const correctness of property getters
> -
>
> Key: MINIFICPP-685
> URL: https://issues.apache.org/jira/browse/MINIFICPP-685
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Arpad Boda
>Assignee: Arpad Boda
>Priority: Minor
> Fix For: 0.6.0
>
>
> Most of property getters are non-const, but they should be.
> Virtual functions shouldn't be modified in minor release, the rest can be
> fixed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (MINIFICPP-268) Implement ManipulateArchive processor
[ https://issues.apache.org/jira/browse/MINIFICPP-268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mr TheSegfault resolved MINIFICPP-268. -- Resolution: Fixed > Implement ManipulateArchive processor > - > > Key: MINIFICPP-268 > URL: https://issues.apache.org/jira/browse/MINIFICPP-268 > Project: NiFi MiNiFi C++ > Issue Type: New Feature >Reporter: Caleb Johnson >Assignee: Caleb Johnson >Priority: Major > Fix For: 0.3.0 > > > The ManipulateArchive processor extends Un/FocusArchive by allowing users to > perform the following operations on archives supported by same: > * Touch: create an empty file at the specified path; > * Remove: repacks an archive, but skips the specified entry when rebuilding; > * Move: rename an archive entry; > * Copy: duplicate an archive entry to another path. > Touch, move and copy operations take optional 'before' and 'after' parameters > to control where the modified entry is to be placed relative to an existing > one. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MINIFICPP-684) ExtractText processor doesn't handle "Size limit" property
[ https://issues.apache.org/jira/browse/MINIFICPP-684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700444#comment-16700444 ] ASF GitHub Bot commented on MINIFICPP-684: -- Github user phrocker commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/446 > > @arpadboda I'll update the commit on merge to what was intended if you don't have a chance to in the next few minutes. > > Done. I had some issues with pushing to apache's git servers last week. Will try again today, sorry for the delay. > ExtractText processor doesn't handle "Size limit" property > -- > > Key: MINIFICPP-684 > URL: https://issues.apache.org/jira/browse/MINIFICPP-684 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Arpad Boda >Assignee: Arpad Boda >Priority: Major > Fix For: 0.6.0 > > > Size limit property is ignored (cannot even be set), the whole content is > extracted to the attribute. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp issue #446: MINIFICPP-684 - ExtractText processor doesn't ha...
Github user phrocker commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/446 > > @arpadboda I'll update the commit on merge to what was intended if you don't have a chance to in the next few minutes. > > Done. I had some issues with pushing to apache's git servers last week. Will try again today, sorry for the delay. ---
[GitHub] nifi-minifi-cpp pull request #446: MINIFICPP-684 - ExtractText processor doe...
Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/446 ---
[jira] [Commented] (MINIFICPP-684) ExtractText processor doesn't handle "Size limit" property
[ https://issues.apache.org/jira/browse/MINIFICPP-684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700448#comment-16700448 ] ASF GitHub Bot commented on MINIFICPP-684: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/446 > ExtractText processor doesn't handle "Size limit" property > -- > > Key: MINIFICPP-684 > URL: https://issues.apache.org/jira/browse/MINIFICPP-684 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Arpad Boda >Assignee: Arpad Boda >Priority: Major > Fix For: 0.6.0 > > > Size limit property is ignored (cannot even be set), the whole content is > extracted to the attribute. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5843) Unclear validation message for ScriptingComponentHelper
Alex Savitsky created NIFI-5843:
---
Summary: Unclear validation message for ScriptingComponentHelper
Key: NIFI-5843
URL: https://issues.apache.org/jira/browse/NIFI-5843
Project: Apache NiFi
Issue Type: Improvement
Components: Core Framework
Affects Versions: 1.8.0
Reporter: Alex Savitsky
ScriptingComponentHelper.customValidate (from nifi-scripting-processors)
creates an unclear validation message
{quote}'' is invalid because Exactly one of Script File or Script Body must be
set
{quote}
as it doesn't specify the validation subject. Since there are technically two
invalid subjects in this case (Script File and Script Body), I suggest adding
two validation messages, one for each subject.
Current code:
{code:java}
results.add(new ValidationResult.Builder().valid(false).explanation(
"Exactly one of Script File or Script Body must be set").build());
{code}
Proposed fix:
{code:java}
results.add(new ValidationResult.Builder().subject("Script
Body").valid(false).explanation(
"exactly one of Script File or Script Body must be set").build());
results.add(new ValidationResult.Builder().subject("Script
File").valid(false).explanation(
"exactly one of Script File or Script Body must be set").build());
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #2671: NiFi-5102 - Adding Processors for MarkLogic DB
Github user ryanjdew commented on the issue: https://github.com/apache/nifi/pull/2671 Following up, are there any other concerns with this PR? If needed, I can provide credentials to a MarkLogic instance for testing. ---
[jira] [Resolved] (MINIFICPP-684) ExtractText processor doesn't handle "Size limit" property
[ https://issues.apache.org/jira/browse/MINIFICPP-684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mr TheSegfault resolved MINIFICPP-684. -- Resolution: Fixed > ExtractText processor doesn't handle "Size limit" property > -- > > Key: MINIFICPP-684 > URL: https://issues.apache.org/jira/browse/MINIFICPP-684 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Arpad Boda >Assignee: Arpad Boda >Priority: Major > Fix For: 0.6.0 > > > Size limit property is ignored (cannot even be set), the whole content is > extracted to the attribute. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2671: NiFi-5102 - Adding Processors for MarkLogic DB
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2671 @ryanjdew We have addressed Travis-CI related tests issues and the build has been stable now since. It could break again as people add timing dependent tests that behave wildly different on slow environments but we'll see. This PR, assuming the L&N and such are sorted now, is just tricky because we need a committer with time to devote to learning MarkLogic enough to setup an environment and verify function or leverage a provided instance (not a sustainable model). It is a good example where the limits of what the community can reasonably support comes into play. Now, this said this is probably a really cool and useful thing to offer folks and beneficial to both the NiFi userbase and MarkLogic user base. This is why I suggested MarkLogic folks just hang onto this code in some public github repo and have it be ALv2. They can publish their nars into maven central or wherever they do and provide instructions to it. I'd be supportive, and I'd assume the community at large would, of having links to such extensions on the apache website. This feels to me like the best tradeoff right now for all parties. ---
[GitHub] nifi issue #2671: NiFi-5102 - Adding Processors for MarkLogic DB
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2671 as a good example where we should have though it through more as well we have some cool work done by the InfluxDB folks. They're now wanting to improve it in https://github.com/apache/nifi/pull/2743 But the reality is we just don't have people knowledgeable enough to do reliable code reviews/testing of this. ---
[GitHub] nifi pull request #3183: NIFI-5826 Fix to escaped backslash
GitHub user bdesert opened a pull request: https://github.com/apache/nifi/pull/3183 NIFI-5826 Fix to escaped backslash Fixing escaped backslash by unescaping it in java just before compiling regex. - For a consistency with expression language regex-based evaluators, all record path operators will follow the same escape sequence for regex. Single backslash should be defined as double backslash. Examples: - replaceRegex(/col1, '\\s', "_") - will replace all whitespaces (spaces, tabs) with underscore. - replaceRegex(/col1, '\\.', ",") - will replace all a period with - replaceRegex(/col1, '', "/") - will replace backslash with forward slash - Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/bdesert/nifi NIFI-5826_RegexPath Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3183.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3183 commit 4e0eed56f76556b323f96640d4d3126d35750096 Author: Ed Date: 2018-11-27T17:19:20Z NIFI-5826 Fix to escaped backslash Fixing escaped backslash by unescaping it in java just before compiling regex. For a consistency with expression language regex-based evaluators, all record path operators will follow the same escape sequence for regex. Single backslash should be defined as double backslash. Examples: - replaceRegex(/col1, '\\s', "_") - will replace all whitespaces (spaces, tabs) with underscore. - replaceRegex(/col1, '\\.', ",") - will replace all a period with - replaceRegex(/col1, '', "/") - will replace backslash with forward slash NIFI-5826 Fix to escaped backslash Fixing escaped backslash by unescaping it in java just before compiling regex. For a consistency with expression language regex-based evaluators, all record path operators will follow the same escape sequence for regex. Single backslash should be defined as double backslash. Examples: - replaceRegex(/col1, '\\s', "_") - will replace all whitespaces (spaces, tabs) with underscore. - replaceRegex(/col1, '\\.', ",") - will replace all a period with - replaceRegex(/col1, '', "/") - will replace backslash with forward slash ---
[jira] [Commented] (NIFI-5826) UpdateRecord processor throwing PatternSyntaxException
[ https://issues.apache.org/jira/browse/NIFI-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700808#comment-16700808 ] ASF GitHub Bot commented on NIFI-5826: -- GitHub user bdesert opened a pull request: https://github.com/apache/nifi/pull/3183 NIFI-5826 Fix to escaped backslash Fixing escaped backslash by unescaping it in java just before compiling regex. - For a consistency with expression language regex-based evaluators, all record path operators will follow the same escape sequence for regex. Single backslash should be defined as double backslash. Examples: - replaceRegex(/col1, '\\s', "_") - will replace all whitespaces (spaces, tabs) with underscore. - replaceRegex(/col1, '\\.', ",") - will replace all a period with - replaceRegex(/col1, '', "/") - will replace backslash with forward slash - Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/bdesert/nifi NIFI-5826_RegexPath Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3183.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3183 commit 4e0eed56f76556b323f96640d4d3126d35750096 Author: Ed Date: 2018-11-27T17:19:20Z NIFI-5826 Fix to escaped backslash Fixing escaped backslash by unescaping it in java just before compiling regex. For a consistency with expression language regex-based evaluators, all record path operators will follow the same escape sequence for regex. Single backslash should be defined as double backslash. Examples: - replaceRegex(/col1, '\\s', "_") - will replace all whitespaces (spaces, tabs) with underscore. - replaceRegex(/col1, '\\.', ",") - will replace all a period with - replaceRegex(/col1, '', "/") - will replace backslash with forward slash NIFI-5826 Fix to escaped backslash Fixing escaped backslash by unescaping it in java just before compiling regex. For a consistency with expression language regex-based evaluators, all record path operators will follow the same escape sequence for regex. Single backslash should be defined as double backslash. Examples: - replaceRegex(/col1, '\\s', "_") - will replace all whitespaces (spaces, tabs) with underscore. - replaceRegex(/col1, '\\.', ",") - will replace all a period with - replaceRegex(/col1, '', "/") - will replace backslash with forward slash > UpdateRecord processor throwing PatternSyntaxException > -- > > Key: NIFI-5826 > URL: https://issues.apache.org/jira/browse/NIFI-5826 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.5.0 > Environment: Nifi in docker container >Reporter: ravi kargam >Assignee: Ed Berezitsky >Priority: Minor > Attachments: Upda
[GitHub] nifi issue #2671: NiFi-5102 - Adding Processors for MarkLogic DB
Github user ryanjdew commented on the issue: https://github.com/apache/nifi/pull/2671 @joewitt I understand the concern. We would like to simplify the user experience of using NiFi with MarkLogic processors. We have a current model for creating and releasing NARs via a GitHub releases page, but would you happen to have a good example of a process using Maven publishing with NARs in NiFi? ---
[GitHub] nifi issue #2671: NiFi-5102 - Adding Processors for MarkLogic DB
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2671 If you take a look here https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#core-properties-br and checkout the 'nifi.nar.library.directory' section you can see how someone could be guided to create a directory for custom nars, add that in there, and start up. Once we have support for extensions in the NiFi Registry this will be beautiful/easy. ---
[GitHub] nifi pull request #3183: NIFI-5826 Fix to escaped backslash
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3183#discussion_r236784930
--- Diff:
nifi-commons/nifi-record-path/src/main/java/org/apache/nifi/record/path/util/RecordPathUtils.java
---
@@ -39,4 +39,44 @@ public static String getFirstStringValue(final
RecordPathSegment segment, final
return stringValue;
}
+
+public static String unescapeBackslash(String value) {
+if (value == null || value.isEmpty()) {
+return value;
+}
+// need to escape characters after backslashes
+final StringBuilder sb = new StringBuilder();
+boolean lastCharIsBackslash = false;
+for (int i = 0; i < value.length(); i++) {
+final char c = value.charAt(i);
+
--- End diff --
Is this the entire list that has to be escaped? Maybe you can reference
where that list is specified for maintenance.
---
[GitHub] nifi pull request #3183: NIFI-5826 Fix to escaped backslash
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3183#discussion_r236785266
--- Diff:
nifi-commons/nifi-record-path/src/test/java/org/apache/nifi/record/path/TestRecordPath.java
---
@@ -1008,12 +1008,16 @@ public void testReplaceRegex() {
final List fields = new ArrayList<>();
fields.add(new RecordField("id",
RecordFieldType.INT.getDataType()));
fields.add(new RecordField("name",
RecordFieldType.STRING.getDataType()));
--- End diff --
There should be tests for all the escapable chars
---
[jira] [Commented] (NIFI-5826) UpdateRecord processor throwing PatternSyntaxException
[
https://issues.apache.org/jira/browse/NIFI-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700821#comment-16700821
]
ASF GitHub Bot commented on NIFI-5826:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3183#discussion_r236785266
--- Diff:
nifi-commons/nifi-record-path/src/test/java/org/apache/nifi/record/path/TestRecordPath.java
---
@@ -1008,12 +1008,16 @@ public void testReplaceRegex() {
final List fields = new ArrayList<>();
fields.add(new RecordField("id",
RecordFieldType.INT.getDataType()));
fields.add(new RecordField("name",
RecordFieldType.STRING.getDataType()));
--- End diff --
There should be tests for all the escapable chars
> UpdateRecord processor throwing PatternSyntaxException
> --
>
> Key: NIFI-5826
> URL: https://issues.apache.org/jira/browse/NIFI-5826
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
> Environment: Nifi in docker container
>Reporter: ravi kargam
>Assignee: Ed Berezitsky
>Priority: Minor
> Attachments: UpdateRecord_Config_Exception.JPG
>
>
> with replacement value strategy as Record Path Value,
> I am trying to replace square bracket symbol *[* with parenthesis symbol *(*
> in my employeeName column in my csvReader structure with below syntax
> replaceRegex(/employeeName, "[\[]", "(")
> Processor is throwing following exception.
> RecordPathException: java.util.regex.PatternSyntaxException: Unclosed
> character class near index 4 [\\[]
> It worked fine with other special characters such as \{, }, <, >, ;, _, "
> For double qoute ("), i had to use single escape character, for above listed
> other characters, worked fine without any escape character. Other folks in
> Nifi Slack tried \s, \d, \w, \.
> looks like non of them worked.
> replace function worked for replacing [ and ]characters. didn't test any
> other characters.
> Please address resolve the issue.
> Regards,
> Ravi
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5826) UpdateRecord processor throwing PatternSyntaxException
[
https://issues.apache.org/jira/browse/NIFI-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700822#comment-16700822
]
ASF GitHub Bot commented on NIFI-5826:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3183#discussion_r236784930
--- Diff:
nifi-commons/nifi-record-path/src/main/java/org/apache/nifi/record/path/util/RecordPathUtils.java
---
@@ -39,4 +39,44 @@ public static String getFirstStringValue(final
RecordPathSegment segment, final
return stringValue;
}
+
+public static String unescapeBackslash(String value) {
+if (value == null || value.isEmpty()) {
+return value;
+}
+// need to escape characters after backslashes
+final StringBuilder sb = new StringBuilder();
+boolean lastCharIsBackslash = false;
+for (int i = 0; i < value.length(); i++) {
+final char c = value.charAt(i);
+
--- End diff --
Is this the entire list that has to be escaped? Maybe you can reference
where that list is specified for maintenance.
> UpdateRecord processor throwing PatternSyntaxException
> --
>
> Key: NIFI-5826
> URL: https://issues.apache.org/jira/browse/NIFI-5826
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
> Environment: Nifi in docker container
>Reporter: ravi kargam
>Assignee: Ed Berezitsky
>Priority: Minor
> Attachments: UpdateRecord_Config_Exception.JPG
>
>
> with replacement value strategy as Record Path Value,
> I am trying to replace square bracket symbol *[* with parenthesis symbol *(*
> in my employeeName column in my csvReader structure with below syntax
> replaceRegex(/employeeName, "[\[]", "(")
> Processor is throwing following exception.
> RecordPathException: java.util.regex.PatternSyntaxException: Unclosed
> character class near index 4 [\\[]
> It worked fine with other special characters such as \{, }, <, >, ;, _, "
> For double qoute ("), i had to use single escape character, for above listed
> other characters, worked fine without any escape character. Other folks in
> Nifi Slack tried \s, \d, \w, \.
> looks like non of them worked.
> replace function worked for replacing [ and ]characters. didn't test any
> other characters.
> Please address resolve the issue.
> Regards,
> Ravi
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3183: NIFI-5826 Fix to escaped backslash
Github user bdesert commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3183#discussion_r236789857
--- Diff:
nifi-commons/nifi-record-path/src/main/java/org/apache/nifi/record/path/util/RecordPathUtils.java
---
@@ -39,4 +39,44 @@ public static String getFirstStringValue(final
RecordPathSegment segment, final
return stringValue;
}
+
+public static String unescapeBackslash(String value) {
+if (value == null || value.isEmpty()) {
+return value;
+}
+// need to escape characters after backslashes
+final StringBuilder sb = new StringBuilder();
+boolean lastCharIsBackslash = false;
+for (int i = 0; i < value.length(); i++) {
+final char c = value.charAt(i);
+
--- End diff --
Added to the PR description:
EL References: StringLiteralEvaluator. Special characters (backslash
sequences) will be handled for any string literal value, and will affect the
way regex and escaped chars must be defined. Same logic has been added to
RecordPathUtils.unescapeBackslash.
---
[jira] [Commented] (NIFI-5826) UpdateRecord processor throwing PatternSyntaxException
[
https://issues.apache.org/jira/browse/NIFI-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700840#comment-16700840
]
ASF GitHub Bot commented on NIFI-5826:
--
Github user bdesert commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3183#discussion_r236789857
--- Diff:
nifi-commons/nifi-record-path/src/main/java/org/apache/nifi/record/path/util/RecordPathUtils.java
---
@@ -39,4 +39,44 @@ public static String getFirstStringValue(final
RecordPathSegment segment, final
return stringValue;
}
+
+public static String unescapeBackslash(String value) {
+if (value == null || value.isEmpty()) {
+return value;
+}
+// need to escape characters after backslashes
+final StringBuilder sb = new StringBuilder();
+boolean lastCharIsBackslash = false;
+for (int i = 0; i < value.length(); i++) {
+final char c = value.charAt(i);
+
--- End diff --
Added to the PR description:
EL References: StringLiteralEvaluator. Special characters (backslash
sequences) will be handled for any string literal value, and will affect the
way regex and escaped chars must be defined. Same logic has been added to
RecordPathUtils.unescapeBackslash.
> UpdateRecord processor throwing PatternSyntaxException
> --
>
> Key: NIFI-5826
> URL: https://issues.apache.org/jira/browse/NIFI-5826
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
> Environment: Nifi in docker container
>Reporter: ravi kargam
>Assignee: Ed Berezitsky
>Priority: Minor
> Attachments: UpdateRecord_Config_Exception.JPG
>
>
> with replacement value strategy as Record Path Value,
> I am trying to replace square bracket symbol *[* with parenthesis symbol *(*
> in my employeeName column in my csvReader structure with below syntax
> replaceRegex(/employeeName, "[\[]", "(")
> Processor is throwing following exception.
> RecordPathException: java.util.regex.PatternSyntaxException: Unclosed
> character class near index 4 [\\[]
> It worked fine with other special characters such as \{, }, <, >, ;, _, "
> For double qoute ("), i had to use single escape character, for above listed
> other characters, worked fine without any escape character. Other folks in
> Nifi Slack tried \s, \d, \w, \.
> looks like non of them worked.
> replace function worked for replacing [ and ]characters. didn't test any
> other characters.
> Please address resolve the issue.
> Regards,
> Ravi
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3183: NIFI-5826 Fix to escaped backslash
Github user bdesert commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3183#discussion_r236829982
--- Diff:
nifi-commons/nifi-record-path/src/test/java/org/apache/nifi/record/path/TestRecordPath.java
---
@@ -1008,12 +1008,16 @@ public void testReplaceRegex() {
final List fields = new ArrayList<>();
fields.add(new RecordField("id",
RecordFieldType.INT.getDataType()));
fields.add(new RecordField("name",
RecordFieldType.STRING.getDataType()));
--- End diff --
added tests to cover all of special chars. Also added backslash escape
sequences to test exceptions.
---
[jira] [Commented] (NIFI-5826) UpdateRecord processor throwing PatternSyntaxException
[
https://issues.apache.org/jira/browse/NIFI-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700955#comment-16700955
]
ASF GitHub Bot commented on NIFI-5826:
--
Github user bdesert commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3183#discussion_r236829982
--- Diff:
nifi-commons/nifi-record-path/src/test/java/org/apache/nifi/record/path/TestRecordPath.java
---
@@ -1008,12 +1008,16 @@ public void testReplaceRegex() {
final List fields = new ArrayList<>();
fields.add(new RecordField("id",
RecordFieldType.INT.getDataType()));
fields.add(new RecordField("name",
RecordFieldType.STRING.getDataType()));
--- End diff --
added tests to cover all of special chars. Also added backslash escape
sequences to test exceptions.
> UpdateRecord processor throwing PatternSyntaxException
> --
>
> Key: NIFI-5826
> URL: https://issues.apache.org/jira/browse/NIFI-5826
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
> Environment: Nifi in docker container
>Reporter: ravi kargam
>Assignee: Ed Berezitsky
>Priority: Minor
> Attachments: UpdateRecord_Config_Exception.JPG
>
>
> with replacement value strategy as Record Path Value,
> I am trying to replace square bracket symbol *[* with parenthesis symbol *(*
> in my employeeName column in my csvReader structure with below syntax
> replaceRegex(/employeeName, "[\[]", "(")
> Processor is throwing following exception.
> RecordPathException: java.util.regex.PatternSyntaxException: Unclosed
> character class near index 4 [\\[]
> It worked fine with other special characters such as \{, }, <, >, ;, _, "
> For double qoute ("), i had to use single escape character, for above listed
> other characters, worked fine without any escape character. Other folks in
> Nifi Slack tried \s, \d, \w, \.
> looks like non of them worked.
> replace function worked for replacing [ and ]characters. didn't test any
> other characters.
> Please address resolve the issue.
> Regards,
> Ravi
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3183: NIFI-5826 Fix to escaped backslash
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3183#discussion_r236830742
--- Diff:
nifi-commons/nifi-record-path/src/main/java/org/apache/nifi/record/path/util/RecordPathUtils.java
---
@@ -39,4 +39,44 @@ public static String getFirstStringValue(final
RecordPathSegment segment, final
return stringValue;
}
+
+public static String unescapeBackslash(String value) {
+if (value == null || value.isEmpty()) {
+return value;
+}
+// need to escape characters after backslashes
+final StringBuilder sb = new StringBuilder();
+boolean lastCharIsBackslash = false;
+for (int i = 0; i < value.length(); i++) {
+final char c = value.charAt(i);
+
--- End diff --
I am sorry, I meant if you could document the code. That way someone who
goes in can tell that you are escaping these specific characters per some
specification
---
[jira] [Commented] (NIFI-5826) UpdateRecord processor throwing PatternSyntaxException
[
https://issues.apache.org/jira/browse/NIFI-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700960#comment-16700960
]
ASF GitHub Bot commented on NIFI-5826:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3183#discussion_r236830742
--- Diff:
nifi-commons/nifi-record-path/src/main/java/org/apache/nifi/record/path/util/RecordPathUtils.java
---
@@ -39,4 +39,44 @@ public static String getFirstStringValue(final
RecordPathSegment segment, final
return stringValue;
}
+
+public static String unescapeBackslash(String value) {
+if (value == null || value.isEmpty()) {
+return value;
+}
+// need to escape characters after backslashes
+final StringBuilder sb = new StringBuilder();
+boolean lastCharIsBackslash = false;
+for (int i = 0; i < value.length(); i++) {
+final char c = value.charAt(i);
+
--- End diff --
I am sorry, I meant if you could document the code. That way someone who
goes in can tell that you are escaping these specific characters per some
specification
> UpdateRecord processor throwing PatternSyntaxException
> --
>
> Key: NIFI-5826
> URL: https://issues.apache.org/jira/browse/NIFI-5826
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
> Environment: Nifi in docker container
>Reporter: ravi kargam
>Assignee: Ed Berezitsky
>Priority: Minor
> Attachments: UpdateRecord_Config_Exception.JPG
>
>
> with replacement value strategy as Record Path Value,
> I am trying to replace square bracket symbol *[* with parenthesis symbol *(*
> in my employeeName column in my csvReader structure with below syntax
> replaceRegex(/employeeName, "[\[]", "(")
> Processor is throwing following exception.
> RecordPathException: java.util.regex.PatternSyntaxException: Unclosed
> character class near index 4 [\\[]
> It worked fine with other special characters such as \{, }, <, >, ;, _, "
> For double qoute ("), i had to use single escape character, for above listed
> other characters, worked fine without any escape character. Other folks in
> Nifi Slack tried \s, \d, \w, \.
> looks like non of them worked.
> replace function worked for replacing [ and ]characters. didn't test any
> other characters.
> Please address resolve the issue.
> Regards,
> Ravi
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-5844) Kryo Record Reader & Writer
Aaron Leon created NIFI-5844: Summary: Kryo Record Reader & Writer Key: NIFI-5844 URL: https://issues.apache.org/jira/browse/NIFI-5844 Project: Apache NiFi Issue Type: New Feature Reporter: Aaron Leon Kryo is a "fast and efficient binary object graph serialization framework for Java" ([https://github.com/EsotericSoftware/kryo)]. It is supported by Apache Spark and Apache Hive and used for internal serialization in Apache Storm. As a result, Kryo is used directly and indirectly by many large companies such as Spotify, Yahoo, Twitter and Groupon. IMO It would be a worthwhile investigation to support Kryo serialization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5844) Kryo Record Reader & Writer
[ https://issues.apache.org/jira/browse/NIFI-5844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Leon updated NIFI-5844: - Labels: features (was: ) > Kryo Record Reader & Writer > --- > > Key: NIFI-5844 > URL: https://issues.apache.org/jira/browse/NIFI-5844 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Aaron Leon >Priority: Minor > Labels: features > > Kryo is a "fast and efficient binary object graph serialization framework for > Java" ([https://github.com/EsotericSoftware/kryo)]. It is supported by Apache > Spark and Apache Hive and used for internal serialization in Apache Storm. As > a result, Kryo is used directly and indirectly by many large companies such > as Spotify, Yahoo, Twitter and Groupon. > IMO It would be a worthwhile investigation to support Kryo serialization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5172) PutElasticSearchRecord does to fail individual recods
[ https://issues.apache.org/jira/browse/NIFI-5172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700967#comment-16700967 ] Joseph Percivall commented on NIFI-5172: I ran into this issue today and I assumed this to be a "bug" given that it differs from the logic in for all the other record based processors and the PutElasticsearchHttp processor most of the core logic was copied from(including the comment stating the "correct" logic[1]). Addressing [~mike.thomsen]'s comment, at least on the HTTP endpoint, ES will tell you which one failed as that's the logic currently in PutElasticsearchHttp[2]. Talking with [~mattyb149], the logic for adding it would be something along the lines of a merging of the error handling of PutElasticsearchHttp and with the record creation of SplitRecord[3]. I would argue that this is higher than a minor priority as behaves opposite what we've trained users to expect and potentially leads to duplicate data being ingested. [1] https://issues.apache.org/jira/browse/NIFI-5172 [2] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java#L345 [3] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/SplitRecord.java#L144 > PutElasticSearchRecord does to fail individual recods > - > > Key: NIFI-5172 > URL: https://issues.apache.org/jira/browse/NIFI-5172 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.6.0 >Reporter: Juan C. Sequeiros >Priority: Minor > > My observation and not sure if working as expected but when I send my output > from MergeRecord ( set to 1 ) max number of records and one of those > records has an invalid timestamp "bogusdata" value ES rejects it, rightly so > since on ES we have a schema template more granular and is expecting > timestamp as type "date". > > From USER forums Matt Burgess: > Yes the current behavior is to move the entire input flowfile to > failure if any errors occur. Some other record-aware processors create > separate flow files for failed and successful records, but > PutElasticsearchHttpRecord does not (yet) do that. Please feel free to > write a Jira for this improvement. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-registry pull request #148: NIFIREG-211 Initial work for adding extenio...
Github user kevdoran commented on a diff in the pull request: https://github.com/apache/nifi-registry/pull/148#discussion_r236826377 --- Diff: nifi-registry-core/nifi-registry-framework/src/main/resources/db/migration/V3__AddExtensions.sql --- @@ -0,0 +1,62 @@ +-- Licensed to the Apache Software Foundation (ASF) under one or more +-- contributor license agreements. See the NOTICE file distributed with +-- this work for additional information regarding copyright ownership. +-- The ASF licenses this file to You under the Apache License, Version 2.0 +-- (the "License"); you may not use this file except in compliance with +-- the License. You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, software +-- distributed under the License is distributed on an "AS IS" BASIS, +-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +-- See the License for the specific language governing permissions and +-- limitations under the License. + +CREATE TABLE EXTENSION_BUNDLE ( +ID VARCHAR(50) NOT NULL, +BUCKET_ID VARCHAR(50) NOT NULL, +BUNDLE_TYPE VARCHAR(200) NOT NULL, +GROUP_ID VARCHAR(500) NOT NULL, +ARTIFACT_ID VARCHAR(500) NOT NULL, +CONSTRAINT PK__EXTENSION_BUNDLE_ID PRIMARY KEY (ID), +CONSTRAINT FK__EXTENSION_BUNDLE_BUCKET_ITEM_ID FOREIGN KEY (ID) REFERENCES BUCKET_ITEM(ID) ON DELETE CASCADE, +CONSTRAINT FK__EXTENSION_BUNDLE_BUCKET_ID FOREIGN KEY(BUCKET_ID) REFERENCES BUCKET(ID) ON DELETE CASCADE, +CONSTRAINT UNIQUE__EXTENSION_BUNDLE_BUCKET_GROUP_ARTIFACT UNIQUE (BUCKET_ID, GROUP_ID, ARTIFACT_ID) +); + +CREATE TABLE EXTENSION_BUNDLE_VERSION ( +ID VARCHAR(50) NOT NULL, +EXTENSION_BUNDLE_ID VARCHAR(50) NOT NULL, +VERSION VARCHAR(100) NOT NULL, +DEPENDENCY_GROUP_ID VARCHAR(500), --- End diff -- CPP extensions may have more than one dependency, in which case having these in a separate table may be worthwhile. ---
[GitHub] nifi-registry pull request #148: NIFIREG-211 Initial work for adding extenio...
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236837704
--- Diff:
nifi-registry-core/nifi-registry-framework/src/main/java/org/apache/nifi/registry/service/extension/StandardExtensionService.java
---
@@ -0,0 +1,589 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.service.extension;
+
+import org.apache.commons.codec.binary.Hex;
+import org.apache.commons.codec.digest.DigestUtils;
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.Validate;
+import org.apache.nifi.registry.bucket.Bucket;
+import org.apache.nifi.registry.db.entity.BucketEntity;
+import org.apache.nifi.registry.db.entity.ExtensionBundleEntity;
+import org.apache.nifi.registry.db.entity.ExtensionBundleEntityType;
+import org.apache.nifi.registry.db.entity.ExtensionBundleVersionEntity;
+import org.apache.nifi.registry.exception.ResourceNotFoundException;
+import org.apache.nifi.registry.extension.BundleCoordinate;
+import org.apache.nifi.registry.extension.BundleDetails;
+import org.apache.nifi.registry.extension.BundleExtractor;
+import org.apache.nifi.registry.extension.ExtensionBundle;
+import org.apache.nifi.registry.extension.ExtensionBundleContext;
+import
org.apache.nifi.registry.extension.ExtensionBundlePersistenceProvider;
+import org.apache.nifi.registry.extension.ExtensionBundleType;
+import org.apache.nifi.registry.extension.ExtensionBundleVersion;
+import org.apache.nifi.registry.extension.ExtensionBundleVersionDependency;
+import org.apache.nifi.registry.extension.ExtensionBundleVersionMetadata;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoArtifact;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoBucket;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoGroup;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoVersionSummary;
+import org.apache.nifi.registry.properties.NiFiRegistryProperties;
+import
org.apache.nifi.registry.provider.extension.StandardExtensionBundleContext;
+import org.apache.nifi.registry.security.authorization.user.NiFiUserUtils;
+import org.apache.nifi.registry.service.DataModelMapper;
+import org.apache.nifi.registry.service.MetadataService;
+import org.apache.nifi.registry.util.FileUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Service;
+
+import javax.validation.ConstraintViolation;
+import javax.validation.ConstraintViolationException;
+import javax.validation.Validator;
+import java.io.BufferedInputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.security.DigestInputStream;
+import java.security.MessageDigest;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+@Service
+public class StandardExtensionService implements ExtensionService {
+
+private static final Logger LOGGER =
LoggerFactory.getLogger(StandardExtensionService.class);
+
+private final MetadataService metadataService;
+private final Map extractors;
+private final ExtensionBundlePersistenceProvider
bundlePersistenceProvider;
+private final Validator validator;
+private final File extensionsWorkingDir;
+
+@Autowired
+public StandardExtensionService(final MetadataService metadataService,
+final Map extractors,
+final
ExtensionBundlePersistenceProvider bun
[GitHub] nifi-registry pull request #148: NIFIREG-211 Initial work for adding extenio...
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236827312
--- Diff:
nifi-registry-core/nifi-registry-web-api/src/main/java/org/apache/nifi/registry/web/api/ExtensionRepositoryResource.java
---
@@ -0,0 +1,378 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.web.api;
+
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiParam;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.Extension;
+import io.swagger.annotations.ExtensionProperty;
+import org.apache.nifi.registry.bucket.Bucket;
+import org.apache.nifi.registry.bucket.BucketItem;
+import org.apache.nifi.registry.event.EventService;
+import org.apache.nifi.registry.extension.ExtensionBundleVersion;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoArtifact;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoBucket;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoGroup;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoVersion;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoVersionSummary;
+import org.apache.nifi.registry.security.authorization.RequestAction;
+import org.apache.nifi.registry.service.AuthorizationService;
+import org.apache.nifi.registry.service.RegistryService;
+import
org.apache.nifi.registry.service.extension.ExtensionBundleVersionCoordinate;
+import org.apache.nifi.registry.web.link.LinkService;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Component;
+
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.Produces;
+import javax.ws.rs.core.Link;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import javax.ws.rs.core.StreamingOutput;
+import java.util.ArrayList;
+import java.util.Set;
+import java.util.SortedSet;
+
+@Component
+@Path("/extensions/repo")
+@Api(
+value = "extension_repository",
+description = "Interact with extension bundles via the hierarchy
of bucket/group/artifact/version.",
+authorizations = { @Authorization("Authorization") }
+)
+public class ExtensionRepositoryResource extends
AuthorizableApplicationResource {
+
+public static final String CONTENT_DISPOSITION_HEADER =
"content-disposition";
+private final RegistryService registryService;
+private final LinkService linkService;
+
+@Autowired
+public ExtensionRepositoryResource(
+final RegistryService registryService,
+final LinkService linkService,
+final AuthorizationService authorizationService,
+final EventService eventService) {
+super(authorizationService, eventService);
+this.registryService = registryService;
+this.linkService = linkService;
+}
+
+@GET
+@Consumes(MediaType.WILDCARD)
+@Produces(MediaType.APPLICATION_JSON)
+@ApiOperation(
+value = "Gets the names of the buckets the current user is
authorized for in order to browse the repo by bucket",
+response = ExtensionRepoBucket.class,
+responseContainer = "List",
+extensions = {
+@Extension(name = "access-policy", properties = {
+@ExtensionProperty(name = "action", value =
"read"),
+@ExtensionProperty(name = "resource", value =
"/buckets/{bucketId}") })
+}
--- End diff --
I don't think we want an access-policy documented here as read only access
is allowed for all authenticated users.
-
[GitHub] nifi-registry pull request #148: NIFIREG-211 Initial work for adding extenio...
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236810679
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/repo/ExtensionRepoVersion.java
---
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension.repo;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkAdapter;
+
+import javax.ws.rs.core.Link;
+import javax.xml.bind.annotation.XmlElement;
+import javax.xml.bind.annotation.XmlRootElement;
+import javax.xml.bind.annotation.adapters.XmlJavaTypeAdapter;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionRepoVersion {
--- End diff --
these dynamically generated links worked well in my testing
---
[GitHub] nifi-registry pull request #148: NIFIREG-211 Initial work for adding extenio...
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236808042
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
--- End diff --
This is a minor nitpick, but for the JSON field name here, I would have a
slight preference for just using `sha256` as the field name as that is
consistent with the
`/extensions/repo/{bucket}/{group}/{artifact}/{version}/sha256` endpoint (and
the field description in the REST API documentation clarifies that it is the
hex digest).
---
[jira] [Commented] (NIFIREG-211) Add extension bundles as a type of versioned item
[
https://issues.apache.org/jira/browse/NIFIREG-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700996#comment-16700996
]
ASF GitHub Bot commented on NIFIREG-211:
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236827312
--- Diff:
nifi-registry-core/nifi-registry-web-api/src/main/java/org/apache/nifi/registry/web/api/ExtensionRepositoryResource.java
---
@@ -0,0 +1,378 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.web.api;
+
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiParam;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.Extension;
+import io.swagger.annotations.ExtensionProperty;
+import org.apache.nifi.registry.bucket.Bucket;
+import org.apache.nifi.registry.bucket.BucketItem;
+import org.apache.nifi.registry.event.EventService;
+import org.apache.nifi.registry.extension.ExtensionBundleVersion;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoArtifact;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoBucket;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoGroup;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoVersion;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoVersionSummary;
+import org.apache.nifi.registry.security.authorization.RequestAction;
+import org.apache.nifi.registry.service.AuthorizationService;
+import org.apache.nifi.registry.service.RegistryService;
+import
org.apache.nifi.registry.service.extension.ExtensionBundleVersionCoordinate;
+import org.apache.nifi.registry.web.link.LinkService;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Component;
+
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.Produces;
+import javax.ws.rs.core.Link;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import javax.ws.rs.core.StreamingOutput;
+import java.util.ArrayList;
+import java.util.Set;
+import java.util.SortedSet;
+
+@Component
+@Path("/extensions/repo")
+@Api(
+value = "extension_repository",
+description = "Interact with extension bundles via the hierarchy
of bucket/group/artifact/version.",
+authorizations = { @Authorization("Authorization") }
+)
+public class ExtensionRepositoryResource extends
AuthorizableApplicationResource {
+
+public static final String CONTENT_DISPOSITION_HEADER =
"content-disposition";
+private final RegistryService registryService;
+private final LinkService linkService;
+
+@Autowired
+public ExtensionRepositoryResource(
+final RegistryService registryService,
+final LinkService linkService,
+final AuthorizationService authorizationService,
+final EventService eventService) {
+super(authorizationService, eventService);
+this.registryService = registryService;
+this.linkService = linkService;
+}
+
+@GET
+@Consumes(MediaType.WILDCARD)
+@Produces(MediaType.APPLICATION_JSON)
+@ApiOperation(
+value = "Gets the names of the buckets the current user is
authorized for in order to browse the repo by bucket",
+response = ExtensionRepoBucket.class,
+responseContainer = "List",
+extensions = {
+@Extension(name = "access-policy", properties = {
+@ExtensionProperty(name = "action", value =
"read"),
+
[jira] [Commented] (NIFIREG-211) Add extension bundles as a type of versioned item
[ https://issues.apache.org/jira/browse/NIFIREG-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700995#comment-16700995 ] ASF GitHub Bot commented on NIFIREG-211: Github user kevdoran commented on a diff in the pull request: https://github.com/apache/nifi-registry/pull/148#discussion_r236826377 --- Diff: nifi-registry-core/nifi-registry-framework/src/main/resources/db/migration/V3__AddExtensions.sql --- @@ -0,0 +1,62 @@ +-- Licensed to the Apache Software Foundation (ASF) under one or more +-- contributor license agreements. See the NOTICE file distributed with +-- this work for additional information regarding copyright ownership. +-- The ASF licenses this file to You under the Apache License, Version 2.0 +-- (the "License"); you may not use this file except in compliance with +-- the License. You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, software +-- distributed under the License is distributed on an "AS IS" BASIS, +-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +-- See the License for the specific language governing permissions and +-- limitations under the License. + +CREATE TABLE EXTENSION_BUNDLE ( +ID VARCHAR(50) NOT NULL, +BUCKET_ID VARCHAR(50) NOT NULL, +BUNDLE_TYPE VARCHAR(200) NOT NULL, +GROUP_ID VARCHAR(500) NOT NULL, +ARTIFACT_ID VARCHAR(500) NOT NULL, +CONSTRAINT PK__EXTENSION_BUNDLE_ID PRIMARY KEY (ID), +CONSTRAINT FK__EXTENSION_BUNDLE_BUCKET_ITEM_ID FOREIGN KEY (ID) REFERENCES BUCKET_ITEM(ID) ON DELETE CASCADE, +CONSTRAINT FK__EXTENSION_BUNDLE_BUCKET_ID FOREIGN KEY(BUCKET_ID) REFERENCES BUCKET(ID) ON DELETE CASCADE, +CONSTRAINT UNIQUE__EXTENSION_BUNDLE_BUCKET_GROUP_ARTIFACT UNIQUE (BUCKET_ID, GROUP_ID, ARTIFACT_ID) +); + +CREATE TABLE EXTENSION_BUNDLE_VERSION ( +ID VARCHAR(50) NOT NULL, +EXTENSION_BUNDLE_ID VARCHAR(50) NOT NULL, +VERSION VARCHAR(100) NOT NULL, +DEPENDENCY_GROUP_ID VARCHAR(500), --- End diff -- CPP extensions may have more than one dependency, in which case having these in a separate table may be worthwhile. > Add extension bundles as a type of versioned item > - > > Key: NIFIREG-211 > URL: https://issues.apache.org/jira/browse/NIFIREG-211 > Project: NiFi Registry > Issue Type: Improvement >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Major > Fix For: 0.4.0 > > > This ticket is to capture the work for adding extension bundles to NiFi > Registry. > This work may require several follow on tickets, but at a high-level will > include some of the following: > - Add a new type of item called an extension bundle, where each bundle > can contain one ore extensions or APIs > > - Support bundles for traditional NiFi (aka NARs) and also bundles for > MiNiFi CPP > > - Ability to upload the binary artifact for a bundle and extract the > metadata about the bundle, and metadata about the extensions contained > in the bundle (more on this later) > > - Provide a pluggable storage provider for saving the content of each > extension bundle so that we can have different implementations like > local fileysystem, S3, and other object stores > > - Provide a REST API for listing and retrieving available bundles, > integrate this into the registry Java client and NiFi CLI > - Security considerations such as checksums and cryptographic signatures for > bundles -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFIREG-211) Add extension bundles as a type of versioned item
[
https://issues.apache.org/jira/browse/NIFIREG-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700994#comment-16700994
]
ASF GitHub Bot commented on NIFIREG-211:
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236837704
--- Diff:
nifi-registry-core/nifi-registry-framework/src/main/java/org/apache/nifi/registry/service/extension/StandardExtensionService.java
---
@@ -0,0 +1,589 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.service.extension;
+
+import org.apache.commons.codec.binary.Hex;
+import org.apache.commons.codec.digest.DigestUtils;
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.Validate;
+import org.apache.nifi.registry.bucket.Bucket;
+import org.apache.nifi.registry.db.entity.BucketEntity;
+import org.apache.nifi.registry.db.entity.ExtensionBundleEntity;
+import org.apache.nifi.registry.db.entity.ExtensionBundleEntityType;
+import org.apache.nifi.registry.db.entity.ExtensionBundleVersionEntity;
+import org.apache.nifi.registry.exception.ResourceNotFoundException;
+import org.apache.nifi.registry.extension.BundleCoordinate;
+import org.apache.nifi.registry.extension.BundleDetails;
+import org.apache.nifi.registry.extension.BundleExtractor;
+import org.apache.nifi.registry.extension.ExtensionBundle;
+import org.apache.nifi.registry.extension.ExtensionBundleContext;
+import
org.apache.nifi.registry.extension.ExtensionBundlePersistenceProvider;
+import org.apache.nifi.registry.extension.ExtensionBundleType;
+import org.apache.nifi.registry.extension.ExtensionBundleVersion;
+import org.apache.nifi.registry.extension.ExtensionBundleVersionDependency;
+import org.apache.nifi.registry.extension.ExtensionBundleVersionMetadata;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoArtifact;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoBucket;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoGroup;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoVersionSummary;
+import org.apache.nifi.registry.properties.NiFiRegistryProperties;
+import
org.apache.nifi.registry.provider.extension.StandardExtensionBundleContext;
+import org.apache.nifi.registry.security.authorization.user.NiFiUserUtils;
+import org.apache.nifi.registry.service.DataModelMapper;
+import org.apache.nifi.registry.service.MetadataService;
+import org.apache.nifi.registry.util.FileUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Service;
+
+import javax.validation.ConstraintViolation;
+import javax.validation.ConstraintViolationException;
+import javax.validation.Validator;
+import java.io.BufferedInputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.security.DigestInputStream;
+import java.security.MessageDigest;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+@Service
+public class StandardExtensionService implements ExtensionService {
+
+private static final Logger LOGGER =
LoggerFactory.getLogger(StandardExtensionService.class);
+
+private final MetadataService metadataService;
+private final Map extractors;
+private final ExtensionBundlePersistenceProvider
bundlePersistenceProvider;
+private final Validator validator;
+private final File extensionsWorking
[jira] [Commented] (NIFIREG-211) Add extension bundles as a type of versioned item
[
https://issues.apache.org/jira/browse/NIFIREG-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700997#comment-16700997
]
ASF GitHub Bot commented on NIFIREG-211:
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236808042
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
--- End diff --
This is a minor nitpick, but for the JSON field name here, I would have a
slight preference for just using `sha256` as the field name as that is
consistent with the
`/extensions/repo/{bucket}/{group}/{artifact}/{version}/sha256` endpoint (and
the field description in the REST API documentation clarifies that it is the
hex digest).
> Add extension bundles as a type of versioned item
> -
>
> Key: NIFIREG-211
> URL: https://issues.apache.org/jira/browse/NIFIREG-211
> Project: NiFi Registry
> Issue Type: Improvement
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Major
> Fix For: 0.4.0
>
>
> This ticket is to capture the work for adding extension bundles to NiFi
> Registry.
> This work may require several follow on tickets, but at a high-level will
> include some of the following:
> - Add a new type of item called an extension bundle, where each bundle
> can contain one ore extensions or APIs
>
> - Support bundles for traditional NiFi (aka NARs) and also bundles for
> MiNiFi CPP
>
> - Ability to upload the binary artifact for a bundle and extract the
> metadata about the bundle, and metadata about the extensions contained
> in the bundle (more on this later)
>
> - Provide a pluggable storage provider for saving the content of each
> extension bundle so that we can have different implementations like
> local fileysystem, S3, and other object stores
>
> - Provide a REST API for listing and retrieving available bundles,
> integrate this into the registry Java client and NiFi CLI
> - Security considerations such as checksums and cryptographic signatures for
> bundles
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFIREG-211) Add extension bundles as a type of versioned item
[
https://issues.apache.org/jira/browse/NIFIREG-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700993#comment-16700993
]
ASF GitHub Bot commented on NIFIREG-211:
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236810679
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/repo/ExtensionRepoVersion.java
---
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension.repo;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkAdapter;
+
+import javax.ws.rs.core.Link;
+import javax.xml.bind.annotation.XmlElement;
+import javax.xml.bind.annotation.XmlRootElement;
+import javax.xml.bind.annotation.adapters.XmlJavaTypeAdapter;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionRepoVersion {
--- End diff --
these dynamically generated links worked well in my testing
> Add extension bundles as a type of versioned item
> -
>
> Key: NIFIREG-211
> URL: https://issues.apache.org/jira/browse/NIFIREG-211
> Project: NiFi Registry
> Issue Type: Improvement
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Major
> Fix For: 0.4.0
>
>
> This ticket is to capture the work for adding extension bundles to NiFi
> Registry.
> This work may require several follow on tickets, but at a high-level will
> include some of the following:
> - Add a new type of item called an extension bundle, where each bundle
> can contain one ore extensions or APIs
>
> - Support bundles for traditional NiFi (aka NARs) and also bundles for
> MiNiFi CPP
>
> - Ability to upload the binary artifact for a bundle and extract the
> metadata about the bundle, and metadata about the extensions contained
> in the bundle (more on this later)
>
> - Provide a pluggable storage provider for saving the content of each
> extension bundle so that we can have different implementations like
> local fileysystem, S3, and other object stores
>
> - Provide a REST API for listing and retrieving available bundles,
> integrate this into the registry Java client and NiFi CLI
> - Security considerations such as checksums and cryptographic signatures for
> bundles
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFIREG-211) Add extension bundles as a type of versioned item
[
https://issues.apache.org/jira/browse/NIFIREG-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701003#comment-16701003
]
ASF GitHub Bot commented on NIFIREG-211:
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236839456
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
+
+
+@ApiModelProperty(value = "The id of this version of the extension
bundle")
+public String getId() {
+return id;
+}
+
+public void setId(String id) {
+this.id = id;
+}
+
+@ApiModelProperty(value = "The id of the extension bundle this version
is for")
+public String getExtensionBundleId() {
+return extensionBundleId;
+}
+
+public void setExtensionBundleId(String extensionBundleId) {
+this.extensionBundleId = extensionBundleId;
+}
+
+@ApiModelProperty(value = "The id of the bucket the extension bundle
belongs to", required = true)
+public String getBucketId() {
+return bucketId;
+}
+
+public void setBucketId(String bucketId) {
+this.bucketId = bucketId;
+}
+
+@ApiModelProperty(value = "The version of the extension bundle")
+public String getVersion() {
+return version;
+}
+
+public void setVersion(String version) {
+this.version = version;
+}
+
+@ApiModelProperty(value = "The optional bundle dependency (i.e.
another bundle this bundle is dependent on)")
+public ExtensionBundleVersionDependency getDependency() {
+return dependency;
+}
+
+public void setDependency(ExtensionBundleVersionDependency dependency)
{
+this.dependency = dependency;
+}
+
+@ApiModelProperty(value = "The timestamp of the create date of this
version")
+public long getTimestamp() {
+return timestamp;
+}
+
+public void setTimestamp(long timestamp) {
+this.timestamp = timestamp;
+}
+
+@ApiModelProperty(value = "The identity that created this version")
+public String getAuthor() {
--- End diff --
Understanding that "authors" can take many forms, is this a standardized
String representation, or could it be "alopresto", "Andy LoPresto", and "Andy
LoPresto " are all valid and used simultaneously, causing
lookup/resolution issues later? Would using a custom datatype here make sense?
> Add extension bundles as a type of versioned item
> -
>
> Key: NIFIREG-211
> URL: https://issues.apache.org/jira/browse/NIFIREG-211
>
[GitHub] nifi-registry pull request #148: NIFIREG-211 Initial work for adding extenio...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236839456
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
+
+
+@ApiModelProperty(value = "The id of this version of the extension
bundle")
+public String getId() {
+return id;
+}
+
+public void setId(String id) {
+this.id = id;
+}
+
+@ApiModelProperty(value = "The id of the extension bundle this version
is for")
+public String getExtensionBundleId() {
+return extensionBundleId;
+}
+
+public void setExtensionBundleId(String extensionBundleId) {
+this.extensionBundleId = extensionBundleId;
+}
+
+@ApiModelProperty(value = "The id of the bucket the extension bundle
belongs to", required = true)
+public String getBucketId() {
+return bucketId;
+}
+
+public void setBucketId(String bucketId) {
+this.bucketId = bucketId;
+}
+
+@ApiModelProperty(value = "The version of the extension bundle")
+public String getVersion() {
+return version;
+}
+
+public void setVersion(String version) {
+this.version = version;
+}
+
+@ApiModelProperty(value = "The optional bundle dependency (i.e.
another bundle this bundle is dependent on)")
+public ExtensionBundleVersionDependency getDependency() {
+return dependency;
+}
+
+public void setDependency(ExtensionBundleVersionDependency dependency)
{
+this.dependency = dependency;
+}
+
+@ApiModelProperty(value = "The timestamp of the create date of this
version")
+public long getTimestamp() {
+return timestamp;
+}
+
+public void setTimestamp(long timestamp) {
+this.timestamp = timestamp;
+}
+
+@ApiModelProperty(value = "The identity that created this version")
+public String getAuthor() {
--- End diff --
Understanding that "authors" can take many forms, is this a standardized
String representation, or could it be "alopresto", "Andy LoPresto", and "Andy
LoPresto " are all valid and used simultaneously, causing
lookup/resolution issues later? Would using a custom datatype here make sense?
---
[jira] [Commented] (NIFIREG-211) Add extension bundles as a type of versioned item
[
https://issues.apache.org/jira/browse/NIFIREG-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701005#comment-16701005
]
ASF GitHub Bot commented on NIFIREG-211:
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236840020
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
+
+
+@ApiModelProperty(value = "The id of this version of the extension
bundle")
+public String getId() {
+return id;
+}
+
+public void setId(String id) {
+this.id = id;
+}
+
+@ApiModelProperty(value = "The id of the extension bundle this version
is for")
+public String getExtensionBundleId() {
+return extensionBundleId;
+}
+
+public void setExtensionBundleId(String extensionBundleId) {
+this.extensionBundleId = extensionBundleId;
+}
+
+@ApiModelProperty(value = "The id of the bucket the extension bundle
belongs to", required = true)
+public String getBucketId() {
+return bucketId;
+}
+
+public void setBucketId(String bucketId) {
+this.bucketId = bucketId;
+}
+
+@ApiModelProperty(value = "The version of the extension bundle")
+public String getVersion() {
+return version;
+}
+
+public void setVersion(String version) {
+this.version = version;
+}
+
+@ApiModelProperty(value = "The optional bundle dependency (i.e.
another bundle this bundle is dependent on)")
+public ExtensionBundleVersionDependency getDependency() {
+return dependency;
+}
+
+public void setDependency(ExtensionBundleVersionDependency dependency)
{
+this.dependency = dependency;
+}
+
+@ApiModelProperty(value = "The timestamp of the create date of this
version")
+public long getTimestamp() {
+return timestamp;
+}
+
+public void setTimestamp(long timestamp) {
+this.timestamp = timestamp;
+}
+
+@ApiModelProperty(value = "The identity that created this version")
+public String getAuthor() {
+return author;
+}
+
+public void setAuthor(String author) {
+this.author = author;
+}
+
+@ApiModelProperty(value = "The description for this version")
+public String getDescription() {
+return description;
+}
+
+public void setDescription(String description) {
+this.description = description;
+}
+
+@ApiModelProperty(value = "The hex representation of the SHA-256
digest of the binary content for this version
[GitHub] nifi-registry pull request #148: NIFIREG-211 Initial work for adding extenio...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236840020
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
+
+
+@ApiModelProperty(value = "The id of this version of the extension
bundle")
+public String getId() {
+return id;
+}
+
+public void setId(String id) {
+this.id = id;
+}
+
+@ApiModelProperty(value = "The id of the extension bundle this version
is for")
+public String getExtensionBundleId() {
+return extensionBundleId;
+}
+
+public void setExtensionBundleId(String extensionBundleId) {
+this.extensionBundleId = extensionBundleId;
+}
+
+@ApiModelProperty(value = "The id of the bucket the extension bundle
belongs to", required = true)
+public String getBucketId() {
+return bucketId;
+}
+
+public void setBucketId(String bucketId) {
+this.bucketId = bucketId;
+}
+
+@ApiModelProperty(value = "The version of the extension bundle")
+public String getVersion() {
+return version;
+}
+
+public void setVersion(String version) {
+this.version = version;
+}
+
+@ApiModelProperty(value = "The optional bundle dependency (i.e.
another bundle this bundle is dependent on)")
+public ExtensionBundleVersionDependency getDependency() {
+return dependency;
+}
+
+public void setDependency(ExtensionBundleVersionDependency dependency)
{
+this.dependency = dependency;
+}
+
+@ApiModelProperty(value = "The timestamp of the create date of this
version")
+public long getTimestamp() {
+return timestamp;
+}
+
+public void setTimestamp(long timestamp) {
+this.timestamp = timestamp;
+}
+
+@ApiModelProperty(value = "The identity that created this version")
+public String getAuthor() {
+return author;
+}
+
+public void setAuthor(String author) {
+this.author = author;
+}
+
+@ApiModelProperty(value = "The description for this version")
+public String getDescription() {
+return description;
+}
+
+public void setDescription(String description) {
+this.description = description;
+}
+
+@ApiModelProperty(value = "The hex representation of the SHA-256
digest of the binary content for this version")
+public String getSha256Hex() {
+return sha256Hex;
+}
+
+public void setSha256Hex(String sha256Hex) {
+this.sha256Hex = sha256Hex;
+}
+
+@Override
+public int compareTo(final Ex
[GitHub] nifi-registry pull request #148: NIFIREG-211 Initial work for adding extenio...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236840513
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
--- End diff --
How easy is it to add a new field here? I'd like to propose storage of a
cryptographic signature of some sort, in addition to the SHA-256 checksum,
which can be modified (generated over new contents) by a malicious user. What
inputs (data, fields, encoding, etc.) does the SHA-256 cover?
---
[jira] [Commented] (NIFIREG-211) Add extension bundles as a type of versioned item
[
https://issues.apache.org/jira/browse/NIFIREG-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701008#comment-16701008
]
ASF GitHub Bot commented on NIFIREG-211:
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236840513
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
--- End diff --
How easy is it to add a new field here? I'd like to propose storage of a
cryptographic signature of some sort, in addition to the SHA-256 checksum,
which can be modified (generated over new contents) by a malicious user. What
inputs (data, fields, encoding, etc.) does the SHA-256 cover?
> Add extension bundles as a type of versioned item
> -
>
> Key: NIFIREG-211
> URL: https://issues.apache.org/jira/browse/NIFIREG-211
> Project: NiFi Registry
> Issue Type: Improvement
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Major
> Fix For: 0.4.0
>
>
> This ticket is to capture the work for adding extension bundles to NiFi
> Registry.
> This work may require several follow on tickets, but at a high-level will
> include some of the following:
> - Add a new type of item called an extension bundle, where each bundle
> can contain one ore extensions or APIs
>
> - Support bundles for traditional NiFi (aka NARs) and also bundles for
> MiNiFi CPP
>
> - Ability to upload the binary artifact for a bundle and extract the
> metadata about the bundle, and metadata about the extensions contained
> in the bundle (more on this later)
>
> - Provide a pluggable storage provider for saving the content of each
> extension bundle so that we can have different implementations like
> local fileysystem, S3, and other object stores
>
> - Provide a REST API for listing and retrieving available bundles,
> integrate this into the registry Java client and NiFi CLI
> - Security considerations such as checksums and cryptographic signatures for
> bundles
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3183: NIFI-5826 Fix to escaped backslash
Github user bdesert commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3183#discussion_r236843474
--- Diff:
nifi-commons/nifi-record-path/src/main/java/org/apache/nifi/record/path/util/RecordPathUtils.java
---
@@ -39,4 +39,44 @@ public static String getFirstStringValue(final
RecordPathSegment segment, final
return stringValue;
}
+
+public static String unescapeBackslash(String value) {
+if (value == null || value.isEmpty()) {
+return value;
+}
+// need to escape characters after backslashes
+final StringBuilder sb = new StringBuilder();
+boolean lastCharIsBackslash = false;
+for (int i = 0; i < value.length(); i++) {
+final char c = value.charAt(i);
+
--- End diff --
@ottobackwards I've added Javadocs with reference to antlr parser and
explaining the reasons for this transformation.
---
[jira] [Commented] (NIFI-5826) UpdateRecord processor throwing PatternSyntaxException
[
https://issues.apache.org/jira/browse/NIFI-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701024#comment-16701024
]
ASF GitHub Bot commented on NIFI-5826:
--
Github user bdesert commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3183#discussion_r236843474
--- Diff:
nifi-commons/nifi-record-path/src/main/java/org/apache/nifi/record/path/util/RecordPathUtils.java
---
@@ -39,4 +39,44 @@ public static String getFirstStringValue(final
RecordPathSegment segment, final
return stringValue;
}
+
+public static String unescapeBackslash(String value) {
+if (value == null || value.isEmpty()) {
+return value;
+}
+// need to escape characters after backslashes
+final StringBuilder sb = new StringBuilder();
+boolean lastCharIsBackslash = false;
+for (int i = 0; i < value.length(); i++) {
+final char c = value.charAt(i);
+
--- End diff --
@ottobackwards I've added Javadocs with reference to antlr parser and
explaining the reasons for this transformation.
> UpdateRecord processor throwing PatternSyntaxException
> --
>
> Key: NIFI-5826
> URL: https://issues.apache.org/jira/browse/NIFI-5826
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
> Environment: Nifi in docker container
>Reporter: ravi kargam
>Assignee: Ed Berezitsky
>Priority: Minor
> Attachments: UpdateRecord_Config_Exception.JPG
>
>
> with replacement value strategy as Record Path Value,
> I am trying to replace square bracket symbol *[* with parenthesis symbol *(*
> in my employeeName column in my csvReader structure with below syntax
> replaceRegex(/employeeName, "[\[]", "(")
> Processor is throwing following exception.
> RecordPathException: java.util.regex.PatternSyntaxException: Unclosed
> character class near index 4 [\\[]
> It worked fine with other special characters such as \{, }, <, >, ;, _, "
> For double qoute ("), i had to use single escape character, for above listed
> other characters, worked fine without any escape character. Other folks in
> Nifi Slack tried \s, \d, \w, \.
> looks like non of them worked.
> replace function worked for replacing [ and ]characters. didn't test any
> other characters.
> Please address resolve the issue.
> Regards,
> Ravi
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi-registry pull request #148: NIFIREG-211 Initial work for adding extenio...
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236849392
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
--- End diff --
@kevdoran I'm good with calling it sha256 for consistency
@alopresto It is fairly straight forward to add another field here, just
typical changes like DB schema update, new member field, modify mapping code
(no ORM here :( ). Right now the input of the SHA-256 is the binary content of
the bundle (i.e. the input stream of actual NAR file).
Did you want me to add in the cryptographic signature in this PR (assuming
you point at how to calculate it), or do you want to wait til this is merged
and add as follow on work?
It can be done either way, there is still a bit more work to do before this
is ready anyway.
---
[jira] [Commented] (NIFIREG-211) Add extension bundles as a type of versioned item
[
https://issues.apache.org/jira/browse/NIFIREG-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701048#comment-16701048
]
ASF GitHub Bot commented on NIFIREG-211:
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236849392
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
--- End diff --
@kevdoran I'm good with calling it sha256 for consistency
@alopresto It is fairly straight forward to add another field here, just
typical changes like DB schema update, new member field, modify mapping code
(no ORM here :( ). Right now the input of the SHA-256 is the binary content of
the bundle (i.e. the input stream of actual NAR file).
Did you want me to add in the cryptographic signature in this PR (assuming
you point at how to calculate it), or do you want to wait til this is merged
and add as follow on work?
It can be done either way, there is still a bit more work to do before this
is ready anyway.
> Add extension bundles as a type of versioned item
> -
>
> Key: NIFIREG-211
> URL: https://issues.apache.org/jira/browse/NIFIREG-211
> Project: NiFi Registry
> Issue Type: Improvement
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Major
> Fix For: 0.4.0
>
>
> This ticket is to capture the work for adding extension bundles to NiFi
> Registry.
> This work may require several follow on tickets, but at a high-level will
> include some of the following:
> - Add a new type of item called an extension bundle, where each bundle
> can contain one ore extensions or APIs
>
> - Support bundles for traditional NiFi (aka NARs) and also bundles for
> MiNiFi CPP
>
> - Ability to upload the binary artifact for a bundle and extract the
> metadata about the bundle, and metadata about the extensions contained
> in the bundle (more on this later)
>
> - Provide a pluggable storage provider for saving the content of each
> extension bundle so that we can have different implementations like
> local fileysystem, S3, and other object stores
>
> - Provide a REST API for listing and retrieving available bundles,
> integrate this into the registry Java client and NiFi CLI
> - Security considerations such as checksums and cryptographic signatures for
> bundles
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi-registry pull request #148: NIFIREG-211 Initial work for adding extenio...
Github user bbende commented on a diff in the pull request: https://github.com/apache/nifi-registry/pull/148#discussion_r236849686 --- Diff: nifi-registry-core/nifi-registry-framework/src/main/resources/db/migration/V3__AddExtensions.sql --- @@ -0,0 +1,62 @@ +-- Licensed to the Apache Software Foundation (ASF) under one or more +-- contributor license agreements. See the NOTICE file distributed with +-- this work for additional information regarding copyright ownership. +-- The ASF licenses this file to You under the Apache License, Version 2.0 +-- (the "License"); you may not use this file except in compliance with +-- the License. You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, software +-- distributed under the License is distributed on an "AS IS" BASIS, +-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +-- See the License for the specific language governing permissions and +-- limitations under the License. + +CREATE TABLE EXTENSION_BUNDLE ( +ID VARCHAR(50) NOT NULL, +BUCKET_ID VARCHAR(50) NOT NULL, +BUNDLE_TYPE VARCHAR(200) NOT NULL, +GROUP_ID VARCHAR(500) NOT NULL, +ARTIFACT_ID VARCHAR(500) NOT NULL, +CONSTRAINT PK__EXTENSION_BUNDLE_ID PRIMARY KEY (ID), +CONSTRAINT FK__EXTENSION_BUNDLE_BUCKET_ITEM_ID FOREIGN KEY (ID) REFERENCES BUCKET_ITEM(ID) ON DELETE CASCADE, +CONSTRAINT FK__EXTENSION_BUNDLE_BUCKET_ID FOREIGN KEY(BUCKET_ID) REFERENCES BUCKET(ID) ON DELETE CASCADE, +CONSTRAINT UNIQUE__EXTENSION_BUNDLE_BUCKET_GROUP_ARTIFACT UNIQUE (BUCKET_ID, GROUP_ID, ARTIFACT_ID) +); + +CREATE TABLE EXTENSION_BUNDLE_VERSION ( +ID VARCHAR(50) NOT NULL, +EXTENSION_BUNDLE_ID VARCHAR(50) NOT NULL, +VERSION VARCHAR(100) NOT NULL, +DEPENDENCY_GROUP_ID VARCHAR(500), --- End diff -- That is a good point, it probably makes sense to separate out a separate table where bundle version can have one-to-many dependencies, and NARs will just so happen to only ever have one. Will work on this update. ---
[GitHub] nifi-registry pull request #148: NIFIREG-211 Initial work for adding extenio...
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236849819
--- Diff:
nifi-registry-core/nifi-registry-framework/src/main/java/org/apache/nifi/registry/service/extension/StandardExtensionService.java
---
@@ -0,0 +1,589 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.service.extension;
+
+import org.apache.commons.codec.binary.Hex;
+import org.apache.commons.codec.digest.DigestUtils;
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.Validate;
+import org.apache.nifi.registry.bucket.Bucket;
+import org.apache.nifi.registry.db.entity.BucketEntity;
+import org.apache.nifi.registry.db.entity.ExtensionBundleEntity;
+import org.apache.nifi.registry.db.entity.ExtensionBundleEntityType;
+import org.apache.nifi.registry.db.entity.ExtensionBundleVersionEntity;
+import org.apache.nifi.registry.exception.ResourceNotFoundException;
+import org.apache.nifi.registry.extension.BundleCoordinate;
+import org.apache.nifi.registry.extension.BundleDetails;
+import org.apache.nifi.registry.extension.BundleExtractor;
+import org.apache.nifi.registry.extension.ExtensionBundle;
+import org.apache.nifi.registry.extension.ExtensionBundleContext;
+import
org.apache.nifi.registry.extension.ExtensionBundlePersistenceProvider;
+import org.apache.nifi.registry.extension.ExtensionBundleType;
+import org.apache.nifi.registry.extension.ExtensionBundleVersion;
+import org.apache.nifi.registry.extension.ExtensionBundleVersionDependency;
+import org.apache.nifi.registry.extension.ExtensionBundleVersionMetadata;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoArtifact;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoBucket;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoGroup;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoVersionSummary;
+import org.apache.nifi.registry.properties.NiFiRegistryProperties;
+import
org.apache.nifi.registry.provider.extension.StandardExtensionBundleContext;
+import org.apache.nifi.registry.security.authorization.user.NiFiUserUtils;
+import org.apache.nifi.registry.service.DataModelMapper;
+import org.apache.nifi.registry.service.MetadataService;
+import org.apache.nifi.registry.util.FileUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Service;
+
+import javax.validation.ConstraintViolation;
+import javax.validation.ConstraintViolationException;
+import javax.validation.Validator;
+import java.io.BufferedInputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.security.DigestInputStream;
+import java.security.MessageDigest;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+@Service
+public class StandardExtensionService implements ExtensionService {
+
+private static final Logger LOGGER =
LoggerFactory.getLogger(StandardExtensionService.class);
+
+private final MetadataService metadataService;
+private final Map extractors;
+private final ExtensionBundlePersistenceProvider
bundlePersistenceProvider;
+private final Validator validator;
+private final File extensionsWorkingDir;
+
+@Autowired
+public StandardExtensionService(final MetadataService metadataService,
+final Map extractors,
+final
ExtensionBundlePersistenceProvider bundl
[jira] [Commented] (NIFIREG-211) Add extension bundles as a type of versioned item
[
https://issues.apache.org/jira/browse/NIFIREG-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701051#comment-16701051
]
ASF GitHub Bot commented on NIFIREG-211:
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236849819
--- Diff:
nifi-registry-core/nifi-registry-framework/src/main/java/org/apache/nifi/registry/service/extension/StandardExtensionService.java
---
@@ -0,0 +1,589 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.service.extension;
+
+import org.apache.commons.codec.binary.Hex;
+import org.apache.commons.codec.digest.DigestUtils;
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.Validate;
+import org.apache.nifi.registry.bucket.Bucket;
+import org.apache.nifi.registry.db.entity.BucketEntity;
+import org.apache.nifi.registry.db.entity.ExtensionBundleEntity;
+import org.apache.nifi.registry.db.entity.ExtensionBundleEntityType;
+import org.apache.nifi.registry.db.entity.ExtensionBundleVersionEntity;
+import org.apache.nifi.registry.exception.ResourceNotFoundException;
+import org.apache.nifi.registry.extension.BundleCoordinate;
+import org.apache.nifi.registry.extension.BundleDetails;
+import org.apache.nifi.registry.extension.BundleExtractor;
+import org.apache.nifi.registry.extension.ExtensionBundle;
+import org.apache.nifi.registry.extension.ExtensionBundleContext;
+import
org.apache.nifi.registry.extension.ExtensionBundlePersistenceProvider;
+import org.apache.nifi.registry.extension.ExtensionBundleType;
+import org.apache.nifi.registry.extension.ExtensionBundleVersion;
+import org.apache.nifi.registry.extension.ExtensionBundleVersionDependency;
+import org.apache.nifi.registry.extension.ExtensionBundleVersionMetadata;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoArtifact;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoBucket;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoGroup;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoVersionSummary;
+import org.apache.nifi.registry.properties.NiFiRegistryProperties;
+import
org.apache.nifi.registry.provider.extension.StandardExtensionBundleContext;
+import org.apache.nifi.registry.security.authorization.user.NiFiUserUtils;
+import org.apache.nifi.registry.service.DataModelMapper;
+import org.apache.nifi.registry.service.MetadataService;
+import org.apache.nifi.registry.util.FileUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Service;
+
+import javax.validation.ConstraintViolation;
+import javax.validation.ConstraintViolationException;
+import javax.validation.Validator;
+import java.io.BufferedInputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.security.DigestInputStream;
+import java.security.MessageDigest;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+@Service
+public class StandardExtensionService implements ExtensionService {
+
+private static final Logger LOGGER =
LoggerFactory.getLogger(StandardExtensionService.class);
+
+private final MetadataService metadataService;
+private final Map extractors;
+private final ExtensionBundlePersistenceProvider
bundlePersistenceProvider;
+private final Validator validator;
+private final File extensionsWorkingDi
[jira] [Commented] (NIFIREG-211) Add extension bundles as a type of versioned item
[ https://issues.apache.org/jira/browse/NIFIREG-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701049#comment-16701049 ] ASF GitHub Bot commented on NIFIREG-211: Github user bbende commented on a diff in the pull request: https://github.com/apache/nifi-registry/pull/148#discussion_r236849686 --- Diff: nifi-registry-core/nifi-registry-framework/src/main/resources/db/migration/V3__AddExtensions.sql --- @@ -0,0 +1,62 @@ +-- Licensed to the Apache Software Foundation (ASF) under one or more +-- contributor license agreements. See the NOTICE file distributed with +-- this work for additional information regarding copyright ownership. +-- The ASF licenses this file to You under the Apache License, Version 2.0 +-- (the "License"); you may not use this file except in compliance with +-- the License. You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, software +-- distributed under the License is distributed on an "AS IS" BASIS, +-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +-- See the License for the specific language governing permissions and +-- limitations under the License. + +CREATE TABLE EXTENSION_BUNDLE ( +ID VARCHAR(50) NOT NULL, +BUCKET_ID VARCHAR(50) NOT NULL, +BUNDLE_TYPE VARCHAR(200) NOT NULL, +GROUP_ID VARCHAR(500) NOT NULL, +ARTIFACT_ID VARCHAR(500) NOT NULL, +CONSTRAINT PK__EXTENSION_BUNDLE_ID PRIMARY KEY (ID), +CONSTRAINT FK__EXTENSION_BUNDLE_BUCKET_ITEM_ID FOREIGN KEY (ID) REFERENCES BUCKET_ITEM(ID) ON DELETE CASCADE, +CONSTRAINT FK__EXTENSION_BUNDLE_BUCKET_ID FOREIGN KEY(BUCKET_ID) REFERENCES BUCKET(ID) ON DELETE CASCADE, +CONSTRAINT UNIQUE__EXTENSION_BUNDLE_BUCKET_GROUP_ARTIFACT UNIQUE (BUCKET_ID, GROUP_ID, ARTIFACT_ID) +); + +CREATE TABLE EXTENSION_BUNDLE_VERSION ( +ID VARCHAR(50) NOT NULL, +EXTENSION_BUNDLE_ID VARCHAR(50) NOT NULL, +VERSION VARCHAR(100) NOT NULL, +DEPENDENCY_GROUP_ID VARCHAR(500), --- End diff -- That is a good point, it probably makes sense to separate out a separate table where bundle version can have one-to-many dependencies, and NARs will just so happen to only ever have one. Will work on this update. > Add extension bundles as a type of versioned item > - > > Key: NIFIREG-211 > URL: https://issues.apache.org/jira/browse/NIFIREG-211 > Project: NiFi Registry > Issue Type: Improvement >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Major > Fix For: 0.4.0 > > > This ticket is to capture the work for adding extension bundles to NiFi > Registry. > This work may require several follow on tickets, but at a high-level will > include some of the following: > - Add a new type of item called an extension bundle, where each bundle > can contain one ore extensions or APIs > > - Support bundles for traditional NiFi (aka NARs) and also bundles for > MiNiFi CPP > > - Ability to upload the binary artifact for a bundle and extract the > metadata about the bundle, and metadata about the extensions contained > in the bundle (more on this later) > > - Provide a pluggable storage provider for saving the content of each > extension bundle so that we can have different implementations like > local fileysystem, S3, and other object stores > > - Provide a REST API for listing and retrieving available bundles, > integrate this into the registry Java client and NiFi CLI > - Security considerations such as checksums and cryptographic signatures for > bundles -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-registry pull request #148: NIFIREG-211 Initial work for adding extenio...
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236850477
--- Diff:
nifi-registry-core/nifi-registry-web-api/src/main/java/org/apache/nifi/registry/web/api/ExtensionRepositoryResource.java
---
@@ -0,0 +1,378 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.web.api;
+
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiParam;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.Extension;
+import io.swagger.annotations.ExtensionProperty;
+import org.apache.nifi.registry.bucket.Bucket;
+import org.apache.nifi.registry.bucket.BucketItem;
+import org.apache.nifi.registry.event.EventService;
+import org.apache.nifi.registry.extension.ExtensionBundleVersion;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoArtifact;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoBucket;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoGroup;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoVersion;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoVersionSummary;
+import org.apache.nifi.registry.security.authorization.RequestAction;
+import org.apache.nifi.registry.service.AuthorizationService;
+import org.apache.nifi.registry.service.RegistryService;
+import
org.apache.nifi.registry.service.extension.ExtensionBundleVersionCoordinate;
+import org.apache.nifi.registry.web.link.LinkService;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Component;
+
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.Produces;
+import javax.ws.rs.core.Link;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import javax.ws.rs.core.StreamingOutput;
+import java.util.ArrayList;
+import java.util.Set;
+import java.util.SortedSet;
+
+@Component
+@Path("/extensions/repo")
+@Api(
+value = "extension_repository",
+description = "Interact with extension bundles via the hierarchy
of bucket/group/artifact/version.",
+authorizations = { @Authorization("Authorization") }
+)
+public class ExtensionRepositoryResource extends
AuthorizableApplicationResource {
+
+public static final String CONTENT_DISPOSITION_HEADER =
"content-disposition";
+private final RegistryService registryService;
+private final LinkService linkService;
+
+@Autowired
+public ExtensionRepositoryResource(
+final RegistryService registryService,
+final LinkService linkService,
+final AuthorizationService authorizationService,
+final EventService eventService) {
+super(authorizationService, eventService);
+this.registryService = registryService;
+this.linkService = linkService;
+}
+
+@GET
+@Consumes(MediaType.WILDCARD)
+@Produces(MediaType.APPLICATION_JSON)
+@ApiOperation(
+value = "Gets the names of the buckets the current user is
authorized for in order to browse the repo by bucket",
+response = ExtensionRepoBucket.class,
+responseContainer = "List",
+extensions = {
+@Extension(name = "access-policy", properties = {
+@ExtensionProperty(name = "action", value =
"read"),
+@ExtensionProperty(name = "resource", value =
"/buckets/{bucketId}") })
+}
--- End diff --
That sounds consistent with the getBuckets end-point, so I'll remove that.
---
[GitHub] nifi-registry pull request #148: NIFIREG-211 Initial work for adding extenio...
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236852286
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
+
+
+@ApiModelProperty(value = "The id of this version of the extension
bundle")
+public String getId() {
+return id;
+}
+
+public void setId(String id) {
+this.id = id;
+}
+
+@ApiModelProperty(value = "The id of the extension bundle this version
is for")
+public String getExtensionBundleId() {
+return extensionBundleId;
+}
+
+public void setExtensionBundleId(String extensionBundleId) {
+this.extensionBundleId = extensionBundleId;
+}
+
+@ApiModelProperty(value = "The id of the bucket the extension bundle
belongs to", required = true)
+public String getBucketId() {
+return bucketId;
+}
+
+public void setBucketId(String bucketId) {
+this.bucketId = bucketId;
+}
+
+@ApiModelProperty(value = "The version of the extension bundle")
+public String getVersion() {
+return version;
+}
+
+public void setVersion(String version) {
+this.version = version;
+}
+
+@ApiModelProperty(value = "The optional bundle dependency (i.e.
another bundle this bundle is dependent on)")
+public ExtensionBundleVersionDependency getDependency() {
+return dependency;
+}
+
+public void setDependency(ExtensionBundleVersionDependency dependency)
{
+this.dependency = dependency;
+}
+
+@ApiModelProperty(value = "The timestamp of the create date of this
version")
+public long getTimestamp() {
+return timestamp;
+}
+
+public void setTimestamp(long timestamp) {
+this.timestamp = timestamp;
+}
+
+@ApiModelProperty(value = "The identity that created this version")
+public String getAuthor() {
--- End diff --
Currently the author for extension bundles works the same as the author for
versioned flows, it is the value of NiFiUserUtils.getNiFiUserIdentity() during
the request that created the entity.
Personally I view this as ok since we have no way of knowing that the
client cert CN=alopresto is the same as the kerberos principal
alopre...@nifi.com, unless identity mappings are configured that map these to
the same value.
---
[jira] [Commented] (NIFIREG-211) Add extension bundles as a type of versioned item
[
https://issues.apache.org/jira/browse/NIFIREG-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701055#comment-16701055
]
ASF GitHub Bot commented on NIFIREG-211:
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236852286
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
+
+
+@ApiModelProperty(value = "The id of this version of the extension
bundle")
+public String getId() {
+return id;
+}
+
+public void setId(String id) {
+this.id = id;
+}
+
+@ApiModelProperty(value = "The id of the extension bundle this version
is for")
+public String getExtensionBundleId() {
+return extensionBundleId;
+}
+
+public void setExtensionBundleId(String extensionBundleId) {
+this.extensionBundleId = extensionBundleId;
+}
+
+@ApiModelProperty(value = "The id of the bucket the extension bundle
belongs to", required = true)
+public String getBucketId() {
+return bucketId;
+}
+
+public void setBucketId(String bucketId) {
+this.bucketId = bucketId;
+}
+
+@ApiModelProperty(value = "The version of the extension bundle")
+public String getVersion() {
+return version;
+}
+
+public void setVersion(String version) {
+this.version = version;
+}
+
+@ApiModelProperty(value = "The optional bundle dependency (i.e.
another bundle this bundle is dependent on)")
+public ExtensionBundleVersionDependency getDependency() {
+return dependency;
+}
+
+public void setDependency(ExtensionBundleVersionDependency dependency)
{
+this.dependency = dependency;
+}
+
+@ApiModelProperty(value = "The timestamp of the create date of this
version")
+public long getTimestamp() {
+return timestamp;
+}
+
+public void setTimestamp(long timestamp) {
+this.timestamp = timestamp;
+}
+
+@ApiModelProperty(value = "The identity that created this version")
+public String getAuthor() {
--- End diff --
Currently the author for extension bundles works the same as the author for
versioned flows, it is the value of NiFiUserUtils.getNiFiUserIdentity() during
the request that created the entity.
Personally I view this as ok since we have no way of knowing that the
client cert CN=alopresto is the same as the kerberos principal
alopre...@nifi.com, unless identity mappings are configured that map these to
the same value.
> Add extension bundles as a type of versioned item
>
[jira] [Commented] (NIFIREG-211) Add extension bundles as a type of versioned item
[
https://issues.apache.org/jira/browse/NIFIREG-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701052#comment-16701052
]
ASF GitHub Bot commented on NIFIREG-211:
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236850477
--- Diff:
nifi-registry-core/nifi-registry-web-api/src/main/java/org/apache/nifi/registry/web/api/ExtensionRepositoryResource.java
---
@@ -0,0 +1,378 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.web.api;
+
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiParam;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import io.swagger.annotations.Authorization;
+import io.swagger.annotations.Extension;
+import io.swagger.annotations.ExtensionProperty;
+import org.apache.nifi.registry.bucket.Bucket;
+import org.apache.nifi.registry.bucket.BucketItem;
+import org.apache.nifi.registry.event.EventService;
+import org.apache.nifi.registry.extension.ExtensionBundleVersion;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoArtifact;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoBucket;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoGroup;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoVersion;
+import org.apache.nifi.registry.extension.repo.ExtensionRepoVersionSummary;
+import org.apache.nifi.registry.security.authorization.RequestAction;
+import org.apache.nifi.registry.service.AuthorizationService;
+import org.apache.nifi.registry.service.RegistryService;
+import
org.apache.nifi.registry.service.extension.ExtensionBundleVersionCoordinate;
+import org.apache.nifi.registry.web.link.LinkService;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Component;
+
+import javax.ws.rs.Consumes;
+import javax.ws.rs.GET;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.Produces;
+import javax.ws.rs.core.Link;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import javax.ws.rs.core.StreamingOutput;
+import java.util.ArrayList;
+import java.util.Set;
+import java.util.SortedSet;
+
+@Component
+@Path("/extensions/repo")
+@Api(
+value = "extension_repository",
+description = "Interact with extension bundles via the hierarchy
of bucket/group/artifact/version.",
+authorizations = { @Authorization("Authorization") }
+)
+public class ExtensionRepositoryResource extends
AuthorizableApplicationResource {
+
+public static final String CONTENT_DISPOSITION_HEADER =
"content-disposition";
+private final RegistryService registryService;
+private final LinkService linkService;
+
+@Autowired
+public ExtensionRepositoryResource(
+final RegistryService registryService,
+final LinkService linkService,
+final AuthorizationService authorizationService,
+final EventService eventService) {
+super(authorizationService, eventService);
+this.registryService = registryService;
+this.linkService = linkService;
+}
+
+@GET
+@Consumes(MediaType.WILDCARD)
+@Produces(MediaType.APPLICATION_JSON)
+@ApiOperation(
+value = "Gets the names of the buckets the current user is
authorized for in order to browse the repo by bucket",
+response = ExtensionRepoBucket.class,
+responseContainer = "List",
+extensions = {
+@Extension(name = "access-policy", properties = {
+@ExtensionProperty(name = "action", value =
"read"),
+
[GitHub] nifi-registry pull request #148: NIFIREG-211 Initial work for adding extenio...
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236853713
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
+
+
+@ApiModelProperty(value = "The id of this version of the extension
bundle")
+public String getId() {
+return id;
+}
+
+public void setId(String id) {
+this.id = id;
+}
+
+@ApiModelProperty(value = "The id of the extension bundle this version
is for")
+public String getExtensionBundleId() {
+return extensionBundleId;
+}
+
+public void setExtensionBundleId(String extensionBundleId) {
+this.extensionBundleId = extensionBundleId;
+}
+
+@ApiModelProperty(value = "The id of the bucket the extension bundle
belongs to", required = true)
+public String getBucketId() {
+return bucketId;
+}
+
+public void setBucketId(String bucketId) {
+this.bucketId = bucketId;
+}
+
+@ApiModelProperty(value = "The version of the extension bundle")
+public String getVersion() {
+return version;
+}
+
+public void setVersion(String version) {
+this.version = version;
+}
+
+@ApiModelProperty(value = "The optional bundle dependency (i.e.
another bundle this bundle is dependent on)")
+public ExtensionBundleVersionDependency getDependency() {
+return dependency;
+}
+
+public void setDependency(ExtensionBundleVersionDependency dependency)
{
+this.dependency = dependency;
+}
+
+@ApiModelProperty(value = "The timestamp of the create date of this
version")
+public long getTimestamp() {
+return timestamp;
+}
+
+public void setTimestamp(long timestamp) {
+this.timestamp = timestamp;
+}
+
+@ApiModelProperty(value = "The identity that created this version")
+public String getAuthor() {
+return author;
+}
+
+public void setAuthor(String author) {
+this.author = author;
+}
+
+@ApiModelProperty(value = "The description for this version")
+public String getDescription() {
+return description;
+}
+
+public void setDescription(String description) {
+this.description = description;
+}
+
+@ApiModelProperty(value = "The hex representation of the SHA-256
digest of the binary content for this version")
+public String getSha256Hex() {
+return sha256Hex;
+}
+
+public void setSha256Hex(String sha256Hex) {
+this.sha256Hex = sha256Hex;
+}
+
+@Override
+public int compareTo(final Exten
[GitHub] nifi-registry pull request #148: NIFIREG-211 Initial work for adding extenio...
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236853819
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
+
+
+@ApiModelProperty(value = "The id of this version of the extension
bundle")
+public String getId() {
+return id;
+}
+
+public void setId(String id) {
+this.id = id;
+}
+
+@ApiModelProperty(value = "The id of the extension bundle this version
is for")
+public String getExtensionBundleId() {
+return extensionBundleId;
+}
+
+public void setExtensionBundleId(String extensionBundleId) {
+this.extensionBundleId = extensionBundleId;
+}
+
+@ApiModelProperty(value = "The id of the bucket the extension bundle
belongs to", required = true)
+public String getBucketId() {
+return bucketId;
+}
+
+public void setBucketId(String bucketId) {
+this.bucketId = bucketId;
+}
+
+@ApiModelProperty(value = "The version of the extension bundle")
+public String getVersion() {
+return version;
+}
+
+public void setVersion(String version) {
+this.version = version;
+}
+
+@ApiModelProperty(value = "The optional bundle dependency (i.e.
another bundle this bundle is dependent on)")
+public ExtensionBundleVersionDependency getDependency() {
+return dependency;
+}
+
+public void setDependency(ExtensionBundleVersionDependency dependency)
{
+this.dependency = dependency;
+}
+
+@ApiModelProperty(value = "The timestamp of the create date of this
version")
+public long getTimestamp() {
+return timestamp;
+}
+
+public void setTimestamp(long timestamp) {
+this.timestamp = timestamp;
+}
+
+@ApiModelProperty(value = "The identity that created this version")
+public String getAuthor() {
+return author;
+}
+
+public void setAuthor(String author) {
+this.author = author;
+}
+
+@ApiModelProperty(value = "The description for this version")
+public String getDescription() {
+return description;
+}
+
+public void setDescription(String description) {
+this.description = description;
+}
+
+@ApiModelProperty(value = "The hex representation of the SHA-256
digest of the binary content for this version")
+public String getSha256Hex() {
+return sha256Hex;
+}
+
+public void setSha256Hex(String sha256Hex) {
+this.sha256Hex = sha256Hex;
+}
+
+@Override
+public int compareTo(final Exten
[jira] [Commented] (NIFIREG-211) Add extension bundles as a type of versioned item
[
https://issues.apache.org/jira/browse/NIFIREG-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701062#comment-16701062
]
ASF GitHub Bot commented on NIFIREG-211:
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236853713
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
+
+
+@ApiModelProperty(value = "The id of this version of the extension
bundle")
+public String getId() {
+return id;
+}
+
+public void setId(String id) {
+this.id = id;
+}
+
+@ApiModelProperty(value = "The id of the extension bundle this version
is for")
+public String getExtensionBundleId() {
+return extensionBundleId;
+}
+
+public void setExtensionBundleId(String extensionBundleId) {
+this.extensionBundleId = extensionBundleId;
+}
+
+@ApiModelProperty(value = "The id of the bucket the extension bundle
belongs to", required = true)
+public String getBucketId() {
+return bucketId;
+}
+
+public void setBucketId(String bucketId) {
+this.bucketId = bucketId;
+}
+
+@ApiModelProperty(value = "The version of the extension bundle")
+public String getVersion() {
+return version;
+}
+
+public void setVersion(String version) {
+this.version = version;
+}
+
+@ApiModelProperty(value = "The optional bundle dependency (i.e.
another bundle this bundle is dependent on)")
+public ExtensionBundleVersionDependency getDependency() {
+return dependency;
+}
+
+public void setDependency(ExtensionBundleVersionDependency dependency)
{
+this.dependency = dependency;
+}
+
+@ApiModelProperty(value = "The timestamp of the create date of this
version")
+public long getTimestamp() {
+return timestamp;
+}
+
+public void setTimestamp(long timestamp) {
+this.timestamp = timestamp;
+}
+
+@ApiModelProperty(value = "The identity that created this version")
+public String getAuthor() {
+return author;
+}
+
+public void setAuthor(String author) {
+this.author = author;
+}
+
+@ApiModelProperty(value = "The description for this version")
+public String getDescription() {
+return description;
+}
+
+public void setDescription(String description) {
+this.description = description;
+}
+
+@ApiModelProperty(value = "The hex representation of the SHA-256
digest of the binary content for this version")
[jira] [Commented] (NIFIREG-211) Add extension bundles as a type of versioned item
[
https://issues.apache.org/jira/browse/NIFIREG-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701064#comment-16701064
]
ASF GitHub Bot commented on NIFIREG-211:
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236853819
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
+
+
+@ApiModelProperty(value = "The id of this version of the extension
bundle")
+public String getId() {
+return id;
+}
+
+public void setId(String id) {
+this.id = id;
+}
+
+@ApiModelProperty(value = "The id of the extension bundle this version
is for")
+public String getExtensionBundleId() {
+return extensionBundleId;
+}
+
+public void setExtensionBundleId(String extensionBundleId) {
+this.extensionBundleId = extensionBundleId;
+}
+
+@ApiModelProperty(value = "The id of the bucket the extension bundle
belongs to", required = true)
+public String getBucketId() {
+return bucketId;
+}
+
+public void setBucketId(String bucketId) {
+this.bucketId = bucketId;
+}
+
+@ApiModelProperty(value = "The version of the extension bundle")
+public String getVersion() {
+return version;
+}
+
+public void setVersion(String version) {
+this.version = version;
+}
+
+@ApiModelProperty(value = "The optional bundle dependency (i.e.
another bundle this bundle is dependent on)")
+public ExtensionBundleVersionDependency getDependency() {
+return dependency;
+}
+
+public void setDependency(ExtensionBundleVersionDependency dependency)
{
+this.dependency = dependency;
+}
+
+@ApiModelProperty(value = "The timestamp of the create date of this
version")
+public long getTimestamp() {
+return timestamp;
+}
+
+public void setTimestamp(long timestamp) {
+this.timestamp = timestamp;
+}
+
+@ApiModelProperty(value = "The identity that created this version")
+public String getAuthor() {
+return author;
+}
+
+public void setAuthor(String author) {
+this.author = author;
+}
+
+@ApiModelProperty(value = "The description for this version")
+public String getDescription() {
+return description;
+}
+
+public void setDescription(String description) {
+this.description = description;
+}
+
+@ApiModelProperty(value = "The hex representation of the SHA-256
digest of the binary content for this version")
[GitHub] nifi-registry pull request #148: NIFIREG-211 Initial work for adding extenio...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236859763
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
--- End diff --
I would like to have a conversation around the cryptographic signatures so
if it's possible to add a new field fairly easily, I'm ok waiting for this to
be in. I am curious if people think there should be any checksum coverage on
the metadata for the extension as well. I like having a calculation over the
binary, but I also think there is value in ensuring the metadata values are
checked as well. Looking for feedback, but this should be independent from this
PR.
---
[GitHub] nifi-registry pull request #148: NIFIREG-211 Initial work for adding extenio...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236860010
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
+
+
+@ApiModelProperty(value = "The id of this version of the extension
bundle")
+public String getId() {
+return id;
+}
+
+public void setId(String id) {
+this.id = id;
+}
+
+@ApiModelProperty(value = "The id of the extension bundle this version
is for")
+public String getExtensionBundleId() {
+return extensionBundleId;
+}
+
+public void setExtensionBundleId(String extensionBundleId) {
+this.extensionBundleId = extensionBundleId;
+}
+
+@ApiModelProperty(value = "The id of the bucket the extension bundle
belongs to", required = true)
+public String getBucketId() {
+return bucketId;
+}
+
+public void setBucketId(String bucketId) {
+this.bucketId = bucketId;
+}
+
+@ApiModelProperty(value = "The version of the extension bundle")
+public String getVersion() {
+return version;
+}
+
+public void setVersion(String version) {
+this.version = version;
+}
+
+@ApiModelProperty(value = "The optional bundle dependency (i.e.
another bundle this bundle is dependent on)")
+public ExtensionBundleVersionDependency getDependency() {
+return dependency;
+}
+
+public void setDependency(ExtensionBundleVersionDependency dependency)
{
+this.dependency = dependency;
+}
+
+@ApiModelProperty(value = "The timestamp of the create date of this
version")
+public long getTimestamp() {
+return timestamp;
+}
+
+public void setTimestamp(long timestamp) {
+this.timestamp = timestamp;
+}
+
+@ApiModelProperty(value = "The identity that created this version")
+public String getAuthor() {
--- End diff --
Understood. I think that long-term, we probably need to re-evaluate how
that is being done, as the identities between NiFi and NiFi Registry may not
always match.
---
[GitHub] nifi-registry pull request #148: NIFIREG-211 Initial work for adding extenio...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236860247
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
+
+
+@ApiModelProperty(value = "The id of this version of the extension
bundle")
+public String getId() {
+return id;
+}
+
+public void setId(String id) {
+this.id = id;
+}
+
+@ApiModelProperty(value = "The id of the extension bundle this version
is for")
+public String getExtensionBundleId() {
+return extensionBundleId;
+}
+
+public void setExtensionBundleId(String extensionBundleId) {
+this.extensionBundleId = extensionBundleId;
+}
+
+@ApiModelProperty(value = "The id of the bucket the extension bundle
belongs to", required = true)
+public String getBucketId() {
+return bucketId;
+}
+
+public void setBucketId(String bucketId) {
+this.bucketId = bucketId;
+}
+
+@ApiModelProperty(value = "The version of the extension bundle")
+public String getVersion() {
+return version;
+}
+
+public void setVersion(String version) {
+this.version = version;
+}
+
+@ApiModelProperty(value = "The optional bundle dependency (i.e.
another bundle this bundle is dependent on)")
+public ExtensionBundleVersionDependency getDependency() {
+return dependency;
+}
+
+public void setDependency(ExtensionBundleVersionDependency dependency)
{
+this.dependency = dependency;
+}
+
+@ApiModelProperty(value = "The timestamp of the create date of this
version")
+public long getTimestamp() {
+return timestamp;
+}
+
+public void setTimestamp(long timestamp) {
+this.timestamp = timestamp;
+}
+
+@ApiModelProperty(value = "The identity that created this version")
+public String getAuthor() {
+return author;
+}
+
+public void setAuthor(String author) {
+this.author = author;
+}
+
+@ApiModelProperty(value = "The description for this version")
+public String getDescription() {
+return description;
+}
+
+public void setDescription(String description) {
+this.description = description;
+}
+
+@ApiModelProperty(value = "The hex representation of the SHA-256
digest of the binary content for this version")
+public String getSha256Hex() {
+return sha256Hex;
+}
+
+public void setSha256Hex(String sha256Hex) {
+this.sha256Hex = sha256Hex;
+}
+
+@Override
+public int compareTo(final Ex
[jira] [Commented] (NIFIREG-211) Add extension bundles as a type of versioned item
[
https://issues.apache.org/jira/browse/NIFIREG-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701087#comment-16701087
]
ASF GitHub Bot commented on NIFIREG-211:
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236860247
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
+
+
+@ApiModelProperty(value = "The id of this version of the extension
bundle")
+public String getId() {
+return id;
+}
+
+public void setId(String id) {
+this.id = id;
+}
+
+@ApiModelProperty(value = "The id of the extension bundle this version
is for")
+public String getExtensionBundleId() {
+return extensionBundleId;
+}
+
+public void setExtensionBundleId(String extensionBundleId) {
+this.extensionBundleId = extensionBundleId;
+}
+
+@ApiModelProperty(value = "The id of the bucket the extension bundle
belongs to", required = true)
+public String getBucketId() {
+return bucketId;
+}
+
+public void setBucketId(String bucketId) {
+this.bucketId = bucketId;
+}
+
+@ApiModelProperty(value = "The version of the extension bundle")
+public String getVersion() {
+return version;
+}
+
+public void setVersion(String version) {
+this.version = version;
+}
+
+@ApiModelProperty(value = "The optional bundle dependency (i.e.
another bundle this bundle is dependent on)")
+public ExtensionBundleVersionDependency getDependency() {
+return dependency;
+}
+
+public void setDependency(ExtensionBundleVersionDependency dependency)
{
+this.dependency = dependency;
+}
+
+@ApiModelProperty(value = "The timestamp of the create date of this
version")
+public long getTimestamp() {
+return timestamp;
+}
+
+public void setTimestamp(long timestamp) {
+this.timestamp = timestamp;
+}
+
+@ApiModelProperty(value = "The identity that created this version")
+public String getAuthor() {
+return author;
+}
+
+public void setAuthor(String author) {
+this.author = author;
+}
+
+@ApiModelProperty(value = "The description for this version")
+public String getDescription() {
+return description;
+}
+
+public void setDescription(String description) {
+this.description = description;
+}
+
+@ApiModelProperty(value = "The hex representation of the SHA-256
digest of the binary content for this version
[jira] [Commented] (NIFIREG-211) Add extension bundles as a type of versioned item
[
https://issues.apache.org/jira/browse/NIFIREG-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701085#comment-16701085
]
ASF GitHub Bot commented on NIFIREG-211:
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236860010
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
+
+
+@ApiModelProperty(value = "The id of this version of the extension
bundle")
+public String getId() {
+return id;
+}
+
+public void setId(String id) {
+this.id = id;
+}
+
+@ApiModelProperty(value = "The id of the extension bundle this version
is for")
+public String getExtensionBundleId() {
+return extensionBundleId;
+}
+
+public void setExtensionBundleId(String extensionBundleId) {
+this.extensionBundleId = extensionBundleId;
+}
+
+@ApiModelProperty(value = "The id of the bucket the extension bundle
belongs to", required = true)
+public String getBucketId() {
+return bucketId;
+}
+
+public void setBucketId(String bucketId) {
+this.bucketId = bucketId;
+}
+
+@ApiModelProperty(value = "The version of the extension bundle")
+public String getVersion() {
+return version;
+}
+
+public void setVersion(String version) {
+this.version = version;
+}
+
+@ApiModelProperty(value = "The optional bundle dependency (i.e.
another bundle this bundle is dependent on)")
+public ExtensionBundleVersionDependency getDependency() {
+return dependency;
+}
+
+public void setDependency(ExtensionBundleVersionDependency dependency)
{
+this.dependency = dependency;
+}
+
+@ApiModelProperty(value = "The timestamp of the create date of this
version")
+public long getTimestamp() {
+return timestamp;
+}
+
+public void setTimestamp(long timestamp) {
+this.timestamp = timestamp;
+}
+
+@ApiModelProperty(value = "The identity that created this version")
+public String getAuthor() {
--- End diff --
Understood. I think that long-term, we probably need to re-evaluate how
that is being done, as the identities between NiFi and NiFi Registry may not
always match.
> Add extension bundles as a type of versioned item
> -
>
> Key: NIFIREG-211
> URL: https://issues.apache.org/jira/browse/NIFIREG-211
> Project: NiFi Registry
> Issue Type: Improvement
>Reporter: Bryan Bende
>Assignee: Bryan B
[jira] [Commented] (NIFIREG-211) Add extension bundles as a type of versioned item
[
https://issues.apache.org/jira/browse/NIFIREG-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701083#comment-16701083
]
ASF GitHub Bot commented on NIFIREG-211:
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236859763
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
--- End diff --
I would like to have a conversation around the cryptographic signatures so
if it's possible to add a new field fairly easily, I'm ok waiting for this to
be in. I am curious if people think there should be any checksum coverage on
the metadata for the extension as well. I like having a calculation over the
binary, but I also think there is value in ensuring the metadata values are
checked as well. Looking for feedback, but this should be independent from this
PR.
> Add extension bundles as a type of versioned item
> -
>
> Key: NIFIREG-211
> URL: https://issues.apache.org/jira/browse/NIFIREG-211
> Project: NiFi Registry
> Issue Type: Improvement
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Major
> Fix For: 0.4.0
>
>
> This ticket is to capture the work for adding extension bundles to NiFi
> Registry.
> This work may require several follow on tickets, but at a high-level will
> include some of the following:
> - Add a new type of item called an extension bundle, where each bundle
> can contain one ore extensions or APIs
>
> - Support bundles for traditional NiFi (aka NARs) and also bundles for
> MiNiFi CPP
>
> - Ability to upload the binary artifact for a bundle and extract the
> metadata about the bundle, and metadata about the extensions contained
> in the bundle (more on this later)
>
> - Provide a pluggable storage provider for saving the content of each
> extension bundle so that we can have different implementations like
> local fileysystem, S3, and other object stores
>
> - Provide a REST API for listing and retrieving available bundles,
> integrate this into the registry Java client and NiFi CLI
> - Security considerations such as checksums and cryptographic signatures for
> bundles
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #3183: NIFI-5826 Fix to escaped backslash
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/3183 +1, LGTM ---
[jira] [Commented] (NIFI-5826) UpdateRecord processor throwing PatternSyntaxException
[
https://issues.apache.org/jira/browse/NIFI-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701099#comment-16701099
]
ASF GitHub Bot commented on NIFI-5826:
--
Github user ottobackwards commented on the issue:
https://github.com/apache/nifi/pull/3183
+1, LGTM
> UpdateRecord processor throwing PatternSyntaxException
> --
>
> Key: NIFI-5826
> URL: https://issues.apache.org/jira/browse/NIFI-5826
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
> Environment: Nifi in docker container
>Reporter: ravi kargam
>Assignee: Ed Berezitsky
>Priority: Minor
> Attachments: UpdateRecord_Config_Exception.JPG
>
>
> with replacement value strategy as Record Path Value,
> I am trying to replace square bracket symbol *[* with parenthesis symbol *(*
> in my employeeName column in my csvReader structure with below syntax
> replaceRegex(/employeeName, "[\[]", "(")
> Processor is throwing following exception.
> RecordPathException: java.util.regex.PatternSyntaxException: Unclosed
> character class near index 4 [\\[]
> It worked fine with other special characters such as \{, }, <, >, ;, _, "
> For double qoute ("), i had to use single escape character, for above listed
> other characters, worked fine without any escape character. Other folks in
> Nifi Slack tried \s, \d, \w, \.
> looks like non of them worked.
> replace function worked for replacing [ and ]characters. didn't test any
> other characters.
> Please address resolve the issue.
> Regards,
> Ravi
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi-registry pull request #148: NIFIREG-211 Initial work for adding extenio...
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236865785
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
--- End diff --
In this case, all of the metadata comes from extracting information from
within the binary (i.e. reading the MANIFEST file inside the NAR, or reading
the symbol table of a C++ object), so wouldn't a checksum of only the binary
suffice then?
---
[jira] [Commented] (NIFIREG-211) Add extension bundles as a type of versioned item
[
https://issues.apache.org/jira/browse/NIFIREG-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701107#comment-16701107
]
ASF GitHub Bot commented on NIFIREG-211:
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236865785
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
--- End diff --
In this case, all of the metadata comes from extracting information from
within the binary (i.e. reading the MANIFEST file inside the NAR, or reading
the symbol table of a C++ object), so wouldn't a checksum of only the binary
suffice then?
> Add extension bundles as a type of versioned item
> -
>
> Key: NIFIREG-211
> URL: https://issues.apache.org/jira/browse/NIFIREG-211
> Project: NiFi Registry
> Issue Type: Improvement
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Major
> Fix For: 0.4.0
>
>
> This ticket is to capture the work for adding extension bundles to NiFi
> Registry.
> This work may require several follow on tickets, but at a high-level will
> include some of the following:
> - Add a new type of item called an extension bundle, where each bundle
> can contain one ore extensions or APIs
>
> - Support bundles for traditional NiFi (aka NARs) and also bundles for
> MiNiFi CPP
>
> - Ability to upload the binary artifact for a bundle and extract the
> metadata about the bundle, and metadata about the extensions contained
> in the bundle (more on this later)
>
> - Provide a pluggable storage provider for saving the content of each
> extension bundle so that we can have different implementations like
> local fileysystem, S3, and other object stores
>
> - Provide a REST API for listing and retrieving available bundles,
> integrate this into the registry Java client and NiFi CLI
> - Security considerations such as checksums and cryptographic signatures for
> bundles
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5826) UpdateRecord processor throwing PatternSyntaxException
[
https://issues.apache.org/jira/browse/NIFI-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ed Berezitsky updated NIFI-5826:
Affects Version/s: 1.6.0
1.7.0
1.8.0
1.7.1
Attachment: NIFI-5826_PR-3183.patch
Status: Patch Available (was: In Progress)
> UpdateRecord processor throwing PatternSyntaxException
> --
>
> Key: NIFI-5826
> URL: https://issues.apache.org/jira/browse/NIFI-5826
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.7.1, 1.8.0, 1.7.0, 1.6.0, 1.5.0
> Environment: Nifi in docker container
>Reporter: ravi kargam
>Assignee: Ed Berezitsky
>Priority: Minor
> Attachments: NIFI-5826_PR-3183.patch,
> UpdateRecord_Config_Exception.JPG
>
>
> with replacement value strategy as Record Path Value,
> I am trying to replace square bracket symbol *[* with parenthesis symbol *(*
> in my employeeName column in my csvReader structure with below syntax
> replaceRegex(/employeeName, "[\[]", "(")
> Processor is throwing following exception.
> RecordPathException: java.util.regex.PatternSyntaxException: Unclosed
> character class near index 4 [\\[]
> It worked fine with other special characters such as \{, }, <, >, ;, _, "
> For double qoute ("), i had to use single escape character, for above listed
> other characters, worked fine without any escape character. Other folks in
> Nifi Slack tried \s, \d, \w, \.
> looks like non of them worked.
> replace function worked for replacing [ and ]characters. didn't test any
> other characters.
> Please address resolve the issue.
> Regards,
> Ravi
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-5845) Support OTHER and SQLXML JDBC types in database processors
Matt Burgess created NIFI-5845: -- Summary: Support OTHER and SQLXML JDBC types in database processors Key: NIFI-5845 URL: https://issues.apache.org/jira/browse/NIFI-5845 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: Matt Burgess Currently, OTHER and SQLXML JDBC types are not supported by the database processors. When the processor(s) try to create a schema, it does not recognize these types and will fail. For example, this happens with PostgreSQL's JSONB and XML types, respectively. The meaning of OTHER is "the SQL type is database-specific and gets mapped to a Java object that can be accessed via the methods getObject and setObject." Since they are Java Objects, we could represent them as Strings in the schema and use the value of toString(). This doesn't allow us to represent the JSON fields as a nested record, but we don't have enough information per se, as an OTHER value could be anything. Perhaps in a future improvement we could try to parse the object as JSON, and continue if successful. However the schema currently has to be determined beforehand, so I'm not sure if that would work. For SQLXML, we can use the SQLXML interface to get at the String or even an InputStream, and since we know it's XML, we could parse it into records the same way XMLRecordReader does. However that would/should involve some refactor, to get the common utilities/methods into something like a nifi-xml-record-utils package. Also for this Jira it would be inconsistent with the handling of JSON fields (if reported as type OTHER). This might make a good companion Jira to the aforementioned one, to try and change JSON/XML fields into nested records. I verified these types with PostgreSQL's JSON, JSONB, and XML types, as well as MySQL's JSON type. Oracle's XMLTYPE returns a different code (2007), and I couldn't find a way to set the connection property to return SQLXML instead. Even with that, you have to add two additional JARs to the classpath, and in my experiments it didn't seem to work when I added them to the Driver Resources property. Apache Derby doesn't have either type so I wasn't able to write unit tests to illustrate this. This Jira proposes to add support for OTHER and SQLXML JDBC types to the database processors, interpreting their values as Strings. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-5845) Support OTHER and SQLXML JDBC types in database processors
[ https://issues.apache.org/jira/browse/NIFI-5845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-5845: -- Assignee: Matt Burgess > Support OTHER and SQLXML JDBC types in database processors > -- > > Key: NIFI-5845 > URL: https://issues.apache.org/jira/browse/NIFI-5845 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > Currently, OTHER and SQLXML JDBC types are not supported by the database > processors. When the processor(s) try to create a schema, it does not > recognize these types and will fail. For example, this happens with > PostgreSQL's JSONB and XML types, respectively. > The meaning of OTHER is "the SQL type is database-specific and gets mapped to > a Java object that can be accessed via the methods getObject and setObject." > Since they are Java Objects, we could represent them as Strings in the schema > and use the value of toString(). This doesn't allow us to represent the JSON > fields as a nested record, but we don't have enough information per se, as an > OTHER value could be anything. Perhaps in a future improvement we could try > to parse the object as JSON, and continue if successful. However the schema > currently has to be determined beforehand, so I'm not sure if that would work. > For SQLXML, we can use the SQLXML interface to get at the String or even an > InputStream, and since we know it's XML, we could parse it into records the > same way XMLRecordReader does. However that would/should involve some > refactor, to get the common utilities/methods into something like a > nifi-xml-record-utils package. Also for this Jira it would be inconsistent > with the handling of JSON fields (if reported as type OTHER). This might make > a good companion Jira to the aforementioned one, to try and change JSON/XML > fields into nested records. > I verified these types with PostgreSQL's JSON, JSONB, and XML types, as well > as MySQL's JSON type. Oracle's XMLTYPE returns a different code (2007), and I > couldn't find a way to set the connection property to return SQLXML instead. > Even with that, you have to add two additional JARs to the classpath, and in > my experiments it didn't seem to work when I added them to the Driver > Resources property. Apache Derby doesn't have either type so I wasn't able to > write unit tests to illustrate this. > This Jira proposes to add support for OTHER and SQLXML JDBC types to the > database processors, interpreting their values as Strings. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3184: NIFI-5845: Add support for OTHER and SQLXML JDBC ty...
GitHub user mattyb149 opened a pull request: https://github.com/apache/nifi/pull/3184 NIFI-5845: Add support for OTHER and SQLXML JDBC types to SQL/Hive processors Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mattyb149/nifi NIFI-5845 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3184.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3184 commit a8009f9a0010b99703713a38263006a7a9e5f5d5 Author: Matthew Burgess Date: 2018-11-27T23:35:29Z NIFI-5845: Add support for OTHER and SQLXML JDBC types to SQL/Hive processors ---
[jira] [Updated] (NIFI-5845) Support OTHER and SQLXML JDBC types in database processors
[ https://issues.apache.org/jira/browse/NIFI-5845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-5845: --- Status: Patch Available (was: In Progress) > Support OTHER and SQLXML JDBC types in database processors > -- > > Key: NIFI-5845 > URL: https://issues.apache.org/jira/browse/NIFI-5845 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > Currently, OTHER and SQLXML JDBC types are not supported by the database > processors. When the processor(s) try to create a schema, it does not > recognize these types and will fail. For example, this happens with > PostgreSQL's JSONB and XML types, respectively. > The meaning of OTHER is "the SQL type is database-specific and gets mapped to > a Java object that can be accessed via the methods getObject and setObject." > Since they are Java Objects, we could represent them as Strings in the schema > and use the value of toString(). This doesn't allow us to represent the JSON > fields as a nested record, but we don't have enough information per se, as an > OTHER value could be anything. Perhaps in a future improvement we could try > to parse the object as JSON, and continue if successful. However the schema > currently has to be determined beforehand, so I'm not sure if that would work. > For SQLXML, we can use the SQLXML interface to get at the String or even an > InputStream, and since we know it's XML, we could parse it into records the > same way XMLRecordReader does. However that would/should involve some > refactor, to get the common utilities/methods into something like a > nifi-xml-record-utils package. Also for this Jira it would be inconsistent > with the handling of JSON fields (if reported as type OTHER). This might make > a good companion Jira to the aforementioned one, to try and change JSON/XML > fields into nested records. > I verified these types with PostgreSQL's JSON, JSONB, and XML types, as well > as MySQL's JSON type. Oracle's XMLTYPE returns a different code (2007), and I > couldn't find a way to set the connection property to return SQLXML instead. > Even with that, you have to add two additional JARs to the classpath, and in > my experiments it didn't seem to work when I added them to the Driver > Resources property. Apache Derby doesn't have either type so I wasn't able to > write unit tests to illustrate this. > This Jira proposes to add support for OTHER and SQLXML JDBC types to the > database processors, interpreting their values as Strings. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5845) Support OTHER and SQLXML JDBC types in database processors
[ https://issues.apache.org/jira/browse/NIFI-5845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701170#comment-16701170 ] ASF GitHub Bot commented on NIFI-5845: -- GitHub user mattyb149 opened a pull request: https://github.com/apache/nifi/pull/3184 NIFI-5845: Add support for OTHER and SQLXML JDBC types to SQL/Hive processors Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mattyb149/nifi NIFI-5845 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3184.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3184 commit a8009f9a0010b99703713a38263006a7a9e5f5d5 Author: Matthew Burgess Date: 2018-11-27T23:35:29Z NIFI-5845: Add support for OTHER and SQLXML JDBC types to SQL/Hive processors > Support OTHER and SQLXML JDBC types in database processors > -- > > Key: NIFI-5845 > URL: https://issues.apache.org/jira/browse/NIFI-5845 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > Currently, OTHER and SQLXML JDBC types are not supported by the database > processors. When the processor(s) try to create a schema, it does not > recognize these types and will fail. For example, this happens with > PostgreSQL's JSONB and XML types, respectively. > The meaning of OTHER is "the SQL type is database-specific and gets mapped to > a Java object that can be accessed via the methods getObject and setObject." > Since they are Java Objects, we could represent them as Strings in the schema > and use the value of toString(). This doesn't allow us to represent the JSON > fields as a nested record, but we don't have enough information per se, as an > OTHER value could be anything. Perhaps in a future improvement we could try > to parse the object as JSON, and continue if successful. However the schema > currently has to be determined beforehand, so I'm not sure if that would work. > For SQLXML, we can use the SQLXML interface to get at the String or even an > InputStream, and since we know it's XML, we could parse it into records the > same way XMLRecordReader does. However that would/should involve some > refactor, to get the common utilities/methods into something like a > nifi-xml-record-utils package. Also for this Jira it would be inconsistent > with the handling of JSON fields (if reported as type OTHER). This might make > a good companion Jira to the aforementioned one, to try and change JSON/XML > fields into nested records. > I verified these types with PostgreSQL's JSON, JSONB, and XML types, as well > as MySQL's JSON type. Oracle's XMLTYPE returns a different code (2007), and I > couldn't find a way to set the connection property to return SQLXML instead. > Even with that, you hav
[GitHub] nifi-registry pull request #148: NIFIREG-211 Initial work for adding extenio...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236898636
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
--- End diff --
Ok. Is there any way to craft an instance of a binary with different
metadata that generates the same checksum? Can you point me to the code which
performs this inspection and calculation? Where does the checksum reside in
transmission (i.e. the binary and checksum are sent separately, or it's just
calculated on one end and there is nothing to compare it to?)?
---
[jira] [Commented] (NIFIREG-211) Add extension bundles as a type of versioned item
[
https://issues.apache.org/jira/browse/NIFIREG-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701201#comment-16701201
]
ASF GitHub Bot commented on NIFIREG-211:
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236898636
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
--- End diff --
Ok. Is there any way to craft an instance of a binary with different
metadata that generates the same checksum? Can you point me to the code which
performs this inspection and calculation? Where does the checksum reside in
transmission (i.e. the binary and checksum are sent separately, or it's just
calculated on one end and there is nothing to compare it to?)?
> Add extension bundles as a type of versioned item
> -
>
> Key: NIFIREG-211
> URL: https://issues.apache.org/jira/browse/NIFIREG-211
> Project: NiFi Registry
> Issue Type: Improvement
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Major
> Fix For: 0.4.0
>
>
> This ticket is to capture the work for adding extension bundles to NiFi
> Registry.
> This work may require several follow on tickets, but at a high-level will
> include some of the following:
> - Add a new type of item called an extension bundle, where each bundle
> can contain one ore extensions or APIs
>
> - Support bundles for traditional NiFi (aka NARs) and also bundles for
> MiNiFi CPP
>
> - Ability to upload the binary artifact for a bundle and extract the
> metadata about the bundle, and metadata about the extensions contained
> in the bundle (more on this later)
>
> - Provide a pluggable storage provider for saving the content of each
> extension bundle so that we can have different implementations like
> local fileysystem, S3, and other object stores
>
> - Provide a REST API for listing and retrieving available bundles,
> integrate this into the registry Java client and NiFi CLI
> - Security considerations such as checksums and cryptographic signatures for
> bundles
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (NIFI-5550) Cluster should handle replicating DELETE request with non-zero content length header
[
https://issues.apache.org/jira/browse/NIFI-5550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andy LoPresto resolved NIFI-5550.
-
Resolution: Fixed
Fix Version/s: 1.8.0
This issue was resolved by PR 3035 in NIFI-5628.
> Cluster should handle replicating DELETE request with non-zero content length
> header
>
>
> Key: NIFI-5550
> URL: https://issues.apache.org/jira/browse/NIFI-5550
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0, 1.6.0, 1.7.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Critical
> Labels: cluster, http, request_replication, security
> Fix For: 1.8.0
>
>
> When sending a {{DELETE}} request to a node in the cluster, the cluster
> coordinator attempts to replicate the request to all nodes in the cluster
> before committing it (*two phase commit*). According to [RFC
> 7231|https://tools.ietf.org/html/rfc7231#section-4.3.5], a {{DELETE}} request
> *may* have a body, but the body has no defined purpose:
> {quote}
> A payload within a DELETE request message has no defined semantics; sending a
> payload body on a DELETE request might cause some existing implementations to
> reject the request.
> {quote}
> The cluster request replication does not expect a request body when a DELETE
> request is received, so when it replicates the request to other nodes, it
> does not include the body, if present. However, the {{Content-Length}} header
> is forwarded in tact, and if this header contains a non-zero length, the
> other nodes will wait to receive the expected body. This can cause request
> time outs.
> The solution is to intercept {{DELETE}} requests with a non-zero
> {{Content-Length}} header and overwrite that value to {{0}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #3182: NIFI-5838 - Improve the schema validation method in Kite p...
Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/3182 @pvillard31 I understand the intent of this change and it looks good if we want to invalidate schema URL setting if it doesn't have URI.schema. Just wondering if we close the `FileSystem fs`, too, the lingering `BLOCKED on org.apache.hadoop.ipc.Client$Connection` threads can be discarded? Current code only closes opened input streams, but doesn't close FileSystem. ---
[jira] [Commented] (NIFI-5838) Misconfigured Kite processors can block NiFi
[
https://issues.apache.org/jira/browse/NIFI-5838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701322#comment-16701322
]
ASF GitHub Bot commented on NIFI-5838:
--
Github user ijokarumawak commented on the issue:
https://github.com/apache/nifi/pull/3182
@pvillard31 I understand the intent of this change and it looks good if we
want to invalidate schema URL setting if it doesn't have URI.schema.
Just wondering if we close the `FileSystem fs`, too, the lingering `BLOCKED
on org.apache.hadoop.ipc.Client$Connection` threads can be discarded? Current
code only closes opened input streams, but doesn't close FileSystem.
> Misconfigured Kite processors can block NiFi
>
>
> Key: NIFI-5838
> URL: https://issues.apache.org/jira/browse/NIFI-5838
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Critical
>
> I faced the following situation today: no way to access the NiFi UI (it was
> just hanging forever or throwing timeouts), no change in case of NiFi
> restart, even with disabling the auto resume feature.
> By looking at the thread dump, I found a lot of:
> {noformat}
> "NiFi Web Server-142" Id=142 BLOCKED on
> org.apache.hadoop.ipc.Client$Connection@3843e7a6
> at org.apache.hadoop.ipc.Client$Connection.addCall(Client.java:463)
> at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:375)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1522)
> at org.apache.hadoop.ipc.Client.call(Client.java:1451)
> at org.apache.hadoop.ipc.Client.call(Client.java:1412)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy348.getBlockLocations(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:255)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy349.getBlockLocations(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1226)
> at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1213)
> at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1201)
> at
> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:306)
> at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:272)
> - waiting on java.lang.Object@46d84263
> at org.apache.hadoop.hdfs.DFSInputStream.(DFSInputStream.java:264)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1526)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:304)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:299)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:312)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
> at
> org.apache.nifi.processors.kite.AbstractKiteProcessor.getSchema(AbstractKiteProcessor.java:132)
> at
> org.apache.nifi.processors.kite.AbstractKiteProcessor$3.validate(AbstractKiteProcessor.java:172)
> at
> org.apache.nifi.components.PropertyDescriptor.validate(PropertyDescriptor.java:200)
> at
> org.apache.nifi.components.AbstractConfigurableComponent.validate(AbstractConfigurableComponent.java:103)
> at
> org.apache.nifi.controller.AbstractConfiguredComponent.validate(AbstractConfiguredComponent.java:329)
> at
> org.apache.nifi.controller.StandardProcessorNode.isValid(StandardProcessorNode.java:968)
> at
> org.apache.nifi.groups.StandardProcessGroup.getCounts(StandardProcessGroup.java:227)
> at
> org.apache.nifi.groups.StandardProcessGroup.getCounts(StandardProcessGroup.java:261)
> at
> org.apache.nifi.groups.StandardProcessGroup.getCounts(StandardProcessGroup.java:261)
> at
> org.apache.nifi.groups.StandardProcessGroup.getCounts(StandardProcessGroup.java:261)
> at
> org.apache.nifi.web.StandardNiFiServiceFacade.getSiteToSiteDetails(StandardNiFiServiceFacade.java:2609)
> at
> o
[jira] [Commented] (NIFI-5842) AdditionalDetails of Kafka Processor0.10+ is not correct.
[
https://issues.apache.org/jira/browse/NIFI-5842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701343#comment-16701343
]
Koji Kawamura commented on NIFI-5842:
-
Hi [~kei miyauchi]
There is also a note on dynamic JAAS config in the additionalDetails:
{quote}Alternatively, starting with Apache NiFi 1.2.0 which uses the Kafka
0.10.2 client, the JAAS configuration when using GSSAPI can be provided by
specifying the Kerberos Principal and Kerberos Keytab directly in the processor
properties. This will dynamically create a JAAS configuration like above, and
will take precedence over the java.security.auth.login.config system property.
{quote}
Does the note suffice? If you have any idea to make it more clear, please let
us know. Thanks!
> AdditionalDetails of Kafka Processor0.10+ is not correct.
> -
>
> Key: NIFI-5842
> URL: https://issues.apache.org/jira/browse/NIFI-5842
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: kei miyauchi
>Priority: Trivial
>
> On additionalDetails of Kafka processor say:
> _> If the SASL mechanism is GSSAPI, then the client must provide a JAAS
> configuration to authenticate_
> _> If the SASL mechanism is PLAIN, then client must provide a JAAS
> configuration to authenticate_
> This is already not true because of NIFI-3528.
> Setting Principal and Keytab is enough.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5843) Unclear validation message for ScriptingComponentHelper
[
https://issues.apache.org/jira/browse/NIFI-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701352#comment-16701352
]
Koji Kawamura commented on NIFI-5843:
-
[~alex_savitsky] Thanks for the suggestion.
Alternatively, we could combine the subjects like:
results.add(new ValidationResult.Builder().subject("Script Body or Script
File").valid(false).explanation("exactly one of Script File or Script
Body must be set").build());
Do you mind submitting a PR if possible? I'm happy to review. Please refer the
Contributor Guide for details and ask anything not clear.
https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide#ContributorGuide-providingCodeOrDocumentationContributionProvidingcodeordocumentationcontributions
> Unclear validation message for ScriptingComponentHelper
> ---
>
> Key: NIFI-5843
> URL: https://issues.apache.org/jira/browse/NIFI-5843
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.8.0
>Reporter: Alex Savitsky
>Priority: Trivial
>
> ScriptingComponentHelper.customValidate (from nifi-scripting-processors)
> creates an unclear validation message
> {quote}'' is invalid because Exactly one of Script File or Script Body must
> be set
> {quote}
> as it doesn't specify the validation subject. Since there are technically two
> invalid subjects in this case (Script File and Script Body), I suggest adding
> two validation messages, one for each subject.
> Current code:
> {code:java}
> results.add(new ValidationResult.Builder().valid(false).explanation(
> "Exactly one of Script File or Script Body must be set").build());
> {code}
> Proposed fix:
> {code:java}
> results.add(new ValidationResult.Builder().subject("Script
> Body").valid(false).explanation(
> "exactly one of Script File or Script Body must be set").build());
> results.add(new ValidationResult.Builder().subject("Script
> File").valid(false).explanation(
> "exactly one of Script File or Script Body must be set").build());
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3184: NIFI-5845: Add support for OTHER and SQLXML JDBC ty...
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3184#discussion_r236940096
--- Diff:
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/util/hive/HiveJdbcCommon.java
---
@@ -402,6 +409,9 @@ public static long convertToCsvStream(final ResultSet
rs, final OutputStream out
rowValues.add("");
}
break;
+case SQLXML:
+
rowValues.add(StringEscapeUtils.escapeCsv(((SQLXML) value).getString()));
--- End diff --
Shouldn't we take care of null `value` here?
---
[GitHub] nifi pull request #3184: NIFI-5845: Add support for OTHER and SQLXML JDBC ty...
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3184#discussion_r236940080
--- Diff:
nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors/src/main/java/org/apache/nifi/util/hive/HiveJdbcCommon.java
---
@@ -402,6 +409,9 @@ public static long convertToCsvStream(final ResultSet
rs, final OutputStream out
rowValues.add("");
}
break;
+case SQLXML:
+
rowValues.add(StringEscapeUtils.escapeCsv(((java.sql.SQLXML)
value).getString()));
--- End diff --
Shouldn't we take care of null `value` here, too?
---
