[jira] [Commented] (NIFI-2115) Enhanced About Box Version Information
[ https://issues.apache.org/jira/browse/NIFI-2115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355901#comment-15355901 ] James Wing commented on NIFI-2115: -- [~rmoran], I agree that changes to the About dialog should be modest. It may not be obvious from the screenshot in the PR, but initial impact to the About dialog in my code so far is just "..." following the version, which expands the detail info on click. I had not considered the System Diagnostics dialog, but that may be a good idea. We would have plenty of real estate on a new tab there. The downside is that it is tucked away a bit, and the version in the about dialog will appear to be complete. Maybe we could link from About to the System Diagnostics for detail? Cluster versions are still not clear to me. The version information appears to be particular to whichever API endpoint you happen to be talking to. I think you are correct that we would need to enumerate the nodes and their respective versions. > Enhanced About Box Version Information > -- > > Key: NIFI-2115 > URL: https://issues.apache.org/jira/browse/NIFI-2115 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI, Tools and Build >Affects Versions: 1.0.0 >Reporter: James Wing >Assignee: James Wing >Priority: Minor > > The UI's About dialog and underlying API provide the version of NiFi, like > "0.7.0-SNAPSHOT". For many bug reports and troubleshooting requests, this is > not very precise, especially around rapidly changing code or > platform-dependent behavior. It would help if NiFi captured and displayed > additional information: > * NiFi build commit hash > * NiFi build branch > * NiFi build date/time > * Java Version > * Java Vendor (Oracle, OpenJDK) > * OS > Then a simple copy/paste from the about box would provide very specific > information about a particular NiFi installation and which code it was > derived from. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2115) Enhanced About Box Version Information
[ https://issues.apache.org/jira/browse/NIFI-2115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355866#comment-15355866 ] James Wing commented on NIFI-2115: -- [~mosermw], I agree, I will try to figure out a more graceful and invisible scheme for when build number and branch are not available. > Enhanced About Box Version Information > -- > > Key: NIFI-2115 > URL: https://issues.apache.org/jira/browse/NIFI-2115 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI, Tools and Build >Affects Versions: 1.0.0 >Reporter: James Wing >Assignee: James Wing >Priority: Minor > > The UI's About dialog and underlying API provide the version of NiFi, like > "0.7.0-SNAPSHOT". For many bug reports and troubleshooting requests, this is > not very precise, especially around rapidly changing code or > platform-dependent behavior. It would help if NiFi captured and displayed > additional information: > * NiFi build commit hash > * NiFi build branch > * NiFi build date/time > * Java Version > * Java Vendor (Oracle, OpenJDK) > * OS > Then a simple copy/paste from the about box would provide very specific > information about a particular NiFi installation and which code it was > derived from. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2115) Enhanced About Box Version Information
[ https://issues.apache.org/jira/browse/NIFI-2115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15349829#comment-15349829 ] James Wing commented on NIFI-2115: -- Good points, [~joewitt], I don't have great answers: * I admit I have not considered a cluster scenario carefully. I tried to follow the existing data path to the about API and UI dialog. I'll look into how that works for a cluster. * I assumed (wrongly?) that releases were built from git rather than git -> source -> convenience binaries. I'll read up on the release process and consider what the options are. > Enhanced About Box Version Information > -- > > Key: NIFI-2115 > URL: https://issues.apache.org/jira/browse/NIFI-2115 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI, Tools and Build >Affects Versions: 1.0.0 >Reporter: James Wing >Assignee: James Wing >Priority: Minor > > The UI's About dialog and underlying API provide the version of NiFi, like > "0.7.0-SNAPSHOT". For many bug reports and troubleshooting requests, this is > not very precise, especially around rapidly changing code or > platform-dependent behavior. It would help if NiFi captured and displayed > additional information: > * NiFi build commit hash > * NiFi build branch > * NiFi build date/time > * Java Version > * Java Vendor (Oracle, OpenJDK) > * OS > Then a simple copy/paste from the about box would provide very specific > information about a particular NiFi installation and which code it was > derived from. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2115) Enhanced About Box Version Information
[
https://issues.apache.org/jira/browse/NIFI-2115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15349823#comment-15349823
]
James Wing commented on NIFI-2115:
--
I tested building from a source zip. There are two concerns -- building
outside a Git repo that provides commit and branch info, and building without
Git tools installed at all. The plugin included in this PR,
buildnumber-maven-plugin, has a {{revisionOnScmFailure}} property configured to
"unknown source" when there is no git environment. The plugin will log
warnings like the following:
{code}
[WARNING] Cannot get the revision information from the scm repository,
proceeding with revision of unknown source :
Exception while executing SCM command.
[INFO] Storing buildNumber: unknown source at timestamp: 1466883469071
[WARNING] Cannot get the branch information from the git repository:
Error while executing command.
{code}
For a source build, the timestamp is the only useful addition, and the data
captured looks like this (as seen in nifi.properties):
{code}
nifi.build.number=unknown source
nifi.build.branch=UNKNOWN_BRANCH
nifi.build.timestamp=2016-06-25T20:16:40Z
{code}
I admit it's not elegant, but it does not break. Testing is complicated by the
requirement for Git to support Bower in nifi-web-ui (see NIFI-1846).
> Enhanced About Box Version Information
> --
>
> Key: NIFI-2115
> URL: https://issues.apache.org/jira/browse/NIFI-2115
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core UI, Tools and Build
>Affects Versions: 1.0.0
>Reporter: James Wing
>Assignee: James Wing
>Priority: Minor
>
> The UI's About dialog and underlying API provide the version of NiFi, like
> "0.7.0-SNAPSHOT". For many bug reports and troubleshooting requests, this is
> not very precise, especially around rapidly changing code or
> platform-dependent behavior. It would help if NiFi captured and displayed
> additional information:
> * NiFi build commit hash
> * NiFi build branch
> * NiFi build date/time
> * Java Version
> * Java Vendor (Oracle, OpenJDK)
> * OS
> Then a simple copy/paste from the about box would provide very specific
> information about a particular NiFi installation and which code it was
> derived from.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-2115) Enhanced About Box Version Information
[ https://issues.apache.org/jira/browse/NIFI-2115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Wing updated NIFI-2115: - Fix Version/s: (was: 1.0.0) > Enhanced About Box Version Information > -- > > Key: NIFI-2115 > URL: https://issues.apache.org/jira/browse/NIFI-2115 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI, Tools and Build >Affects Versions: 1.0.0 >Reporter: James Wing >Assignee: James Wing >Priority: Minor > > The UI's About dialog and underlying API provide the version of NiFi, like > "0.7.0-SNAPSHOT". For many bug reports and troubleshooting requests, this is > not very precise, especially around rapidly changing code or > platform-dependent behavior. It would help if NiFi captured and displayed > additional information: > * NiFi build commit hash > * NiFi build branch > * NiFi build date/time > * Java Version > * Java Vendor (Oracle, OpenJDK) > * OS > Then a simple copy/paste from the about box would provide very specific > information about a particular NiFi installation and which code it was > derived from. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2115) Enhanced About Box Version Information
James Wing created NIFI-2115: Summary: Enhanced About Box Version Information Key: NIFI-2115 URL: https://issues.apache.org/jira/browse/NIFI-2115 Project: Apache NiFi Issue Type: Improvement Components: Core UI, Tools and Build Affects Versions: 1.0.0 Reporter: James Wing Assignee: James Wing Priority: Minor Fix For: 1.0.0 The UI's About dialog and underlying API provide the version of NiFi, like "0.7.0-SNAPSHOT". For many bug reports and troubleshooting requests, this is not very precise, especially around rapidly changing code or platform-dependent behavior. It would help if NiFi captured and displayed additional information: * NiFi build commit hash * NiFi build branch * NiFi build date/time * Java Version * Java Vendor (Oracle, OpenJDK) * OS Then a simple copy/paste from the about box would provide very specific information about a particular NiFi installation and which code it was derived from. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2062) Upgrade AWS SDK
[ https://issues.apache.org/jira/browse/NIFI-2062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Wing updated NIFI-2062: - Fix Version/s: 1.0.0 > Upgrade AWS SDK > --- > > Key: NIFI-2062 > URL: https://issues.apache.org/jira/browse/NIFI-2062 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.0.0 >Reporter: James Wing >Assignee: James Wing > Fix For: 1.0.0 > > > I propose to update NiFi's current AWS SDK from version 1.10.32 (Nov 3, 2015) > to version 1.11.8 (Jun 16, 2016). Known impact: > * *API Signature Version* - I expect the largest impact will be to S3 > processors, where the default API signature version will change from v2 to v4 > for a wider number of operations. See release notes for > [1.11.0|https://aws.amazon.com/releasenotes/Java/9979983567247718]. > * *Region Endpoint Info* - The AWS SDK contains baked-in configuration data > about AWS regions and the services they support, which is periodically > updated. > * *Service Updates* - Potentially relevant updates to future NiFi support for > AWS IoT and other newer services. > I am not aware of any shocking changes for the services NiFi currently > supports (S3, SNS, SQS, DynamoDB, Lambda, Kinesis Firehose). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (NIFI-2063) Install Script Relative Path Mismatch from Init Dir
[
https://issues.apache.org/jira/browse/NIFI-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Wing closed NIFI-2063.
Changes have been pushed to 0.x and master.
> Install Script Relative Path Mismatch from Init Dir
> ---
>
> Key: NIFI-2063
> URL: https://issues.apache.org/jira/browse/NIFI-2063
> Project: Apache NiFi
> Issue Type: Bug
> Components: Configuration
>Affects Versions: 1.0.0, 0.7.0
> Environment: Linux/Unix with SysVInit
>Reporter: James Wing
>Assignee: Yolanda M. Davis
> Fix For: 1.0.0, 0.7.0
>
>
> Ryan Hendrickson noticed that installing and running NiFi as a service in
> 0.7.0-SNAPSHOT is not working right. Running commands like the following:
> {code}
> $ sudo /opt/nifi/nifi-0.7.0-SNAPSHOT/bin/nifi.sh install
> Service nifi installed
> $ sudo service nifi start
> /etc/init.d/nifi: line 28: /etc/init.d/nifi-env.sh: No such file or directory
> {code}
> Results in errors loading nifi-env.sh
> *Used-To-Be*
> As I understand it, {{nifi.sh install}} used to:
> 1. copy bin/nifi.sh to /etc/init.d/nifi
> 2. Run sed to change NIFI_HOME initializations to the path to nifi.sh, where
> install was invoked
> 3. When {{service nifi start}} was run, the NIFI_HOME path was hard-coded in
> the service script file.
> *As-Is*
> 1. nifi.sh now gets NIFI_HOME from bin/nifi-env.sh, which is stored alongside
> nifi.sh.
> 2. install() still calls sed to update NIFI_HOME=, but the initialization is
> now in nifi-env.sh, so nothing is replaced.
> 3. When {{service nifi start}} is called, relative path to
> {{SCRIPT_DIR/nifi-env.sh}} resolves to {{/etc/init.d/nifi-env.sh}}, which
> does not exist.
> *To-Be*
> A possible solution might be to have sed update SCRIPT_DIR to be hard-coded,
> and change to that directory to run nifi-env.sh.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-2063) Install Script Relative Path Mismatch from Init Dir
[
https://issues.apache.org/jira/browse/NIFI-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15342590#comment-15342590
]
James Wing commented on NIFI-2063:
--
I also have not experienced problems using NiFi 0.6.1, a run.as user configured
in boostrap.conf, and rebooting.
> Install Script Relative Path Mismatch from Init Dir
> ---
>
> Key: NIFI-2063
> URL: https://issues.apache.org/jira/browse/NIFI-2063
> Project: Apache NiFi
> Issue Type: Bug
> Components: Configuration
>Affects Versions: 1.0.0, 0.7.0
> Environment: Linux/Unix with SysVInit
>Reporter: James Wing
>Assignee: Yolanda M. Davis
> Fix For: 0.7.0
>
>
> Ryan Hendrickson noticed that installing and running NiFi as a service in
> 0.7.0-SNAPSHOT is not working right. Running commands like the following:
> {code}
> $ sudo /opt/nifi/nifi-0.7.0-SNAPSHOT/bin/nifi.sh install
> Service nifi installed
> $ sudo service nifi start
> /etc/init.d/nifi: line 28: /etc/init.d/nifi-env.sh: No such file or directory
> {code}
> Results in errors loading nifi-env.sh
> *Used-To-Be*
> As I understand it, {{nifi.sh install}} used to:
> 1. copy bin/nifi.sh to /etc/init.d/nifi
> 2. Run sed to change NIFI_HOME initializations to the path to nifi.sh, where
> install was invoked
> 3. When {{service nifi start}} was run, the NIFI_HOME path was hard-coded in
> the service script file.
> *As-Is*
> 1. nifi.sh now gets NIFI_HOME from bin/nifi-env.sh, which is stored alongside
> nifi.sh.
> 2. install() still calls sed to update NIFI_HOME=, but the initialization is
> now in nifi-env.sh, so nothing is replaced.
> 3. When {{service nifi start}} is called, relative path to
> {{SCRIPT_DIR/nifi-env.sh}} resolves to {{/etc/init.d/nifi-env.sh}}, which
> does not exist.
> *To-Be*
> A possible solution might be to have sed update SCRIPT_DIR to be hard-coded,
> and change to that directory to run nifi-env.sh.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (NIFI-2063) Install Script Relative Path Mismatch from Init Dir
James Wing created NIFI-2063:
Summary: Install Script Relative Path Mismatch from Init Dir
Key: NIFI-2063
URL: https://issues.apache.org/jira/browse/NIFI-2063
Project: Apache NiFi
Issue Type: Bug
Components: Configuration
Affects Versions: 1.0.0, 0.7.0
Environment: Linux/Unix with SysVInit
Reporter: James Wing
Fix For: 0.7.0
Ryan Hendrickson noticed that installing and running NiFi as a service in
0.7.0-SNAPSHOT is not working right. Running commands like the following:
{code}
$ sudo /opt/nifi/nifi-0.7.0-SNAPSHOT/bin/nifi.sh install
Service nifi installed
$ sudo service nifi start
/etc/init.d/nifi: line 28: /etc/init.d/nifi-env.sh: No such file or directory
{code}
Results in errors loading nifi-env.sh
*Used-To-Be*
As I understand it, {{nifi.sh install}} used to:
1. copy bin/nifi.sh to /etc/init.d/nifi
2. Run sed to change NIFI_HOME initializations to the path to nifi.sh, where
install was invoked
3. When {{service nifi start}} was run, the NIFI_HOME path was hard-coded in
the service script file.
*As-Is*
1. nifi.sh now gets NIFI_HOME from bin/nifi-env.sh, which is stored alongside
nifi.sh.
2. install() still calls sed to update NIFI_HOME=, but the initialization is
now in nifi-env.sh, so nothing is replaced.
3. When {{service nifi start}} is called, relative path to
{{SCRIPT_DIR/nifi-env.sh}} resolves to {{/etc/init.d/nifi-env.sh}}, which does
not exist.
*To-Be*
A possible solution might be to have sed update SCRIPT_DIR to be hard-coded,
and change to that directory to run nifi-env.sh.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (NIFI-2062) Upgrade AWS SDK
James Wing created NIFI-2062: Summary: Upgrade AWS SDK Key: NIFI-2062 URL: https://issues.apache.org/jira/browse/NIFI-2062 Project: Apache NiFi Issue Type: Improvement Components: Extensions Affects Versions: 1.0.0 Reporter: James Wing Assignee: James Wing I propose to update NiFi's current AWS SDK from version 1.10.32 (Nov 3, 2015) to version 1.11.8 (Jun 16, 2016). Known impact: * *API Signature Version* - I expect the largest impact will be to S3 processors, where the default API signature version will change from v2 to v4 for a wider number of operations. See release notes for [1.11.0|https://aws.amazon.com/releasenotes/Java/9979983567247718]. * *Region Endpoint Info* - The AWS SDK contains baked-in configuration data about AWS regions and the services they support, which is periodically updated. * *Service Updates* - Potentially relevant updates to future NiFi support for AWS IoT and other newer services. I am not aware of any shocking changes for the services NiFi currently supports (S3, SNS, SQS, DynamoDB, Lambda, Kinesis Firehose). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1769) Add support for SSE-KMS and S3 Signature Version 4 Authentication AWS
[ https://issues.apache.org/jira/browse/NIFI-1769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15339873#comment-15339873 ] James Wing commented on NIFI-1769: -- Mostly just time for review and testing. I am not that familiar with S3 Server-Side Encryption using customer keys. > Add support for SSE-KMS and S3 Signature Version 4 Authentication AWS > - > > Key: NIFI-1769 > URL: https://issues.apache.org/jira/browse/NIFI-1769 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 0.5.1 >Reporter: Michiel Moonen >Priority: Minor > Labels: newbie, patch, security > Fix For: 1.0.0 > > > Currently there is no support for SSE-KMS S3 Signature Version 4 > Authentication. This is necessary for enhanced security features -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1769) Add support for SSE-KMS and S3 Signature Version 4 Authentication AWS
[ https://issues.apache.org/jira/browse/NIFI-1769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15339689#comment-15339689 ] James Wing commented on NIFI-1769: -- I think this ticket is a better fit for 1.0 than 0.7.0, unless you object. > Add support for SSE-KMS and S3 Signature Version 4 Authentication AWS > - > > Key: NIFI-1769 > URL: https://issues.apache.org/jira/browse/NIFI-1769 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 0.5.1 >Reporter: Michiel Moonen >Priority: Minor > Labels: newbie, patch, security > Fix For: 1.0.0 > > > Currently there is no support for SSE-KMS S3 Signature Version 4 > Authentication. This is necessary for enhanced security features -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-1769) Add support for SSE-KMS and S3 Signature Version 4 Authentication AWS
[ https://issues.apache.org/jira/browse/NIFI-1769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Wing updated NIFI-1769: - Fix Version/s: 1.0.0 > Add support for SSE-KMS and S3 Signature Version 4 Authentication AWS > - > > Key: NIFI-1769 > URL: https://issues.apache.org/jira/browse/NIFI-1769 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 0.5.1 >Reporter: Michiel Moonen >Priority: Minor > Labels: newbie, patch, security > Fix For: 1.0.0 > > > Currently there is no support for SSE-KMS S3 Signature Version 4 > Authentication. This is necessary for enhanced security features -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1769) Add support for SSE-KMS and S3 Signature Version 4 Authentication AWS
[ https://issues.apache.org/jira/browse/NIFI-1769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336582#comment-15336582 ] James Wing commented on NIFI-1769: -- [~joewitt], I would be happy to try. > Add support for SSE-KMS and S3 Signature Version 4 Authentication AWS > - > > Key: NIFI-1769 > URL: https://issues.apache.org/jira/browse/NIFI-1769 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 0.5.1 >Reporter: Michiel Moonen >Priority: Minor > Labels: newbie, patch, security > > Currently there is no support for SSE-KMS S3 Signature Version 4 > Authentication. This is necessary for enhanced security features -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (NIFI-1933) SplitJson enhancements for flowfile names and additional options
[ https://issues.apache.org/jira/browse/NIFI-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15330035#comment-15330035 ] James Wing edited comment on NIFI-1933 at 6/14/16 6:04 PM: --- I understand, [~Daniel Cave], I'll remove the 0.7.0 tag for now. Nothing in this ticket needs to block the 0.7.0 release. Nothing has been merged, and the features are equally applicable to the master branch. was (Author: jameswing): Nothing in this ticket needs to block the 0.7.0 release. Nothing has been merged, and the features are equally applicable to the master branch. If there is time, we can review and capture some functionality. > SplitJson enhancements for flowfile names and additional options > > > Key: NIFI-1933 > URL: https://issues.apache.org/jira/browse/NIFI-1933 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.0.0, 0.7.0 >Reporter: Daniel Cave >Assignee: Alex Halldin >Priority: Minor > Fix For: 1.0.0 > > Attachments: > 0001-NIFI-1933-Updates-to-SplitJSON.java-now-can-split-on.patch > > > 1) SplitJson output flowfiles are replicating the parent filename across all > children. This results in overwriting of failure if trying to write the > files without later renaming in another processor (that may or may not result > in meaningful names). SplitJson will now rename children with an appended > index number to the parent: -.json > 2) SplitJson now has the option to include the split upon element as the new > root element of each split. This configurable and disabled by default to > conform with backwards compatibility. > 3) SplitJson now will correctly handle single element splits and no longer > simply fail due to the element not being an array. Also configurable and > disabled by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-1933) SplitJson enhancements for flowfile names and additional options
[ https://issues.apache.org/jira/browse/NIFI-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Wing updated NIFI-1933: - Fix Version/s: (was: 0.7.0) > SplitJson enhancements for flowfile names and additional options > > > Key: NIFI-1933 > URL: https://issues.apache.org/jira/browse/NIFI-1933 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.0.0, 0.7.0 >Reporter: Daniel Cave >Assignee: Alex Halldin >Priority: Minor > Fix For: 1.0.0 > > Attachments: > 0001-NIFI-1933-Updates-to-SplitJSON.java-now-can-split-on.patch > > > 1) SplitJson output flowfiles are replicating the parent filename across all > children. This results in overwriting of failure if trying to write the > files without later renaming in another processor (that may or may not result > in meaningful names). SplitJson will now rename children with an appended > index number to the parent: -.json > 2) SplitJson now has the option to include the split upon element as the new > root element of each split. This configurable and disabled by default to > conform with backwards compatibility. > 3) SplitJson now will correctly handle single element splits and no longer > simply fail due to the element not being an array. Also configurable and > disabled by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1933) SplitJson enhancements for flowfile names and additional options
[ https://issues.apache.org/jira/browse/NIFI-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15330035#comment-15330035 ] James Wing commented on NIFI-1933: -- Nothing in this ticket needs to block the 0.7.0 release. Nothing has been merged, and the features are equally applicable to the master branch. If there is time, we can review and capture some functionality. > SplitJson enhancements for flowfile names and additional options > > > Key: NIFI-1933 > URL: https://issues.apache.org/jira/browse/NIFI-1933 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.0.0, 0.7.0 >Reporter: Daniel Cave >Assignee: Alex Halldin >Priority: Minor > Fix For: 1.0.0, 0.7.0 > > Attachments: > 0001-NIFI-1933-Updates-to-SplitJSON.java-now-can-split-on.patch > > > 1) SplitJson output flowfiles are replicating the parent filename across all > children. This results in overwriting of failure if trying to write the > files without later renaming in another processor (that may or may not result > in meaningful names). SplitJson will now rename children with an appended > index number to the parent: -.json > 2) SplitJson now has the option to include the split upon element as the new > root element of each split. This configurable and disabled by default to > conform with backwards compatibility. > 3) SplitJson now will correctly handle single element splits and no longer > simply fail due to the element not being an array. Also configurable and > disabled by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1933) SplitJson enhancements for flowfile names and additional options
[ https://issues.apache.org/jira/browse/NIFI-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15328406#comment-15328406 ] James Wing commented on NIFI-1933: -- Reviewing for potential inclusion in 0.7.0. > SplitJson enhancements for flowfile names and additional options > > > Key: NIFI-1933 > URL: https://issues.apache.org/jira/browse/NIFI-1933 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.0.0, 0.7.0 >Reporter: Daniel Cave >Assignee: Alex Halldin >Priority: Minor > Fix For: 1.0.0, 0.7.0 > > Attachments: > 0001-NIFI-1933-Updates-to-SplitJSON.java-now-can-split-on.patch > > > 1) SplitJson output flowfiles are replicating the parent filename across all > children. This results in overwriting of failure if trying to write the > files without later renaming in another processor (that may or may not result > in meaningful names). SplitJson will now rename children with an appended > index number to the parent: -.json > 2) SplitJson now has the option to include the split upon element as the new > root element of each split. This configurable and disabled by default to > conform with backwards compatibility. > 3) SplitJson now will correctly handle single element splits and no longer > simply fail due to the element not being an array. Also configurable and > disabled by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (NIFI-1887) Update Admin Guide for Processor Scheduling Timeout
[
https://issues.apache.org/jira/browse/NIFI-1887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Wing resolved NIFI-1887.
--
Resolution: Fixed
> Update Admin Guide for Processor Scheduling Timeout
> ---
>
> Key: NIFI-1887
> URL: https://issues.apache.org/jira/browse/NIFI-1887
> Project: Apache NiFi
> Issue Type: Bug
> Components: Documentation & Website
>Affects Versions: 0.6.1
>Reporter: James Wing
>Assignee: James Wing
>Priority: Minor
> Labels: documentation
> Fix For: 1.0.0, 0.7.0
>
>
> In a [recent StackOverflow
> post|http://stackoverflow.com/questions/37222737/apache-nifi-timeout-while-waiting-for-onscheduled],
> it was noted that the Admin Guide description of the
> {{nifi.processor.scheduling.timeout}} property is out of date with the 60
> second timeout now specified in the source code from [commit
> c7df94e|https://github.com/apache/nifi/commit/c7df94e00f5ac456b2ca094841ffee9325c2ea53#diff-bf3824cc09a89e6ea5f0f4aec0134c3fL1389].
> The documentation should be updated to match the current behavior.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-1887) Update Admin Guide for Processor Scheduling Timeout
[
https://issues.apache.org/jira/browse/NIFI-1887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Wing updated NIFI-1887:
-
Fix Version/s: 1.0.0
> Update Admin Guide for Processor Scheduling Timeout
> ---
>
> Key: NIFI-1887
> URL: https://issues.apache.org/jira/browse/NIFI-1887
> Project: Apache NiFi
> Issue Type: Bug
> Components: Documentation & Website
>Affects Versions: 0.6.1
>Reporter: James Wing
>Assignee: James Wing
>Priority: Minor
> Labels: documentation
> Fix For: 1.0.0, 0.7.0
>
>
> In a [recent StackOverflow
> post|http://stackoverflow.com/questions/37222737/apache-nifi-timeout-while-waiting-for-onscheduled],
> it was noted that the Admin Guide description of the
> {{nifi.processor.scheduling.timeout}} property is out of date with the 60
> second timeout now specified in the source code from [commit
> c7df94e|https://github.com/apache/nifi/commit/c7df94e00f5ac456b2ca094841ffee9325c2ea53#diff-bf3824cc09a89e6ea5f0f4aec0134c3fL1389].
> The documentation should be updated to match the current behavior.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (NIFI-1887) Update Admin Guide for Processor Scheduling Timeout
James Wing created NIFI-1887:
Summary: Update Admin Guide for Processor Scheduling Timeout
Key: NIFI-1887
URL: https://issues.apache.org/jira/browse/NIFI-1887
Project: Apache NiFi
Issue Type: Bug
Components: Documentation & Website
Affects Versions: 0.6.1
Reporter: James Wing
Assignee: James Wing
Priority: Minor
Fix For: 0.7.0
In a [recent StackOverflow
post|http://stackoverflow.com/questions/37222737/apache-nifi-timeout-while-waiting-for-onscheduled],
it was noted that the Admin Guide description of the
{{nifi.processor.scheduling.timeout}} property is out of date with the 60
second timeout now specified in the source code from [commit
c7df94e|https://github.com/apache/nifi/commit/c7df94e00f5ac456b2ca094841ffee9325c2ea53#diff-bf3824cc09a89e6ea5f0f4aec0134c3fL1389].
The documentation should be updated to match the current behavior.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-1769) Add support for S3 Signature Version 4 Authentication AWS
[ https://issues.apache.org/jira/browse/NIFI-1769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15246563#comment-15246563 ] James Wing commented on NIFI-1769: -- Can you elaborate a bit on which enhanced security features or scenarios were you hoping to use? NiFi is using the AWS SDK for Java, which provides many of the request signing and in-transit security features under the hood, but there may be settings not exposed to easy configuration in the processor properties. > Add support for S3 Signature Version 4 Authentication AWS > - > > Key: NIFI-1769 > URL: https://issues.apache.org/jira/browse/NIFI-1769 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 0.5.1 >Reporter: Michiel Moonen >Priority: Minor > Labels: newbie, patch, security > > Currently there is no support for S3 Signature Version 4 Authentication. This > is necessary for enhanced security features -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1752) EvaluateXPath invalid when starting processor
[ https://issues.apache.org/jira/browse/NIFI-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15240338#comment-15240338 ] James Wing commented on NIFI-1752: -- Thanks on both counts. > EvaluateXPath invalid when starting processor > - > > Key: NIFI-1752 > URL: https://issues.apache.org/jira/browse/NIFI-1752 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 0.6.0 >Reporter: Joseph Witt > > Guillaume Pool: > Hi, > > If anyone could help I would appreciate it: > > After upgrading to 0.6.0 all my EvaluateXMLPath processors report as invalid > with nothing specific reported as the error. > > I use flowfile attribute as the destination, xpath query similar to > /message/address > > Return type auto-detect / string > > Error: > > 2016-04-11 17:37:12,465 INFO [NiFi Web Server-64629] > org.apache.nifi.web.filter.RequestLogger Attempting request for (anonymous) > PUT > http://sv-hdpdf1.hdp.supergrp.net:8443/nifi-api/controller/process-groups/d02a0bbc-c142-37ba-adaa-ee191b98790d/processors/e49acf40-76d1-3b6e-a67d-dc40153605ad > (source ip: 172.27.0.97) > 2016-04-11 17:37:12,474 INFO [NiFi Web Server-64629] > o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: > EvaluateXPath[id=e49acf40-76d1-3b6e-a67d-dc40153605ad] is not in a valid > state. Returning Conflict response. > > Reports this even if you drop a new processor in. > James Wing: > I am able to see the same EvaluateXPath issue Guillame Pool reported on NiFi > 0.6.0. I created a template gist at > https://gist.github.com/jvwing/7e7948d8eb5ad3643f38966b4ba3ce2e that > illustrates this for me. You don't even need to run the whole thing, just > try to start the EvaluateXPath processor. I see a pop-up error trying to > start it "EvaluateXPath[id=b481c570-686a-4b6e-859c-9ebe27398fe1] is not in a > valid state". > Trying to troubleshoot this, I attached a debugger and found an > IllegalStateException thrown by StandardProcessorNode.verifyCanStart() on > line 1141. The error text was: > 'address' validated against '/message/address' is invalid because Unable to > initialize XPath engine due to > javax.xml.xpath.XPathFactoryConfigurationException: No XPathFactory > implementation found for the object model: http://saxon.sf.net/jaxp/xpath/om > I haven't figured out what change is the root cause, but the trail seems to > lead into validating the dynamic XPath attribute. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1752) EvaluateXPath invalid when starting processor
[ https://issues.apache.org/jira/browse/NIFI-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15240137#comment-15240137 ] James Wing commented on NIFI-1752: -- I don't believe I have the power to close this, but I recommend "resolved" given that the reported symptoms and focus are nominally different from NIFI-1697, even if we are coasting on their fix. > EvaluateXPath invalid when starting processor > - > > Key: NIFI-1752 > URL: https://issues.apache.org/jira/browse/NIFI-1752 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 0.6.0 >Reporter: Joseph Witt > Fix For: 0.7.0 > > > Guillaume Pool: > Hi, > > If anyone could help I would appreciate it: > > After upgrading to 0.6.0 all my EvaluateXMLPath processors report as invalid > with nothing specific reported as the error. > > I use flowfile attribute as the destination, xpath query similar to > /message/address > > Return type auto-detect / string > > Error: > > 2016-04-11 17:37:12,465 INFO [NiFi Web Server-64629] > org.apache.nifi.web.filter.RequestLogger Attempting request for (anonymous) > PUT > http://sv-hdpdf1.hdp.supergrp.net:8443/nifi-api/controller/process-groups/d02a0bbc-c142-37ba-adaa-ee191b98790d/processors/e49acf40-76d1-3b6e-a67d-dc40153605ad > (source ip: 172.27.0.97) > 2016-04-11 17:37:12,474 INFO [NiFi Web Server-64629] > o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: > EvaluateXPath[id=e49acf40-76d1-3b6e-a67d-dc40153605ad] is not in a valid > state. Returning Conflict response. > > Reports this even if you drop a new processor in. > James Wing: > I am able to see the same EvaluateXPath issue Guillame Pool reported on NiFi > 0.6.0. I created a template gist at > https://gist.github.com/jvwing/7e7948d8eb5ad3643f38966b4ba3ce2e that > illustrates this for me. You don't even need to run the whole thing, just > try to start the EvaluateXPath processor. I see a pop-up error trying to > start it "EvaluateXPath[id=b481c570-686a-4b6e-859c-9ebe27398fe1] is not in a > valid state". > Trying to troubleshoot this, I attached a debugger and found an > IllegalStateException thrown by StandardProcessorNode.verifyCanStart() on > line 1141. The error text was: > 'address' validated against '/message/address' is invalid because Unable to > initialize XPath engine due to > javax.xml.xpath.XPathFactoryConfigurationException: No XPathFactory > implementation found for the object model: http://saxon.sf.net/jaxp/xpath/om > I haven't figured out what change is the root cause, but the trail seems to > lead into validating the dynamic XPath attribute. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1752) EvaluateXPath invalid when starting processor
[ https://issues.apache.org/jira/browse/NIFI-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15239808#comment-15239808 ] James Wing commented on NIFI-1752: -- This issue was fixed by code committed for NIFI-1697 (65b5c1a). > EvaluateXPath invalid when starting processor > - > > Key: NIFI-1752 > URL: https://issues.apache.org/jira/browse/NIFI-1752 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 0.6.0 >Reporter: Joseph Witt > Fix For: 0.7.0 > > > Guillaume Pool: > Hi, > > If anyone could help I would appreciate it: > > After upgrading to 0.6.0 all my EvaluateXMLPath processors report as invalid > with nothing specific reported as the error. > > I use flowfile attribute as the destination, xpath query similar to > /message/address > > Return type auto-detect / string > > Error: > > 2016-04-11 17:37:12,465 INFO [NiFi Web Server-64629] > org.apache.nifi.web.filter.RequestLogger Attempting request for (anonymous) > PUT > http://sv-hdpdf1.hdp.supergrp.net:8443/nifi-api/controller/process-groups/d02a0bbc-c142-37ba-adaa-ee191b98790d/processors/e49acf40-76d1-3b6e-a67d-dc40153605ad > (source ip: 172.27.0.97) > 2016-04-11 17:37:12,474 INFO [NiFi Web Server-64629] > o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: > EvaluateXPath[id=e49acf40-76d1-3b6e-a67d-dc40153605ad] is not in a valid > state. Returning Conflict response. > > Reports this even if you drop a new processor in. > James Wing: > I am able to see the same EvaluateXPath issue Guillame Pool reported on NiFi > 0.6.0. I created a template gist at > https://gist.github.com/jvwing/7e7948d8eb5ad3643f38966b4ba3ce2e that > illustrates this for me. You don't even need to run the whole thing, just > try to start the EvaluateXPath processor. I see a pop-up error trying to > start it "EvaluateXPath[id=b481c570-686a-4b6e-859c-9ebe27398fe1] is not in a > valid state". > Trying to troubleshoot this, I attached a debugger and found an > IllegalStateException thrown by StandardProcessorNode.verifyCanStart() on > line 1141. The error text was: > 'address' validated against '/message/address' is invalid because Unable to > initialize XPath engine due to > javax.xml.xpath.XPathFactoryConfigurationException: No XPathFactory > implementation found for the object model: http://saxon.sf.net/jaxp/xpath/om > I haven't figured out what change is the root cause, but the trail seems to > lead into validating the dynamic XPath attribute. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1752) EvaluateXPath invalid when starting processor
[ https://issues.apache.org/jira/browse/NIFI-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15239753#comment-15239753 ] James Wing commented on NIFI-1752: -- We may not need a fix here. It looks like there has already been a post-0.6.0 fix in [NIFI-1697 AmbariReportingTask not working when using PutHDFS|https://issues.apache.org/jira/browse/NIFI-1697]. That ticket includes a change to explicitly use the NarClassLoader via NarClosable for all processor validation ([commit 65b5c1a for StandardProcessorNode.isValid()|https://github.com/apache/nifi/commit/65b5c1a5f1899fd58d306b81983a015e79f9355f#diff-bf3824cc09a89e6ea5f0f4aec0134c3f]). I believe this also fixes the problem for XPathValidator. Using a more centralized fix avoids a situation where individual property validators have to goof around with their ClassLoader. Testing the issue in the 0.x branch with 65b5c1a shows the use of the NarThreadContextClassLoader to get the XPathFactory, and I have not reproduced the problem in the 0.x branch. > EvaluateXPath invalid when starting processor > - > > Key: NIFI-1752 > URL: https://issues.apache.org/jira/browse/NIFI-1752 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 0.6.0 >Reporter: Joseph Witt > Fix For: 0.7.0 > > > Guillaume Pool: > Hi, > > If anyone could help I would appreciate it: > > After upgrading to 0.6.0 all my EvaluateXMLPath processors report as invalid > with nothing specific reported as the error. > > I use flowfile attribute as the destination, xpath query similar to > /message/address > > Return type auto-detect / string > > Error: > > 2016-04-11 17:37:12,465 INFO [NiFi Web Server-64629] > org.apache.nifi.web.filter.RequestLogger Attempting request for (anonymous) > PUT > http://sv-hdpdf1.hdp.supergrp.net:8443/nifi-api/controller/process-groups/d02a0bbc-c142-37ba-adaa-ee191b98790d/processors/e49acf40-76d1-3b6e-a67d-dc40153605ad > (source ip: 172.27.0.97) > 2016-04-11 17:37:12,474 INFO [NiFi Web Server-64629] > o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: > EvaluateXPath[id=e49acf40-76d1-3b6e-a67d-dc40153605ad] is not in a valid > state. Returning Conflict response. > > Reports this even if you drop a new processor in. > James Wing: > I am able to see the same EvaluateXPath issue Guillame Pool reported on NiFi > 0.6.0. I created a template gist at > https://gist.github.com/jvwing/7e7948d8eb5ad3643f38966b4ba3ce2e that > illustrates this for me. You don't even need to run the whole thing, just > try to start the EvaluateXPath processor. I see a pop-up error trying to > start it "EvaluateXPath[id=b481c570-686a-4b6e-859c-9ebe27398fe1] is not in a > valid state". > Trying to troubleshoot this, I attached a debugger and found an > IllegalStateException thrown by StandardProcessorNode.verifyCanStart() on > line 1141. The error text was: > 'address' validated against '/message/address' is invalid because Unable to > initialize XPath engine due to > javax.xml.xpath.XPathFactoryConfigurationException: No XPathFactory > implementation found for the object model: http://saxon.sf.net/jaxp/xpath/om > I haven't figured out what change is the root cause, but the trail seems to > lead into validating the dynamic XPath attribute. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1752) EvaluateXPath invalid when starting processor
[
https://issues.apache.org/jira/browse/NIFI-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15238625#comment-15238625
]
James Wing commented on NIFI-1752:
--
I believe the problem is not concurrency, but thread-specific class loaders
used by XPathFactory. Calls to XPathValidator.validate() on thread
"Timer-Driven Process Thread-?" use a NarThreadContextClassLoader and succeed.
Calls on thread "NiFi Web Server-?" used class loader WebAppClassLoader and
failed loading the same class net.sf.saxon.xpath.XPathFactoryImpl.
I'm not sure why this error seems intermittent and selective to particular
installations. Maybe classpath? On one of my instances I was able to resolve
the issue by dropping a copy of Saxon-HE-9.6.0-5.jar into the {{lib}} folder
and restarting NiFi. This might be a workaround, but I certainly haven't
tested it thoroughly. [~dionosys], would you be willing to try this?
I also do not know if this is new in 0.6.0 or how it might have been
introduced. I'm working on a fix that explicitly uses the XPathValidator's
instance ClassLoader in the call to XPathFactory.newInstance().
> EvaluateXPath invalid when starting processor
> -
>
> Key: NIFI-1752
> URL: https://issues.apache.org/jira/browse/NIFI-1752
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 0.6.0
>Reporter: Joseph Witt
>
> Guillaume Pool:
> Hi,
>
> If anyone could help I would appreciate it:
>
> After upgrading to 0.6.0 all my EvaluateXMLPath processors report as invalid
> with nothing specific reported as the error.
>
> I use flowfile attribute as the destination, xpath query similar to
> /message/address
>
> Return type auto-detect / string
>
> Error:
>
> 2016-04-11 17:37:12,465 INFO [NiFi Web Server-64629]
> org.apache.nifi.web.filter.RequestLogger Attempting request for (anonymous)
> PUT
> http://sv-hdpdf1.hdp.supergrp.net:8443/nifi-api/controller/process-groups/d02a0bbc-c142-37ba-adaa-ee191b98790d/processors/e49acf40-76d1-3b6e-a67d-dc40153605ad
> (source ip: 172.27.0.97)
> 2016-04-11 17:37:12,474 INFO [NiFi Web Server-64629]
> o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException:
> EvaluateXPath[id=e49acf40-76d1-3b6e-a67d-dc40153605ad] is not in a valid
> state. Returning Conflict response.
>
> Reports this even if you drop a new processor in.
> James Wing:
> I am able to see the same EvaluateXPath issue Guillame Pool reported on NiFi
> 0.6.0. I created a template gist at
> https://gist.github.com/jvwing/7e7948d8eb5ad3643f38966b4ba3ce2e that
> illustrates this for me. You don't even need to run the whole thing, just
> try to start the EvaluateXPath processor. I see a pop-up error trying to
> start it "EvaluateXPath[id=b481c570-686a-4b6e-859c-9ebe27398fe1] is not in a
> valid state".
> Trying to troubleshoot this, I attached a debugger and found an
> IllegalStateException thrown by StandardProcessorNode.verifyCanStart() on
> line 1141. The error text was:
> 'address' validated against '/message/address' is invalid because Unable to
> initialize XPath engine due to
> javax.xml.xpath.XPathFactoryConfigurationException: No XPathFactory
> implementation found for the object model: http://saxon.sf.net/jaxp/xpath/om
> I haven't figured out what change is the root cause, but the trail seems to
> lead into validating the dynamic XPath attribute.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-1752) EvaluateXPath invalid when starting processor
[ https://issues.apache.org/jira/browse/NIFI-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236007#comment-15236007 ] James Wing commented on NIFI-1752: -- Not reliably, although it has seemed to fix both environments for long enough to get my hopes up. Sometimes, after another restart, things go back to failing, so maybe the synchronization is irrelevant or simply moves the timing condition around. I'm going to add some logging and try to make sense of it. > EvaluateXPath invalid when starting processor > - > > Key: NIFI-1752 > URL: https://issues.apache.org/jira/browse/NIFI-1752 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 0.6.0 >Reporter: Joseph Witt > > Guillaume Pool: > Hi, > > If anyone could help I would appreciate it: > > After upgrading to 0.6.0 all my EvaluateXMLPath processors report as invalid > with nothing specific reported as the error. > > I use flowfile attribute as the destination, xpath query similar to > /message/address > > Return type auto-detect / string > > Error: > > 2016-04-11 17:37:12,465 INFO [NiFi Web Server-64629] > org.apache.nifi.web.filter.RequestLogger Attempting request for (anonymous) > PUT > http://sv-hdpdf1.hdp.supergrp.net:8443/nifi-api/controller/process-groups/d02a0bbc-c142-37ba-adaa-ee191b98790d/processors/e49acf40-76d1-3b6e-a67d-dc40153605ad > (source ip: 172.27.0.97) > 2016-04-11 17:37:12,474 INFO [NiFi Web Server-64629] > o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: > EvaluateXPath[id=e49acf40-76d1-3b6e-a67d-dc40153605ad] is not in a valid > state. Returning Conflict response. > > Reports this even if you drop a new processor in. > James Wing: > I am able to see the same EvaluateXPath issue Guillame Pool reported on NiFi > 0.6.0. I created a template gist at > https://gist.github.com/jvwing/7e7948d8eb5ad3643f38966b4ba3ce2e that > illustrates this for me. You don't even need to run the whole thing, just > try to start the EvaluateXPath processor. I see a pop-up error trying to > start it "EvaluateXPath[id=b481c570-686a-4b6e-859c-9ebe27398fe1] is not in a > valid state". > Trying to troubleshoot this, I attached a debugger and found an > IllegalStateException thrown by StandardProcessorNode.verifyCanStart() on > line 1141. The error text was: > 'address' validated against '/message/address' is invalid because Unable to > initialize XPath engine due to > javax.xml.xpath.XPathFactoryConfigurationException: No XPathFactory > implementation found for the object model: http://saxon.sf.net/jaxp/xpath/om > I haven't figured out what change is the root cause, but the trail seems to > lead into validating the dynamic XPath attribute. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1752) EvaluateXPath invalid when starting processor
[ https://issues.apache.org/jira/browse/NIFI-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15235883#comment-15235883 ] James Wing commented on NIFI-1752: -- I tried it in two environments, and experience the issue on both: Windows 10 java version "1.8.0_66" Java(TM) SE Runtime Environment (build 1.8.0_66-b18) Java HotSpot(TM) 64-Bit Server VM (build 25.66-b18, mixed mode) Amazon Linux (March 2016) openjdk version "1.8.0_77" OpenJDK Runtime Environment (build 1.8.0_77-b03) OpenJDK 64-Bit Server VM (build 25.77-b03, mixed mode) > EvaluateXPath invalid when starting processor > - > > Key: NIFI-1752 > URL: https://issues.apache.org/jira/browse/NIFI-1752 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 0.6.0 >Reporter: Joseph Witt > > Guillaume Pool: > Hi, > > If anyone could help I would appreciate it: > > After upgrading to 0.6.0 all my EvaluateXMLPath processors report as invalid > with nothing specific reported as the error. > > I use flowfile attribute as the destination, xpath query similar to > /message/address > > Return type auto-detect / string > > Error: > > 2016-04-11 17:37:12,465 INFO [NiFi Web Server-64629] > org.apache.nifi.web.filter.RequestLogger Attempting request for (anonymous) > PUT > http://sv-hdpdf1.hdp.supergrp.net:8443/nifi-api/controller/process-groups/d02a0bbc-c142-37ba-adaa-ee191b98790d/processors/e49acf40-76d1-3b6e-a67d-dc40153605ad > (source ip: 172.27.0.97) > 2016-04-11 17:37:12,474 INFO [NiFi Web Server-64629] > o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: > EvaluateXPath[id=e49acf40-76d1-3b6e-a67d-dc40153605ad] is not in a valid > state. Returning Conflict response. > > Reports this even if you drop a new processor in. > James Wing: > I am able to see the same EvaluateXPath issue Guillame Pool reported on NiFi > 0.6.0. I created a template gist at > https://gist.github.com/jvwing/7e7948d8eb5ad3643f38966b4ba3ce2e that > illustrates this for me. You don't even need to run the whole thing, just > try to start the EvaluateXPath processor. I see a pop-up error trying to > start it "EvaluateXPath[id=b481c570-686a-4b6e-859c-9ebe27398fe1] is not in a > valid state". > Trying to troubleshoot this, I attached a debugger and found an > IllegalStateException thrown by StandardProcessorNode.verifyCanStart() on > line 1141. The error text was: > 'address' validated against '/message/address' is invalid because Unable to > initialize XPath engine due to > javax.xml.xpath.XPathFactoryConfigurationException: No XPathFactory > implementation found for the object model: http://saxon.sf.net/jaxp/xpath/om > I haven't figured out what change is the root cause, but the trail seems to > lead into validating the dynamic XPath attribute. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1752) EvaluateXPath invalid when starting processor
[
https://issues.apache.org/jira/browse/NIFI-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15235704#comment-15235704
]
James Wing commented on NIFI-1752:
--
The underlying exception, "javax.xml.xpath.XPathFactoryConfigurationException:
No XPathFactory implementation found for the object model:
http://saxon.sf.net/jaxp/xpath/om"; is thrown by XPathValidator while executing
on EvaluateXPath.java:436:
{code}
XPathFactory factory =
XPathFactory.newInstance(NamespaceConstant.OBJECT_MODEL_SAXON);
{code}
>From watching in the debugger, *sometimes* the exception is thrown, but also
>*sometimes not* - and validation continues to compile the expression
>successfully and validate the property (seen by watching
>{{evaluator.compile(input);}} on line 441).
One possible explanation would be multi-threading. There are several
validation operations in progress at the time the exceptions occur. And the
[documentation for
XPathFactory|https://docs.oracle.com/javase/7/docs/api/javax/xml/xpath/XPathFactory.html]
states that it is not thread safe.
bq. The XPathFactory class is not thread-safe. In other words, it is the
application's responsibility to ensure that at most one thread is using a
XPathFactory object at any given moment. Implementations are encouraged to mark
methods as synchronized to protect themselves from broken clients.
I made an experimental change to synchronize the creation and use of
XPathFactory, and that seems to resolve the issue on my machine. But I'm not
sure how multi-threading would relate to changes introduced in 0.6.0. Proving
that multi-threading is or is not the actual problem and that a {{synchronize}}
block is the right fix is the next step.
> EvaluateXPath invalid when starting processor
> -
>
> Key: NIFI-1752
> URL: https://issues.apache.org/jira/browse/NIFI-1752
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 0.6.0
>Reporter: Joseph Witt
> Fix For: 1.0.0, 0.7.0, 0.6.1
>
>
> Guillaume Pool:
> Hi,
>
> If anyone could help I would appreciate it:
>
> After upgrading to 0.6.0 all my EvaluateXMLPath processors report as invalid
> with nothing specific reported as the error.
>
> I use flowfile attribute as the destination, xpath query similar to
> /message/address
>
> Return type auto-detect / string
>
> Error:
>
> 2016-04-11 17:37:12,465 INFO [NiFi Web Server-64629]
> org.apache.nifi.web.filter.RequestLogger Attempting request for (anonymous)
> PUT
> http://sv-hdpdf1.hdp.supergrp.net:8443/nifi-api/controller/process-groups/d02a0bbc-c142-37ba-adaa-ee191b98790d/processors/e49acf40-76d1-3b6e-a67d-dc40153605ad
> (source ip: 172.27.0.97)
> 2016-04-11 17:37:12,474 INFO [NiFi Web Server-64629]
> o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException:
> EvaluateXPath[id=e49acf40-76d1-3b6e-a67d-dc40153605ad] is not in a valid
> state. Returning Conflict response.
>
> Reports this even if you drop a new processor in.
> James Wing:
> I am able to see the same EvaluateXPath issue Guillame Pool reported on NiFi
> 0.6.0. I created a template gist at
> https://gist.github.com/jvwing/7e7948d8eb5ad3643f38966b4ba3ce2e that
> illustrates this for me. You don't even need to run the whole thing, just
> try to start the EvaluateXPath processor. I see a pop-up error trying to
> start it "EvaluateXPath[id=b481c570-686a-4b6e-859c-9ebe27398fe1] is not in a
> valid state".
> Trying to troubleshoot this, I attached a debugger and found an
> IllegalStateException thrown by StandardProcessorNode.verifyCanStart() on
> line 1141. The error text was:
> 'address' validated against '/message/address' is invalid because Unable to
> initialize XPath engine due to
> javax.xml.xpath.XPathFactoryConfigurationException: No XPathFactory
> implementation found for the object model: http://saxon.sf.net/jaxp/xpath/om
> I haven't figured out what change is the root cause, but the trail seems to
> lead into validating the dynamic XPath attribute.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-1711) Content Viewer Format Links Do Not Use Proxy Settings
[
https://issues.apache.org/jira/browse/NIFI-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Wing updated NIFI-1711:
-
Attachment: nifi-content-viewer-format-options.png
> Content Viewer Format Links Do Not Use Proxy Settings
> -
>
> Key: NIFI-1711
> URL: https://issues.apache.org/jira/browse/NIFI-1711
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
>Affects Versions: 0.6.0
>Reporter: James Wing
>Priority: Minor
> Attachments: nifi-content-viewer-format-options.png
>
>
> The content viewer format options do not use proxy settings, and generate bad
> links on a NiFi installation behind a proxy. Proxy settings were provided
> via the HTTP headers X-ProxyScheme, X-ProxyHost, and X-ProxyPort. The
> content viewer will initially display unformatted content fine, but then
> selecting different options such as 'formatted' results in a 404.
> !nifi-content-viewer-format-options.png!
> The URL seems to start life in
> [header.jsp|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-content-viewer/src/main/webapp/WEB-INF/jsp/header.jsp]:
> {code}
> <%=
> org.apache.nifi.util.EscapeUtils.escapeHtml(request.getRequestURL().toString())
> %>
> {code}
> Which was sent to the client as follows:
> {code}
> https://nifi/nifi-content-viewer/
> {code}
> On selecting 'formatted',the event handling Javascript concatenates this with
> the new view options to navigate to URLs like this:
> {code}
> https://nifi/nifi-content-viewer/?mode=Formatted&access_token=MjZvmd5...
> {code}
> I believe properly generating links that respect the proxy settings on the
> server would require logic similar to that of
> [ApplicationResource.generateResourceUri()|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/ApplicationResource.java#L97].
> It might also be possible to simply format the links on the client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (NIFI-1711) Content Viewer Format Links Do Not Use Proxy Settings
James Wing created NIFI-1711:
Summary: Content Viewer Format Links Do Not Use Proxy Settings
Key: NIFI-1711
URL: https://issues.apache.org/jira/browse/NIFI-1711
Project: Apache NiFi
Issue Type: Bug
Components: Core UI
Affects Versions: 0.6.0
Reporter: James Wing
Priority: Minor
The content viewer format options do not use proxy settings, and generate bad
links on a NiFi installation behind a proxy. Proxy settings were provided via
the HTTP headers X-ProxyScheme, X-ProxyHost, and X-ProxyPort. The content
viewer will initially display unformatted content fine, but then selecting
different options such as 'formatted' results in a 404.
!nifi-content-viewer-format-options.png!
The URL seems to start life in
[header.jsp|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-content-viewer/src/main/webapp/WEB-INF/jsp/header.jsp]:
{code}
<%=
org.apache.nifi.util.EscapeUtils.escapeHtml(request.getRequestURL().toString())
%>
{code}
Which was sent to the client as follows:
{code}
https://nifi/nifi-content-viewer/
{code}
On selecting 'formatted',the event handling Javascript concatenates this with
the new view options to navigate to URLs like this:
{code}
https://nifi/nifi-content-viewer/?mode=Formatted&access_token=MjZvmd5...
{code}
I believe properly generating links that respect the proxy settings on the
server would require logic similar to that of
[ApplicationResource.generateResourceUri()|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/ApplicationResource.java#L97].
It might also be possible to simply format the links on the client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-1590) No error message if nifi starts with port conflict
[
https://issues.apache.org/jira/browse/NIFI-1590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15190210#comment-15190210
]
James Wing commented on NIFI-1590:
--
Would you please check your *logs/nifi-app.log*? There may be additional
errors there. Although the initial command-line spew doesn't say failure, that
does not mean NiFi won't fail to start up.
NiFi will certainly complain and fail due to conflicting ports. I see this
when I accidentally run two NiFi processes, and an exception like the following
is logged in *logs/nifi-app.log*:
{code}
2016-03-10 23:56:24,141 WARN [main] o.e.j.util.component.AbstractLifeCycle
FAILED ServerConnector@212825d5{HTTP/1.1}{0.0.0.0:8080}:
java.net.BindException: Address already in use
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method) ~[na:1.7.0_95]
at sun.nio.ch.Net.bind(Net.java:463) ~[na:1.7.0_95]
at sun.nio.ch.Net.bind(Net.java:455) ~[na:1.7.0_95]
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
~[na:1.7.0_95]
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
~[na:1.7.0_95]
at
org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:321)
~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at
org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
~[jetty-server-9.2.11.v20150529.jar:9.2.11.v201505
29]
at
org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.Server.doStart(Server.java:366)
[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:663)
[nifi-jetty-0.6.0-SNAPSHOT.jar:0.6.0-SNAPSHOT]
at org.apache.nifi.NiFi.(NiFi.java:137)
[nifi-runtime-0.6.0-SNAPSHOT.jar:0.6.0-SNAPSHOT]
at org.apache.nifi.NiFi.main(NiFi.java:227)
[nifi-runtime-0.6.0-SNAPSHOT.jar:0.6.0-SNAPSHOT]
...
2016-03-10 23:56:24,143 INFO [Thread-1] org.apache.nifi.NiFi Initiating
shutdown of Jetty web server...
...
2016-03-10 23:56:24,819 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server
shutdown completed (nicely or otherwise).
{code}
> No error message if nifi starts with port conflict
> --
>
> Key: NIFI-1590
> URL: https://issues.apache.org/jira/browse/NIFI-1590
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Randy Gelhausen
>
> I have another webapp running on port 8080.
> On starting NiFi with default configs (web port 8080), I don't see any
> unusual behavior except that "nifi.sh run" immediately exits:
> HW11401:nifi-0.5.1.1.1.2.0-32 randy$ ./bin/nifi.sh run
> Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_65.jdk/Contents/Home/
> NiFi home: /Users/randy/nifi-0.5.1.1.1.2.0-32
> Bootstrap Config File: /Users/randy/nifi-0.5.1.1.1.2.0-32/conf/bootstrap.conf
> 2016-03-04 13:21:04,345 INFO [main] org.apache.nifi.bootstrap.Command
> Starting Apache NiFi...
> 2016-03-04 13:21:04,346 INFO [main] org.apache.nifi.bootstrap.Command Working
> Directory: /Users/randy/nifi-0.5.1.1.1.2.0-32
> 2016-03-04 13:21:04,346 INFO [main] org.apache.nifi.bootstrap.Command
> Command:
> /Library/Java/JavaVirtualMachines/jdk1.8.0_65.jdk/Contents/Home/bin/java
> -classpath
> /Users/randy/nifi-0.5.1.1.1.2.0-32/./conf:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/jcl-over-slf4j-1.7.12.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/jul-to-slf4j-1.7.12.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/log4j-over-slf4j-1.7.12.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/logback-classic-1.1.3.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/logback-core-1.1.3.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-api-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-documentation-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-nar-utils-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-properties-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-runtime-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/slf4j-api-1.7.12.jar
> -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx512m -Xms512m
> -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true
> -Djava.awt.headless=true -Djava.protocol.handler.pkgs=sun.net.www.protocol
> -Dnifi.properties.file.path=/Users/randy/nifi-0.5.1.1.1.2.0-32/./conf/ni
[jira] [Created] (NIFI-1614) Simple Username/Password Authentication
James Wing created NIFI-1614: Summary: Simple Username/Password Authentication Key: NIFI-1614 URL: https://issues.apache.org/jira/browse/NIFI-1614 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: James Wing Priority: Minor NiFi should include a simple option for username/password authentication backed by a local file store. NiFi's existing certificate and LDAP authentication schemes are very secure. However, the configuration and setup is complex, making them more suitable for long-lived corporate and government installations, but less accessible for casual or short-term use. Simple username/password authentication would help more users secure more NiFi installations beyond anonymous admin access. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1516) Provide AWS DynamoDB Delete/Put/Get processors
[
https://issues.apache.org/jira/browse/NIFI-1516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169604#comment-15169604
]
James Wing commented on NIFI-1516:
--
We need the key specification to Get or Delete an item, but I don't believe we
need it for a Put. Instead of (PutDynamoDB.java, line 123+):
{code}
try {
if ( rangeKeyValue == null ||
StringUtils.isBlank(rangeKeyValue.toString()) ) {
tableWriteItems.addItemToPut(new
Item().withKeyComponent(hashKeyName, hashKeyValue)
.withJSON(jsonDocument,
IOUtils.toString(baos.toByteArray(),charset)));
} else {
tableWriteItems.addItemToPut(new
Item().withKeyComponent(hashKeyName, hashKeyValue)
.withKeyComponent(rangeKeyName, rangeKeyValue)
.withJSON(jsonDocument,
IOUtils.toString(baos.toByteArray(),charset)));
}
{code}
We could do something like this:
{code}
try {
String jsonContent = IOUtils.toString(baos.toByteArray(),charset);
Item flowFileItem = Item.fromJSON(jsonContent);
tableWriteItems.addItemToPut(flowFileItem);
{code}
That would leave us with a problem mapping unprocessed items. From the
documentation, it appears possible to use {{Item.hashcode()}} for this, but I
have not verified that an unprocessed item will truly hash the same as the
original item.
Doing it this way might be left to another processor in another ticket. I
don't know that either is better or more intuitive for anyone other than myself
:).
> Provide AWS DynamoDB Delete/Put/Get processors
> --
>
> Key: NIFI-1516
> URL: https://issues.apache.org/jira/browse/NIFI-1516
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
>Affects Versions: 0.5.0
> Environment: all
>Reporter: Mans Singh
>Assignee: Mans Singh
>Priority: Minor
> Labels: aws, delete,, dynamodb,, get,, insert,, put,
> Fix For: 0.6.0
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> Implement AWS DynamoDB Get/Put/Delete processors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-1516) Provide AWS DynamoDB Delete/Put/Get processors
[ https://issues.apache.org/jira/browse/NIFI-1516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167537#comment-15167537 ] James Wing commented on NIFI-1516: -- [~mans2singh], I also have some suggestions about the test situation for DynamoDB processors. I published a [commit illustrating my suggestions|https://github.com/jvwing/nifi/commit/e0568acee307e98220d3a97f0e5caf9c5a56f8db] to show the concepts, I'm not sure this is exactly the right code. # Many of the existing tests look like they could be unit tests, especially tests for hash/range key validation where I don't believe any communication with AWS is required. In my commit, I was able to convert all of the integration tests from ITPutDynamoDBTest.java to unit tests in PutDynamoDBTest.java. # Many unit tests could be written using mocks. I've written a crude sample of what this might look like to intercept the outgoing request object model and mock the response. My sample changes AbstractDynamoDBProcessor to provide accessor methods for Amazon's BatchWriteItemApi interface rather than the DynamoDB class. I then wrote a mock implementation of the interfact for tests. Of course, this is not a substitute for integration tests actually hitting AWS, but it may be better than nothing. * PutDynamoDBTest.java, which is copy/paste/modified IPutDynamoDBTest.java to run as local unit tests. * MockBactchWriteItemApi.java, a mock implementation of BatchWriteItemApi * PutDynamoDBWithMockTest.java, a simple illustration of using mocking to run the processor, evaluating the request object model and faking a response. Please let me know if this is of any interest to you, I'd be happy to help build this out. > Provide AWS DynamoDB Delete/Put/Get processors > -- > > Key: NIFI-1516 > URL: https://issues.apache.org/jira/browse/NIFI-1516 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Affects Versions: 0.5.0 > Environment: all >Reporter: Mans Singh >Assignee: Mans Singh >Priority: Minor > Labels: aws, delete,, dynamodb,, get,, insert,, put, > Fix For: 0.6.0 > > Original Estimate: 72h > Remaining Estimate: 72h > > Implement AWS DynamoDB Get/Put/Delete processors. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1516) Provide AWS DynamoDB Delete/Put/Get processors
[
https://issues.apache.org/jira/browse/NIFI-1516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167504#comment-15167504
]
James Wing commented on NIFI-1516:
--
I took a look at the code in the pull request, and I have a couple comments.
First, I believe the JSON Document attribute should be required.
Second, your design requires that users provide a Hash Key, optional Range Key,
and the name of an attribute to hold the FlowFile content as a JSON document.
The resulting DynamoDB Items must always look something like this:
{code}
{
"sampleHashKey": "record1",
"sampleRangeKey": "123"
"sampleDocKey": { ...FlowFile content JSON... }
}
{code}
This works, but it seems like a restrictive data model to me. Did you consider
putting the entire FlowFile document to the DynamoDB item? That would more
flexibility, although it leaves it up to the user to make sure the Hash Key and
Range Key are correct in their FlowFile.
> Provide AWS DynamoDB Delete/Put/Get processors
> --
>
> Key: NIFI-1516
> URL: https://issues.apache.org/jira/browse/NIFI-1516
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
>Affects Versions: 0.5.0
> Environment: all
>Reporter: Mans Singh
>Assignee: Mans Singh
>Priority: Minor
> Labels: aws, delete,, dynamodb,, get,, insert,, put,
> Fix For: 0.6.0
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> Implement AWS DynamoDB Get/Put/Delete processors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-1509) Support AWS Lambda Processor
[ https://issues.apache.org/jira/browse/NIFI-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167438#comment-15167438 ] James Wing commented on NIFI-1509: -- I'm sorry I don't have any say in merging anything, but I'd vote for this one. > Support AWS Lambda Processor > > > Key: NIFI-1509 > URL: https://issues.apache.org/jira/browse/NIFI-1509 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Affects Versions: 0.5.0 > Environment: all >Reporter: Mans Singh >Assignee: Mans Singh >Priority: Minor > Labels: aws, lambda, streaming > Fix For: 0.6.0 > > Original Estimate: 48h > Remaining Estimate: 48h > > Support aws lambda put processor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1509) Support AWS Lambda Processor
[ https://issues.apache.org/jira/browse/NIFI-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15163386#comment-15163386 ] James Wing commented on NIFI-1509: -- And thanks for the capabilities doc update. This processor looks good to me, you write solid code. A Lambda processor opens up a wide, wide, range of routing and transformation scenarios with AWS. It's nice work. > Support AWS Lambda Processor > > > Key: NIFI-1509 > URL: https://issues.apache.org/jira/browse/NIFI-1509 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Affects Versions: 0.5.0 > Environment: all >Reporter: Mans Singh >Assignee: Mans Singh >Priority: Minor > Labels: aws, lambda, streaming > Fix For: 0.6.0 > > Original Estimate: 48h > Remaining Estimate: 48h > > Support aws lambda put processor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1509) Support AWS Lambda Processor
[ https://issues.apache.org/jira/browse/NIFI-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15163368#comment-15163368 ] James Wing commented on NIFI-1509: -- Thanks for explaining, I certainly don't know of a better way. Your error handling seems very comprehensive. > Support AWS Lambda Processor > > > Key: NIFI-1509 > URL: https://issues.apache.org/jira/browse/NIFI-1509 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Affects Versions: 0.5.0 > Environment: all >Reporter: Mans Singh >Assignee: Mans Singh >Priority: Minor > Labels: aws, lambda, streaming > Fix For: 0.6.0 > > Original Estimate: 48h > Remaining Estimate: 48h > > Support aws lambda put processor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1548) Controller Service "Usage" Icon not working
[ https://issues.apache.org/jira/browse/NIFI-1548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15157978#comment-15157978 ] James Wing commented on NIFI-1548: -- The usage button for Reporting Services is also not working. Both appear to be unintended side-effects of resolving merge conflicts for NIFI-259, where a View State button was added to the services grids, but the event handlers for Usage were dropped from nf-settings.js. The commit is [7314af61778b0803a3529551bcc169e981897a6c|https://github.com/apache/nifi/commit/7314af61778b0803a3529551bcc169e981897a6c], with the relevant snippets being * [line 778|https://github.com/apache/nifi/commit/7314af61778b0803a3529551bcc169e981897a6c#diff-9650a4b2c3b898e9fe57038e622eb7a1L778] for controller services * [line 1446|https://github.com/apache/nifi/commit/7314af61778b0803a3529551bcc169e981897a6c#diff-9650a4b2c3b898e9fe57038e622eb7a1L1446] for reporting services > Controller Service "Usage" Icon not working > --- > > Key: NIFI-1548 > URL: https://issues.apache.org/jira/browse/NIFI-1548 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 0.5.0 >Reporter: Brandon DeVries >Priority: Minor > > In The settings page on the "Contoller Services" tab, the "Usage" icon next > to each service doesn't do anything when clicked. Tested on FireFox 24.7.0 & > 31.5.0 and Chrome 36.0.1985.125. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1509) Support AWS Lambda Processor
[
https://issues.apache.org/jira/browse/NIFI-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15157792#comment-15157792
]
James Wing commented on NIFI-1509:
--
I tested the code from the pull request. The build works fine, unit tests
pass, all contrib checks pass OK. I did not run the integration tests, but
used them as inspiration for my manual testing. Everything worked great in my
use of the PutLambda processor running against Lambda function in AWS. Errors
for permissions, function naming, and content formatting were all handled
appropriately. The function response was captured correctly.
I recommend changing the Capabilities description to make it clear the FlowFile
content should be JSON, and the permission needed to call the Lambda function
(as opposed to lambda:InvokeAsync and other API methods). Maybe something like:
{quote}
Sends the FlowFile contents to a specified Amazon Lambda Function. The
FlowFile content must be JSON. Requires the AWS permission
lambda:InvokeFunction.
{quote}
And I have a question: In PutLambda.java, onTrigger method, you call
{{context.yield()}} after catching several exceptions and transferring the
FlowFile to the failure path, but why not in the first catch block around line
195? I'm asking because I don't understand and would like to learn, I have no
basis for believing there is anything wrong with your choice. Other processors
I looked at were inconsistent or never called yield().
{code:title=PutLambda.java}
...
} catch (final InvalidRequestContentException
| InvalidParameterValueException
| RequestTooLargeException
| ResourceNotFoundException
| UnsupportedMediaTypeException unrecoverableException) {
getLogger().error("Failed to invoke lambda {} with
unrecoverable exception {} for flow file {}",
new Object[]{functionName, unrecoverableException,
flowFile});
flowFile = populateExceptionAttributes(session, flowFile,
unrecoverableException);
session.transfer(flowFile, REL_FAILURE);
} catch (final TooManyRequestsException retryableServiceException) {
getLogger().error("Failed to invoke lambda {} with exception {} for
flow file {}, therefore penalizing flowfile",
new Object[]{functionName, retryableServiceException,
flowFile});
flowFile = populateExceptionAttributes(session, flowFile,
retryableServiceException);
flowFile = session.penalize(flowFile);
session.transfer(flowFile, REL_FAILURE);
context.yield();
...
{code}
> Support AWS Lambda Processor
>
>
> Key: NIFI-1509
> URL: https://issues.apache.org/jira/browse/NIFI-1509
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
>Affects Versions: 0.5.0
> Environment: all
>Reporter: Mans Singh
>Assignee: Mans Singh
>Priority: Minor
> Labels: aws, lambda, streaming
> Fix For: 0.6.0
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Support aws lambda put processor
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-1542) PutS3Object Processor Logs Errors Without ListBucketMultipartUploads Permission
[
https://issues.apache.org/jira/browse/NIFI-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15157591#comment-15157591
]
James Wing commented on NIFI-1542:
--
Thanks, [~jskora], for addressing this so quickly, and on a weekend no less!
The fix works good for me, I think it's a solid improvement. I now see
warnings rather than errors logged, the warnings are specific and helpful about
the permission required, and they are logged only at the age off interval.
> PutS3Object Processor Logs Errors Without ListBucketMultipartUploads
> Permission
> ---
>
> Key: NIFI-1542
> URL: https://issues.apache.org/jira/browse/NIFI-1542
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 0.5.0
> Environment: Linux, Chrome
>Reporter: James Wing
>Assignee: Joe Skora
>Priority: Minor
> Fix For: 0.5.1
>
> Attachments: error-checking-s3-multipart-upload-list.png
>
>
> In 0.5.0, I am seeing an error putting objects to Amazon S3 using the
> PutS3Object processor. The *put operation itself succeeds*, but a nasty-gram
> is displayed in the flow and logged as a side effect. The error text is
> "Error checking S3 Multipart Upload list for ", from line 698
> of PutS3Object.java. This code was introduced in NIFI-1107, expanding the
> PutS3Object processor with support for S3 multi-part uploads of larger files.
> Logged in nifi-app.log:
> {code}
> 2016-02-19 20:56:24,312 ERROR [Timer-Driven Process Thread-3]
> o.a.nifi.processors.aws.s3.PutS3Object PutS
> 3Object[id=47df7533-91b2-4635-b865-45323f73f6c3] Error checking S3 Multipart
> Upload list for batchiq-scra
> tch: Access Denied (Service: Amazon S3; Status Code: 403; Error Code:
> AccessDenied; Request ID: 7483E76EA
> 52E33C4)
> 2016-02-19 20:56:24,533 INFO [Timer-Driven Process Thread-3]
> o.a.nifi.processors.aws.s3.PutS3Object PutS3
> Object[id=47df7533-91b2-4635-b865-45323f73f6c3] Successfully put
> StandardFlowFileRecord[uuid=a4cc19f5-cd2
> 4-4be7-8157-21f2609fa8cd,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1455912275117-1, container=default,
> section=1], offset=3072,
> length=1024],offset=0,name=273670477119454,size=1024] to Amazon S3 in 327
> milliseconds
> {code}
> The error-but-success behavior appears to happen because the exception is
> caught, logged, but not re-thrown. The approximate code path is as follows:
> # PutS3Object::onTrigger method calls ageoffS3Uploads regardless of
> single-part/multi-part code path
> # ageoffS3Uploads calls getS3AgeoffListAndAgeoffLocalState
> # getS3AgeoffListAndAgeoffLocalState calls s3.listMultipartUploads
> # After the exception, onTrigger proceeds with the single-part upload path
> *Troubleshooting Notes*
> Listing multi-part S3 uploads requires an additional S3 permission,
> s3:ListBucketMultipartUploads. I would expect the PutS3Object processor to
> only require the s3:PutObject permission for single-part uploads.
> I tested that this behavior did not occur in NiFi 0.4.2. I also tested that
> adding the s3:ListBucketMultipartUploads permission to my AWS credential
> circumvents the error messages from the S3PutObject processor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-1542) PutS3Object Processor Logs Errors Without ListBucketMultipartUploads Permission
[
https://issues.apache.org/jira/browse/NIFI-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15155849#comment-15155849
]
James Wing commented on NIFI-1542:
--
Thanks for explaining, that helps. Is it possible to first check the local
state to see if there are any multipart uploads before checking AWS for the
current status?
I certainly would agree to #2, logging a warning.
For #3, if we add another property, I would suggest something like "Allow
Multipart Upload" as a True/False option. For me, completely opting out seems
easier to understand than grappling with ageoff. I didn't even know that was a
thing until today.
> PutS3Object Processor Logs Errors Without ListBucketMultipartUploads
> Permission
> ---
>
> Key: NIFI-1542
> URL: https://issues.apache.org/jira/browse/NIFI-1542
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 0.5.0
> Environment: Linux, Chrome
>Reporter: James Wing
>Priority: Minor
> Fix For: 0.5.1
>
> Attachments: error-checking-s3-multipart-upload-list.png
>
>
> In 0.5.0, I am seeing an error putting objects to Amazon S3 using the
> PutS3Object processor. The *put operation itself succeeds*, but a nasty-gram
> is displayed in the flow and logged as a side effect. The error text is
> "Error checking S3 Multipart Upload list for ", from line 698
> of PutS3Object.java. This code was introduced in NIFI-1107, expanding the
> PutS3Object processor with support for S3 multi-part uploads of larger files.
> Logged in nifi-app.log:
> {code}
> 2016-02-19 20:56:24,312 ERROR [Timer-Driven Process Thread-3]
> o.a.nifi.processors.aws.s3.PutS3Object PutS
> 3Object[id=47df7533-91b2-4635-b865-45323f73f6c3] Error checking S3 Multipart
> Upload list for batchiq-scra
> tch: Access Denied (Service: Amazon S3; Status Code: 403; Error Code:
> AccessDenied; Request ID: 7483E76EA
> 52E33C4)
> 2016-02-19 20:56:24,533 INFO [Timer-Driven Process Thread-3]
> o.a.nifi.processors.aws.s3.PutS3Object PutS3
> Object[id=47df7533-91b2-4635-b865-45323f73f6c3] Successfully put
> StandardFlowFileRecord[uuid=a4cc19f5-cd2
> 4-4be7-8157-21f2609fa8cd,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1455912275117-1, container=default,
> section=1], offset=3072,
> length=1024],offset=0,name=273670477119454,size=1024] to Amazon S3 in 327
> milliseconds
> {code}
> The error-but-success behavior appears to happen because the exception is
> caught, logged, but not re-thrown. The approximate code path is as follows:
> # PutS3Object::onTrigger method calls ageoffS3Uploads regardless of
> single-part/multi-part code path
> # ageoffS3Uploads calls getS3AgeoffListAndAgeoffLocalState
> # getS3AgeoffListAndAgeoffLocalState calls s3.listMultipartUploads
> # After the exception, onTrigger proceeds with the single-part upload path
> *Troubleshooting Notes*
> Listing multi-part S3 uploads requires an additional S3 permission,
> s3:ListBucketMultipartUploads. I would expect the PutS3Object processor to
> only require the s3:PutObject permission for single-part uploads.
> I tested that this behavior did not occur in NiFi 0.4.2. I also tested that
> adding the s3:ListBucketMultipartUploads permission to my AWS credential
> circumvents the error messages from the S3PutObject processor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-1542) PutS3Object Processor Logs Errors Without ListBucketMultipartUploads Permission
[
https://issues.apache.org/jira/browse/NIFI-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15155799#comment-15155799
]
James Wing commented on NIFI-1542:
--
Would it be possible to move the call to ageoffS3Uploads() from line 380,
before determining single/multi-part path, down to around line 454 where we are
definitely doing multi-part uploads?
It seems odd to both log one message for each file going down the single-part
upload path, but also unnecessary to do one round trip web service call to AWS
for each file. I freely concede that I do not understand the intricacies of
multi-part upload.
> PutS3Object Processor Logs Errors Without ListBucketMultipartUploads
> Permission
> ---
>
> Key: NIFI-1542
> URL: https://issues.apache.org/jira/browse/NIFI-1542
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 0.5.0
> Environment: Linux, Chrome
>Reporter: James Wing
>Priority: Minor
> Fix For: 0.5.1
>
> Attachments: error-checking-s3-multipart-upload-list.png
>
>
> In 0.5.0, I am seeing an error putting objects to Amazon S3 using the
> PutS3Object processor. The *put operation itself succeeds*, but a nasty-gram
> is displayed in the flow and logged as a side effect. The error text is
> "Error checking S3 Multipart Upload list for ", from line 698
> of PutS3Object.java. This code was introduced in NIFI-1107, expanding the
> PutS3Object processor with support for S3 multi-part uploads of larger files.
> Logged in nifi-app.log:
> {code}
> 2016-02-19 20:56:24,312 ERROR [Timer-Driven Process Thread-3]
> o.a.nifi.processors.aws.s3.PutS3Object PutS
> 3Object[id=47df7533-91b2-4635-b865-45323f73f6c3] Error checking S3 Multipart
> Upload list for batchiq-scra
> tch: Access Denied (Service: Amazon S3; Status Code: 403; Error Code:
> AccessDenied; Request ID: 7483E76EA
> 52E33C4)
> 2016-02-19 20:56:24,533 INFO [Timer-Driven Process Thread-3]
> o.a.nifi.processors.aws.s3.PutS3Object PutS3
> Object[id=47df7533-91b2-4635-b865-45323f73f6c3] Successfully put
> StandardFlowFileRecord[uuid=a4cc19f5-cd2
> 4-4be7-8157-21f2609fa8cd,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1455912275117-1, container=default,
> section=1], offset=3072,
> length=1024],offset=0,name=273670477119454,size=1024] to Amazon S3 in 327
> milliseconds
> {code}
> The error-but-success behavior appears to happen because the exception is
> caught, logged, but not re-thrown. The approximate code path is as follows:
> # PutS3Object::onTrigger method calls ageoffS3Uploads regardless of
> single-part/multi-part code path
> # ageoffS3Uploads calls getS3AgeoffListAndAgeoffLocalState
> # getS3AgeoffListAndAgeoffLocalState calls s3.listMultipartUploads
> # After the exception, onTrigger proceeds with the single-part upload path
> *Troubleshooting Notes*
> Listing multi-part S3 uploads requires an additional S3 permission,
> s3:ListBucketMultipartUploads. I would expect the PutS3Object processor to
> only require the s3:PutObject permission for single-part uploads.
> I tested that this behavior did not occur in NiFi 0.4.2. I also tested that
> adding the s3:ListBucketMultipartUploads permission to my AWS credential
> circumvents the error messages from the S3PutObject processor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-1542) PutS3Object Processor Logs Errors Without ListBucketMultipartUploads Permission
[
https://issues.apache.org/jira/browse/NIFI-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Wing updated NIFI-1542:
-
Attachment: error-checking-s3-multipart-upload-list.png
> PutS3Object Processor Logs Errors Without ListBucketMultipartUploads
> Permission
> ---
>
> Key: NIFI-1542
> URL: https://issues.apache.org/jira/browse/NIFI-1542
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 0.5.0
> Environment: Linux, Chrome
>Reporter: James Wing
>Priority: Minor
> Attachments: error-checking-s3-multipart-upload-list.png
>
>
> In 0.5.0, I am seeing an error putting objects to Amazon S3 using the
> PutS3Object processor. The *put operation itself succeeds*, but a nasty-gram
> is displayed in the flow and logged as a side effect. The error text is
> "Error checking S3 Multipart Upload list for ", from line 698
> of PutS3Object.java. This code was introduced in NIFI-1107, expanding the
> PutS3Object processor with support for S3 multi-part uploads of larger files.
> Logged in nifi-app.log:
> {code}
> 2016-02-19 20:56:24,312 ERROR [Timer-Driven Process Thread-3]
> o.a.nifi.processors.aws.s3.PutS3Object PutS
> 3Object[id=47df7533-91b2-4635-b865-45323f73f6c3] Error checking S3 Multipart
> Upload list for batchiq-scra
> tch: Access Denied (Service: Amazon S3; Status Code: 403; Error Code:
> AccessDenied; Request ID: 7483E76EA
> 52E33C4)
> 2016-02-19 20:56:24,533 INFO [Timer-Driven Process Thread-3]
> o.a.nifi.processors.aws.s3.PutS3Object PutS3
> Object[id=47df7533-91b2-4635-b865-45323f73f6c3] Successfully put
> StandardFlowFileRecord[uuid=a4cc19f5-cd2
> 4-4be7-8157-21f2609fa8cd,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1455912275117-1, container=default,
> section=1], offset=3072,
> length=1024],offset=0,name=273670477119454,size=1024] to Amazon S3 in 327
> milliseconds
> {code}
> The error-but-success behavior appears to happen because the exception is
> caught, logged, but not re-thrown. The approximate code path is as follows:
> # PutS3Object::onTrigger method calls ageoffS3Uploads regardless of
> single-part/multi-part code path
> # ageoffS3Uploads calls getS3AgeoffListAndAgeoffLocalState
> # getS3AgeoffListAndAgeoffLocalState calls s3.listMultipartUploads
> # After the exception, onTrigger proceeds with the single-part upload path
> *Troubleshooting Notes*
> Listing multi-part S3 uploads requires an additional S3 permission,
> s3:ListBucketMultipartUploads. I would expect the PutS3Object processor to
> only require the s3:PutObject permission for single-part uploads.
> I tested that this behavior did not occur in NiFi 0.4.2. I also tested that
> adding the s3:ListBucketMultipartUploads permission to my AWS credential
> circumvents the error messages from the S3PutObject processor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (NIFI-1542) PutS3Object Processor Logs Errors Without ListBucketMultipartUploads Permission
James Wing created NIFI-1542:
Summary: PutS3Object Processor Logs Errors Without
ListBucketMultipartUploads Permission
Key: NIFI-1542
URL: https://issues.apache.org/jira/browse/NIFI-1542
Project: Apache NiFi
Issue Type: Bug
Components: Extensions
Affects Versions: 0.5.0
Environment: Linux, Chrome
Reporter: James Wing
Priority: Minor
In 0.5.0, I am seeing an error putting objects to Amazon S3 using the
PutS3Object processor. The *put operation itself succeeds*, but a nasty-gram
is displayed in the flow and logged as a side effect. The error text is "Error
checking S3 Multipart Upload list for ", from line 698 of
PutS3Object.java. This code was introduced in NIFI-1107, expanding the
PutS3Object processor with support for S3 multi-part uploads of larger files.
Logged in nifi-app.log:
{code}
2016-02-19 20:56:24,312 ERROR [Timer-Driven Process Thread-3]
o.a.nifi.processors.aws.s3.PutS3Object PutS
3Object[id=47df7533-91b2-4635-b865-45323f73f6c3] Error checking S3 Multipart
Upload list for batchiq-scra
tch: Access Denied (Service: Amazon S3; Status Code: 403; Error Code:
AccessDenied; Request ID: 7483E76EA
52E33C4)
2016-02-19 20:56:24,533 INFO [Timer-Driven Process Thread-3]
o.a.nifi.processors.aws.s3.PutS3Object PutS3
Object[id=47df7533-91b2-4635-b865-45323f73f6c3] Successfully put
StandardFlowFileRecord[uuid=a4cc19f5-cd2
4-4be7-8157-21f2609fa8cd,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1455912275117-1, container=default,
section=1], offset=3072, length=1024],offset=0,name=273670477119454,size=1024]
to Amazon S3 in 327 milliseconds
{code}
The error-but-success behavior appears to happen because the exception is
caught, logged, but not re-thrown. The approximate code path is as follows:
# PutS3Object::onTrigger method calls ageoffS3Uploads regardless of
single-part/multi-part code path
# ageoffS3Uploads calls getS3AgeoffListAndAgeoffLocalState
# getS3AgeoffListAndAgeoffLocalState calls s3.listMultipartUploads
# After the exception, onTrigger proceeds with the single-part upload path
*Troubleshooting Notes*
Listing multi-part S3 uploads requires an additional S3 permission,
s3:ListBucketMultipartUploads. I would expect the PutS3Object processor to
only require the s3:PutObject permission for single-part uploads.
I tested that this behavior did not occur in NiFi 0.4.2. I also tested that
adding the s3:ListBucketMultipartUploads permission to my AWS credential
circumvents the error messages from the S3PutObject processor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-786) Add other supporting options for configuring credentials for AWS processors
[
https://issues.apache.org/jira/browse/NIFI-786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15146200#comment-15146200
]
James Wing commented on NIFI-786:
-
As a fanboy for Default Credentials, I definitely agree with you and would be
happy to add documentation. There may also be some changes to the properties
and property behavior that would make it more obvious. I'll have to experiment
with that.
> Add other supporting options for configuring credentials for AWS processors
> ---
>
> Key: NIFI-786
> URL: https://issues.apache.org/jira/browse/NIFI-786
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 0.3.0
>Reporter: Michael Kobit
>Priority: Minor
>
> I was looking at https://issues.apache.org/jira/browse/NIFI-770 and looked at
> how the AWS processors credentials are currently configured. As a NFM you
> have a few options with the properties right now:
> 1) set basic, static credentials
> 2) set a credentials properties filepath
> 3) set neither, use anonymous credentials
> I think it would be better if each AWS could rely on a ControllerService that
> returns `AWSCredentialsProvider` (instead of `AWSCredentials`) that gives
> all of the possible implementations that could be used, rather than relying
> on a static credentials. *Provider implementations can be refreshed and can
> also other more complicated implementations, but already have built in
> support for the Static and Properties file that are provided by NiFi today.
> My thinking is that the controller service would be something like
> public interface AwsCredentialsProviderService extends ControllerService {
> AWSCredentialsProvider getCredentialsProvider();
> }
> and you could have `StaticAwsCredentialsProviderService`,
> `PropertiesFileAwsCredentialsProviderService`, and
> `AnonymousAwsCredentialsProviderService` to provide the functionality that is
> supported right now. Additional credential providers could be added later, as
> there a bunch more AWS provided versions that I think could fit in well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-1500) NiFi requires too much write permissions to bootstrap
[
https://issues.apache.org/jira/browse/NIFI-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15142226#comment-15142226
]
James Wing commented on NIFI-1500:
--
Refactoring how the variable data is written may also be a simplification for
upgrades and backups, in addition to the security benefit. The refactored
directory tree in the [upgrade guidance
|https://cwiki.apache.org/confluence/display/NIFI/Upgrading+NiFi] shares some
interesting similarities with what [~trixpan] proposes.
> NiFi requires too much write permissions to bootstrap
> -
>
> Key: NIFI-1500
> URL: https://issues.apache.org/jira/browse/NIFI-1500
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 0.5.0
>Reporter: Andre
> Fix For: 0.6.0
>
>
> As part of NIFI-1461 it was observed that NiFi boot strapping and start up
> process require access that may be described as excessive.
> The following access requirements should be reviewed (all paths are relative
> to NiFi's installation path):
> h4. Issue 1 - {{bin/nifi.sh}} requires write access to {{bin}}
> Access is used to write {{nifi.pid}}
> h4. Issue 2 - NiFi requires write access to {{lib}} crashing otherwise
> {code}
> 2016-02-11 13:00:35,497 INFO [main] org.apache.nifi.NiFi Launching NiFi...
> 2016-02-11 13:00:35,524 INFO [main] org.apache.nifi.BootstrapListener Started
> Bootstrap Listener, Listening for incoming requests on port 52097
> 2016-02-11 13:00:35,545 INFO [main] org.apache.nifi.BootstrapListener
> Successfully initiated communication with Bootstrap
> 2016-02-11 13:00:35,555 WARN [main] org.apache.nifi.nar.NarUnpacker Unable to
> load NAR library bundles due to java.io.IOException:
> /opt/nifi/nifi-0.5.0-SNAPSHOT/./lib directory does not have read/write
> privilege Will proceed without loading any further Nar bundles
> 2016-02-11 13:00:35,559 ERROR [main] org.apache.nifi.NiFi Failure to launch
> NiFi due to java.lang.IllegalStateException: Unable to find the framework NAR
> ClassLoader.
> java.lang.IllegalStateException: Unable to find the framework NAR ClassLoader.
> at org.apache.nifi.NiFi.(NiFi.java:116)
> ~[nifi-runtime-0.5.0-SNAPSHOT.jar:0.5.0-SNAPSHOT]
> at org.apache.nifi.NiFi.main(NiFi.java:227)
> ~[nifi-runtime-0.5.0-SNAPSHOT.jar:0.5.0-SNAPSHOT]
> 2016-02-11 13:00:35,561 INFO [Thread-1] org.apache.nifi.NiFi Initiating
> shutdown of Jetty web server...
> 2016-02-11 13:00:35,561 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server
> shutdown completed (nicely or otherwise).
> {code}
> h4. Issue 3 - NiFi requires write access to {{conf}}
> Access is used by the process to save {{flows.xml.gz}} and to create
> {{templates}} and {{archives}} upon first start.
> h4. Issue 4 - NiFi requires write access to {{.}}
> Application uses access upon first start to create the following directories:
> - \*_repositories;
> - logs;
> - state;
> - work;
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-1461) RPM should create nifi user and give it access to the /opt/nifi and subdirectories
[ https://issues.apache.org/jira/browse/NIFI-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15141694#comment-15141694 ] James Wing commented on NIFI-1461: -- I agree there may be some security improvements to be had by rearranging the distribution files and revisiting permissions. But I'm not convinced that these issues are exclusive to the RPM install. The tarball installation has essentially the same files and permissions, and would be subject to the same vulnerability scenarios. The changes made in this ticket are valuable improvements to RPM usability and security as-is, and I hope we don't let perfect become the enemy of better. > RPM should create nifi user and give it access to the /opt/nifi and > subdirectories > -- > > Key: NIFI-1461 > URL: https://issues.apache.org/jira/browse/NIFI-1461 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 0.4.1 >Reporter: Andre >Assignee: Tony Kurc > Labels: rpm > Fix For: 0.5.0 > > Attachments: > 0001-NIFI-1461-This-closes-204.-Implemented-RPM-before-af.patch, > 0002-NIFI-1461-added-run.as-filtered-property-for-RPM-and.patch, nifi.spec, > nifi.spec > > > currently when installing from RPM the resulting permissions result on admins > having to complete the installation by adding a user and correcting > filesystem permissions. > Ideally the RPM packages should result in installed files ready to be > utilised as a non-root user -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1461) RPM should create nifi user and give it access to the /opt/nifi and subdirectories
[
https://issues.apache.org/jira/browse/NIFI-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15141458#comment-15141458
]
James Wing commented on NIFI-1461:
--
I applied the two attached patches to master and built an RPM with {{mvn clean
install -Prpm}}. The RPM installs correctly, and now runs as 'nifi' by
default. Good stuff!
But it seems that the .zip, .tar.gz, and directory assemblies also have the
{{run.as=nifi}} setting applied to bootstrap.conf when -Prpm is specified in
the build. Running the build again as {{mvn clean install}} (no -Prpm),
results in assemblies without the {{run.as=nifi}}. Since the RPM build is an
optional extra, this might not be a problem, but it did not strike me as
obvious. Is this expected and OK?
> RPM should create nifi user and give it access to the /opt/nifi and
> subdirectories
> --
>
> Key: NIFI-1461
> URL: https://issues.apache.org/jira/browse/NIFI-1461
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 0.4.1
>Reporter: Andre
>Assignee: Tony Kurc
> Labels: rpm
> Fix For: 0.5.0
>
> Attachments:
> 0001-NIFI-1461-This-closes-204.-Implemented-RPM-before-af.patch,
> 0002-NIFI-1461-added-run.as-filtered-property-for-RPM-and.patch, nifi.spec,
> nifi.spec
>
>
> currently when installing from RPM the resulting permissions result on admins
> having to complete the installation by adding a user and correcting
> filesystem permissions.
> Ideally the RPM packages should result in installed files ready to be
> utilised as a non-root user
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-1461) RPM should create nifi user and give it access to the /opt/nifi and subdirectories
[ https://issues.apache.org/jira/browse/NIFI-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139295#comment-15139295 ] James Wing commented on NIFI-1461: -- I reviewed and re-tested the PR up to the fourth commit, 7b3cb71, and it looks good to me. No visible output from getent. Thanks for your efforts on this. > RPM should create nifi user and give it access to the /opt/nifi and > subdirectories > -- > > Key: NIFI-1461 > URL: https://issues.apache.org/jira/browse/NIFI-1461 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 0.4.1 >Reporter: Andre >Assignee: Tony Kurc > Labels: rpm > Attachments: nifi.spec, nifi.spec > > > currently when installing from RPM the resulting permissions result on admins > having to complete the installation by adding a user and correcting > filesystem permissions. > Ideally the RPM packages should result in installed files ready to be > utilised as a non-root user -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (NIFI-1461) RPM should create nifi user and give it access to the /opt/nifi and subdirectories
[ https://issues.apache.org/jira/browse/NIFI-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15137616#comment-15137616 ] James Wing edited comment on NIFI-1461 at 2/8/16 8:22 PM: -- Thanks, [~puspendu.baner...@gmail.com], your latest change in commit ec29694 does successfully fix the lib directory ownership issue. was (Author: jameswing): Thanks, [~puspendu.baner...@gmail.com], your latest change in commit c29694 does successfully fix the lib directory ownership issue. > RPM should create nifi user and give it access to the /opt/nifi and > subdirectories > -- > > Key: NIFI-1461 > URL: https://issues.apache.org/jira/browse/NIFI-1461 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 0.4.1 >Reporter: Andre >Assignee: Tony Kurc > Labels: rpm > Attachments: nifi.spec, nifi.spec > > > currently when installing from RPM the resulting permissions result on admins > having to complete the installation by adding a user and correcting > filesystem permissions. > Ideally the RPM packages should result in installed files ready to be > utilised as a non-root user -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1461) RPM should create nifi user and give it access to the /opt/nifi and subdirectories
[ https://issues.apache.org/jira/browse/NIFI-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15137616#comment-15137616 ] James Wing commented on NIFI-1461: -- Thanks, [~puspendu.baner...@gmail.com], your latest change in commit c29694 does successfully fix the lib directory ownership issue. > RPM should create nifi user and give it access to the /opt/nifi and > subdirectories > -- > > Key: NIFI-1461 > URL: https://issues.apache.org/jira/browse/NIFI-1461 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 0.4.1 >Reporter: Andre >Assignee: Tony Kurc > Labels: rpm > Attachments: nifi.spec, nifi.spec > > > currently when installing from RPM the resulting permissions result on admins > having to complete the installation by adding a user and correcting > filesystem permissions. > Ideally the RPM packages should result in installed files ready to be > utilised as a non-root user -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1461) RPM should create nifi user and give it access to the /opt/nifi and subdirectories
[
https://issues.apache.org/jira/browse/NIFI-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135145#comment-15135145
]
James Wing commented on NIFI-1461:
--
[~puspendu.baner...@gmail.com], I tried building an RPM from [your pull
request|https://github.com/apache/nifi/pull/204] and installing the RPM. The
user and group creation appear to mostly comply with [Fedora's best practice
guidelines for "Dynamic
allocation"|https://fedoraproject.org/wiki/Packaging:UsersAndGroups#Dynamic_allocation],
with a couple of exceptions:
# The output of {{getent}} is not redirected to /dev/null, resulting in output
when running yum install.
# No comment is attached to the user record.
# {{exit 0}} is not called to prevent a failure in user/group creation from
failing the RPM install.
I don't believe these things are all required, but your code was very, very
close to their recommendation and I'm just curious if these differences are by
design or not.
I installed the RPM on Amazon Linux. After installing the RPM, the {{lib}}
directory is still owned by root.
{code}
[root@ip-172-31-28-236 nifi-0.4.2-SNAPSHOT]# ls -l
total 132
drwxr-x--- 2 nifi nifi 4096 Feb 5 20:03 bin
drwxr-x--- 2 nifi nifi 4096 Feb 5 20:03 conf
drwxr-x--- 3 nifi nifi 4096 Feb 5 20:03 docs
drwxr-xr-x 3 root root 4096 Feb 5 20:03 lib
-rw-r- 1 nifi nifi 61811 Feb 5 18:58 LICENSE
-rw-r- 1 nifi nifi 42726 Feb 5 18:35 NOTICE
-rw-r- 1 nifi nifi 4549 Jan 30 21:07 README
{code}
Actually running NiFi as the "nifi" user requires setting the {{run.as}} entry
in bootstrap.conf to "nifi". After doing this and then starting NiFi, this
results in failure due to the lib directory permissions. From nifi-app.log:
{code}
2016-02-05 20:02:13,210 INFO [main] org.apache.nifi.NiFi Launching NiFi...
2016-02-05 20:02:13,237 INFO [main] org.apache.nifi.BootstrapListener Started
Bootstrap Listener, Listening for incoming requests on port 36459
2016-02-05 20:02:13,259 INFO [main] org.apache.nifi.BootstrapListener
Successfully initiated communication with Bootstrap
2016-02-05 20:02:13,272 WARN [main] org.apache.nifi.nar.NarUnpacker Unable to
load NAR library bundles due to java.io.IOException:
/opt/nifi/nifi-0.4.2-SNAPSHOT/./lib directory does not have read/write
privilege Will proceed without loading any further Nar bundles
2016-02-05 20:02:13,277 ERROR [main] org.apache.nifi.NiFi Failure to launch
NiFi due to java.lang.IllegalStateException: Unable to find the framework NAR
ClassLoader.
java.lang.IllegalStateException: Unable to find the framework NAR ClassLoader.
at org.apache.nifi.NiFi.(NiFi.java:116)
~[nifi-runtime-0.4.2-SNAPSHOT.jar:0.4.2-SNAPSHOT]
at org.apache.nifi.NiFi.main(NiFi.java:227)
~[nifi-runtime-0.4.2-SNAPSHOT.jar:0.4.2-SNAPSHOT]
2016-02-05 20:02:13,277 INFO [Thread-1] org.apache.nifi.NiFi Initiating
shutdown of Jetty web server...
2016-02-05 20:02:13,277 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server
shutdown completed (nicely or otherwise).
{code}
I haven't figured out why this is. I tried adding {{username}} and
{{groupname}} parameters to the lib and lib/bootstrap mappings in the POM.xml
file, but it didn't change the ownership of lib.
> RPM should create nifi user and give it access to the /opt/nifi and
> subdirectories
> --
>
> Key: NIFI-1461
> URL: https://issues.apache.org/jira/browse/NIFI-1461
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 0.4.1
>Reporter: Andre
>Assignee: Tony Kurc
> Labels: rpm
> Attachments: nifi.spec
>
>
> currently when installing from RPM the resulting permissions result on admins
> having to complete the installation by adding a user and correcting
> filesystem permissions.
> Ideally the RPM packages should result in installed files ready to be
> utilised as a non-root user
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-1461) RPM should create nifi user and give it access to the /opt/nifi and subdirectories
[ https://issues.apache.org/jira/browse/NIFI-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15134944#comment-15134944 ] James Wing commented on NIFI-1461: -- [~trixpan], would you please elaborate a bit on what you expected the end-state of the RPM install to be? What steps have you had to take that the install should have covered? What commands would you use to run NiFi post-install, which do not work now, but should work as the non-root user? Or to put it another way, how should we verify a fix? > RPM should create nifi user and give it access to the /opt/nifi and > subdirectories > -- > > Key: NIFI-1461 > URL: https://issues.apache.org/jira/browse/NIFI-1461 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 0.4.1 >Reporter: Andre >Assignee: Tony Kurc > Labels: rpm > Attachments: nifi.spec > > > currently when installing from RPM the resulting permissions result on admins > having to complete the installation by adding a user and correcting > filesystem permissions. > Ideally the RPM packages should result in installed files ready to be > utilised as a non-root user -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-1454) Built RPMs Do Not Result in Working NiFi Installation
[
https://issues.apache.org/jira/browse/NIFI-1454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Wing updated NIFI-1454:
-
Attachment: rpm_file_compare.zip
> Built RPMs Do Not Result in Working NiFi Installation
> -
>
> Key: NIFI-1454
> URL: https://issues.apache.org/jira/browse/NIFI-1454
> Project: Apache NiFi
> Issue Type: Bug
> Components: Tools and Build
>Affects Versions: 0.4.1
> Environment: Tested on Amazon Linux with Java 7
>Reporter: James Wing
>Assignee: Joseph Witt
>Priority: Minor
> Labels: build
> Fix For: 0.5.0
>
> Attachments:
> 0001-NIFI-1454-fixed-rpm-name-bootstrap-deps-updated-rpm-.patch,
> rpm_file_compare.zip
>
>
> After building an RPM (mvn clean install -Prpm) and installing it (Amazon
> Linux, Java 1.7.0_91), I ran into the following error attempting to start
> NiFi :
> {noformat}
> # ./nifi.sh start
> Java home: /usr/lib/jvm/java
> NiFi home: /opt/nifi/nifi-0.4.1
> Bootstrap Config File: /opt/nifi/nifi-0.4.1/conf/bootstrap.conf
> Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2625)
> at java.lang.Class.getMethod0(Class.java:2866)
> at java.lang.Class.getMethod(Class.java:1676)
> at sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:494)
> at
> sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:486)
> Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> ... 6 more
> {noformat}
> But the .tar.gz assembly file from the same build works just fine. Comparing
> listings of the two file sets, it appears that the .tar.gz assembly has
> complete set of jar files in lib/bootstrap:
> * activation-1.1.jar
> * antlr-runtime-3.5.2.jar
> * commons-lang3-3.4.jar
> * logback-classic-1.1.3.jar
> * logback-core-1.1.3.jar
> * mail-1.4.7.jar
> * nifi-api-0.4.1.jar
> * nifi-bootstrap-0.4.1.jar
> * nifi-expression-language-0.4.1.jar
> * nifi-processor-utils-0.4.1.jar
> * nifi-security-utils-0.4.1.jar
> * nifi-utils-0.4.1.jar
> * slf4j-api-1.7.12.jar
> while the RPM install's lib/bootstrap has only one of these:
> * nifi-bootstrap-0.4.1.jar
> So I tried to troubleshoot why this is. It looks like the nifi-assembly POM
> is set up to use the same defined set of dependencies for both the
> maven-assembly-plugin and rpm-maven-plugin plugins. The lib folder plan for
> both plugins is something like:
> lib - all dependencies except nifi-bootstrap
> lib/bootstrap - include only nifi-bootstrap
> The problem appears to be a difference between how the two plugins handle
> transitive dependencies. The maven-assembly-plugin has configuration options
> allowing us to specify that the exclusions apply to the named plugins AND
> their transitive dependencies. The rpm-maven-plugin does not have a
> transitive configuration option, and does not appear to do the same thing by
> default. In fact, the RPM is built with nifi-bootstrap-0.4.1.jar in the
> lib/bootstrap folder, but all of its transitive dependencies (such as slf4j
> from the error) are in the lib folder. I believe the RPM installs LICENSE
> and NOTICE in the docs folder for the same reason.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-1454) Built RPMs Do Not Result in Working NiFi Installation
[
https://issues.apache.org/jira/browse/NIFI-1454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15125502#comment-15125502
]
James Wing commented on NIFI-1454:
--
Your patch looks pretty good. I verified that the build works, and that the
generated RPM successfully installs. I also compared the file listings between
the built RPM, the build .tar.gz, and an 0.4.1 .tar.gz to confirm expected
changes and watch for unexpected effects (attached).
There appears to be a typo in the Java Mail dependency spec. The RPM installs
and runs mostly fine, until you configure email notifications, then I got an
exception. The current spec is:
{code:xml|title=nifi-assembly/pom.xml:543|borderStyle=solid}
java.mail:mail
{code}
Instead, I believe it should be "javax.mail":
{code:xml|title=nifi-assembly/pom.xml:543|borderStyle=solid}
javax.mail:mail
{code}
I like the solution approach. Defining the explicit include list of bootstrap
dependencies results in fewer duplicate files installed. It is not necessary
to maintain a long list of NAR file excludes. The simplified 'nifi' name is
also much appreciated.
However, it is necessary to maintain the list of explicit bootstrap includes,
and forgetting to do so will break the resulting RPM installation. I believe
this is a reasonable trade, given that changes to the bootstrap dependencies
are likely to be much fewer and farther between than NAR additions, and more
likely to be performed and reviewed by experienced contributors.
Your solution is better than mine, this looks like the best option available
with the rpm-maven-plugin. Nice work.
> Built RPMs Do Not Result in Working NiFi Installation
> -
>
> Key: NIFI-1454
> URL: https://issues.apache.org/jira/browse/NIFI-1454
> Project: Apache NiFi
> Issue Type: Bug
> Components: Tools and Build
>Affects Versions: 0.4.1
> Environment: Tested on Amazon Linux with Java 7
>Reporter: James Wing
>Assignee: Joseph Witt
>Priority: Minor
> Labels: build
> Fix For: 0.5.0
>
> Attachments:
> 0001-NIFI-1454-fixed-rpm-name-bootstrap-deps-updated-rpm-.patch,
> rpm_file_compare.zip
>
>
> After building an RPM (mvn clean install -Prpm) and installing it (Amazon
> Linux, Java 1.7.0_91), I ran into the following error attempting to start
> NiFi :
> {noformat}
> # ./nifi.sh start
> Java home: /usr/lib/jvm/java
> NiFi home: /opt/nifi/nifi-0.4.1
> Bootstrap Config File: /opt/nifi/nifi-0.4.1/conf/bootstrap.conf
> Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2625)
> at java.lang.Class.getMethod0(Class.java:2866)
> at java.lang.Class.getMethod(Class.java:1676)
> at sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:494)
> at
> sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:486)
> Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> ... 6 more
> {noformat}
> But the .tar.gz assembly file from the same build works just fine. Comparing
> listings of the two file sets, it appears that the .tar.gz assembly has
> complete set of jar files in lib/bootstrap:
> * activation-1.1.jar
> * antlr-runtime-3.5.2.jar
> * commons-lang3-3.4.jar
> * logback-classic-1.1.3.jar
> * logback-core-1.1.3.jar
> * mail-1.4.7.jar
> * nifi-api-0.4.1.jar
> * nifi-bootstrap-0.4.1.jar
> * nifi-expression-language-0.4.1.jar
> * nifi-processor-utils-0.4.1.jar
> * nifi-security-utils-0.4.1.jar
> * nifi-utils-0.4.1.jar
> * slf4j-api-1.7.12.jar
> while the RPM install's lib/bootstrap has only one of these:
> * nifi-bootstrap-0.4.1.jar
> So I tried to troubleshoot why this is. It looks like the nifi-assembly POM
> is set up to use the same defined set of dependencies for both the
> maven-assembly-plugin and rpm-maven-plugin plugins. The lib folder plan for
> both plugins is something like:
> lib - all dependencies except nifi-bootstrap
> lib/bootstrap - include only nifi-bootstrap
> The problem appears to be a difference between how the two plugins handle
> transitive dependencies. The maven-assembly-plugin has configuration options
> allowing us to specify that the e
[jira] [Created] (NIFI-1454) Built RPMs Do Not Result in Working NiFi Installation
James Wing created NIFI-1454:
Summary: Built RPMs Do Not Result in Working NiFi Installation
Key: NIFI-1454
URL: https://issues.apache.org/jira/browse/NIFI-1454
Project: Apache NiFi
Issue Type: Bug
Components: Tools and Build
Affects Versions: 0.4.1
Environment: Tested on Amazon Linux with Java 7
Reporter: James Wing
Priority: Minor
After building an RPM (mvn clean install -Prpm) and installing it (Amazon
Linux, Java 1.7.0_91), I ran into the following error attempting to start NiFi :
{noformat}
# ./nifi.sh start
Java home: /usr/lib/jvm/java
NiFi home: /opt/nifi/nifi-0.4.1
Bootstrap Config File: /opt/nifi/nifi-0.4.1/conf/bootstrap.conf
Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2625)
at java.lang.Class.getMethod0(Class.java:2866)
at java.lang.Class.getMethod(Class.java:1676)
at sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:494)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:486)
Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 6 more
{noformat}
But the .tar.gz assembly file from the same build works just fine. Comparing
listings of the two file sets, it appears that the .tar.gz assembly has
complete set of jar files in lib/bootstrap:
* activation-1.1.jar
* antlr-runtime-3.5.2.jar
* commons-lang3-3.4.jar
* logback-classic-1.1.3.jar
* logback-core-1.1.3.jar
* mail-1.4.7.jar
* nifi-api-0.4.1.jar
* nifi-bootstrap-0.4.1.jar
* nifi-expression-language-0.4.1.jar
* nifi-processor-utils-0.4.1.jar
* nifi-security-utils-0.4.1.jar
* nifi-utils-0.4.1.jar
* slf4j-api-1.7.12.jar
while the RPM install's lib/bootstrap has only one of these:
* nifi-bootstrap-0.4.1.jar
So I tried to troubleshoot why this is. It looks like the nifi-assembly POM is
set up to use the same defined set of dependencies for both the
maven-assembly-plugin and rpm-maven-plugin plugins. The lib folder plan for
both plugins is something like:
lib - all dependencies except nifi-bootstrap
lib/bootstrap - include only nifi-bootstrap
The problem appears to be a difference between how the two plugins handle
transitive dependencies. The maven-assembly-plugin has configuration options
allowing us to specify that the exclusions apply to the named plugins AND their
transitive dependencies. The rpm-maven-plugin does not have a transitive
configuration option, and does not appear to do the same thing by default. In
fact, the RPM is built with nifi-bootstrap-0.4.1.jar in the lib/bootstrap
folder, but all of its transitive dependencies (such as slf4j from the error)
are in the lib folder. I believe the RPM installs LICENSE and NOTICE in the
docs folder for the same reason.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-1325) Enhance AWS S3 fetch to access bucket across accounts
[ https://issues.apache.org/jira/browse/NIFI-1325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15095377#comment-15095377 ] James Wing commented on NIFI-1325: -- The changes in a0ee31f look good to me. I (lightly) reviewed the diffs, built it, and ran a simple flow using both default credentials and assuming a role. No problems. > Enhance AWS S3 fetch to access bucket across accounts > - > > Key: NIFI-1325 > URL: https://issues.apache.org/jira/browse/NIFI-1325 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 0.4.1 > Environment: All >Reporter: Mans Singh >Assignee: Tony Kurc >Priority: Minor > Labels: easyfix > Fix For: 0.4.1 > > Attachments: JW_Patch_On_MS_0005.zip, nifi-1325.patch.zip, > nifi-1325.patch.zip, nifi-1325.patch.zip > > Original Estimate: 48h > Remaining Estimate: 48h > > The AWS S3 Fetch Object component does not allow access to bucket across > accounts. AWS S3 Fetch Object with can be enhanced to provide this > functionality by using assume role session/credentials -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1325) Enhance AWS S3 fetch to access bucket across accounts
[ https://issues.apache.org/jira/browse/NIFI-1325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15095297#comment-15095297 ] James Wing commented on NIFI-1325: -- Mans, thanks for pushing for this feature and taking the lead on getting code together. > Enhance AWS S3 fetch to access bucket across accounts > - > > Key: NIFI-1325 > URL: https://issues.apache.org/jira/browse/NIFI-1325 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 0.4.1 > Environment: All >Reporter: Mans Singh >Assignee: Tony Kurc >Priority: Minor > Labels: easyfix > Fix For: 0.4.1 > > Attachments: JW_Patch_On_MS_0005.zip, nifi-1325.patch.zip, > nifi-1325.patch.zip, nifi-1325.patch.zip > > Original Estimate: 48h > Remaining Estimate: 48h > > The AWS S3 Fetch Object component does not allow access to bucket across > accounts. AWS S3 Fetch Object with can be enhanced to provide this > functionality by using assume role session/credentials -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1325) Enhance AWS S3 fetch to access bucket across accounts
[ https://issues.apache.org/jira/browse/NIFI-1325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15095288#comment-15095288 ] James Wing commented on NIFI-1325: -- I will take a look, thanks. > Enhance AWS S3 fetch to access bucket across accounts > - > > Key: NIFI-1325 > URL: https://issues.apache.org/jira/browse/NIFI-1325 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 0.4.1 > Environment: All >Reporter: Mans Singh >Assignee: Tony Kurc >Priority: Minor > Labels: easyfix > Fix For: 0.4.1 > > Attachments: JW_Patch_On_MS_0005.zip, nifi-1325.patch.zip, > nifi-1325.patch.zip, nifi-1325.patch.zip > > Original Estimate: 48h > Remaining Estimate: 48h > > The AWS S3 Fetch Object component does not allow access to bucket across > accounts. AWS S3 Fetch Object with can be enhanced to provide this > functionality by using assume role session/credentials -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1325) Enhance AWS S3 fetch to access bucket across accounts
[ https://issues.apache.org/jira/browse/NIFI-1325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15094551#comment-15094551 ] James Wing commented on NIFI-1325: -- I like the update, thank you for making those changes. Since the Default Credential in AWS can be sufficient to assume a role, I recommend making two changes to AWSCredentialsProviderControllerService: 1. customValidate: Removing the validation requirement that either API keys or a file be specified in addition to the role information. 1. onConfigured: Creating the default credential after API keys/file credential but before assuming a role, then using the credentialsProvider to do so. I have attached a patch file demonstrating these changes on top of your patch 0005 (JW_Patch_On_MS_0005.zip). > Enhance AWS S3 fetch to access bucket across accounts > - > > Key: NIFI-1325 > URL: https://issues.apache.org/jira/browse/NIFI-1325 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 0.4.1 > Environment: All >Reporter: Mans Singh >Assignee: Tony Kurc >Priority: Minor > Labels: easyfix > Fix For: 0.4.1 > > Attachments: JW_Patch_On_MS_0005.zip, nifi-1325.patch.zip, > nifi-1325.patch.zip, nifi-1325.patch.zip > > Original Estimate: 48h > Remaining Estimate: 48h > > The AWS S3 Fetch Object component does not allow access to bucket across > accounts. AWS S3 Fetch Object with can be enhanced to provide this > functionality by using assume role session/credentials -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (NIFI-1325) Enhance AWS S3 fetch to access bucket across accounts
[ https://issues.apache.org/jira/browse/NIFI-1325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15094551#comment-15094551 ] James Wing edited comment on NIFI-1325 at 1/12/16 7:11 PM: --- I like the update, thank you for making those changes. Since the Default Credential in AWS can be sufficient to assume a role, I recommend making two changes to AWSCredentialsProviderControllerService: 1. customValidate: Removing the validation requirement that either API keys or a file be specified in addition to the role information. 2. onConfigured: Creating the default credential after API keys/file credential but before assuming a role, then using the credentialsProvider to do so. I have attached a patch file demonstrating these changes on top of your patch 0005 (JW_Patch_On_MS_0005.zip). was (Author: jameswing): I like the update, thank you for making those changes. Since the Default Credential in AWS can be sufficient to assume a role, I recommend making two changes to AWSCredentialsProviderControllerService: 1. customValidate: Removing the validation requirement that either API keys or a file be specified in addition to the role information. 1. onConfigured: Creating the default credential after API keys/file credential but before assuming a role, then using the credentialsProvider to do so. I have attached a patch file demonstrating these changes on top of your patch 0005 (JW_Patch_On_MS_0005.zip). > Enhance AWS S3 fetch to access bucket across accounts > - > > Key: NIFI-1325 > URL: https://issues.apache.org/jira/browse/NIFI-1325 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 0.4.1 > Environment: All >Reporter: Mans Singh >Assignee: Tony Kurc >Priority: Minor > Labels: easyfix > Fix For: 0.4.1 > > Attachments: JW_Patch_On_MS_0005.zip, nifi-1325.patch.zip, > nifi-1325.patch.zip, nifi-1325.patch.zip > > Original Estimate: 48h > Remaining Estimate: 48h > > The AWS S3 Fetch Object component does not allow access to bucket across > accounts. AWS S3 Fetch Object with can be enhanced to provide this > functionality by using assume role session/credentials -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-1325) Enhance AWS S3 fetch to access bucket across accounts
[ https://issues.apache.org/jira/browse/NIFI-1325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Wing updated NIFI-1325: - Attachment: JW_Patch_On_MS_0005.zip Changes to allow the Default Credential to assume role. This patch should be applied on top of Mans Singh's 0005-updated-ControllerService-file.patch. > Enhance AWS S3 fetch to access bucket across accounts > - > > Key: NIFI-1325 > URL: https://issues.apache.org/jira/browse/NIFI-1325 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 0.4.1 > Environment: All >Reporter: Mans Singh >Assignee: Tony Kurc >Priority: Minor > Labels: easyfix > Fix For: 0.4.1 > > Attachments: JW_Patch_On_MS_0005.zip, nifi-1325.patch.zip, > nifi-1325.patch.zip, nifi-1325.patch.zip > > Original Estimate: 48h > Remaining Estimate: 48h > > The AWS S3 Fetch Object component does not allow access to bucket across > accounts. AWS S3 Fetch Object with can be enhanced to provide this > functionality by using assume role session/credentials -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1325) Enhance AWS S3 fetch to access bucket across accounts
[
https://issues.apache.org/jira/browse/NIFI-1325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15093066#comment-15093066
]
James Wing commented on NIFI-1325:
--
>From reviewing the patch on 2015-01-11:
Mans, you appear to have faithfully implemented the
AbstractAWSProcessor-subclass design suggested on the dev email list. I have
to admit that I don't really like this design. It feels weird to me to have a
deprecated class which we are still actively using without any future intention
of dropping. It's deprecated because we don't want processor developers to
directly inherit from it, but it's still in active use in the inheritance
chain. The backward compatibility problem is ugly any way I look at it, but I
guess I would prefer a solution involving interfaces or annotations to signal
AWSCredentialProvider readiness over the subclass.
Things I like:
* Your implementation worked in my brief testing
* Individual components can still specify credentials separate from the shared
service
Suggestions:
* Please consider using
[DefaultAWSCredentialsProviderChain|http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html]
instead of AnonymousAWSCredentialProvider() as the unconfigured default
credential. This provider covers a number of common AWS credential
configuration strategies, including EC2 instance profiles/roles.
* If we do have anonymous credentials, I believe we could use the same
StaticCredentialProvider used for basic creds, I don't think we need a custom
AnonymousAWSCredentialsProvider.
* "Assume Role Name" could be renamed "Assume Role Session Name" to clarify
that it identifies the session, not the role. [See the RoleSessionName argument
in AssumeRole action in the STS SDK
docs|http://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html].
* You could add some attributes to the AWSCredentialsProviderServiceImpl to
improve the presentation in the Add Controller Services dialog:
{code}
@CapabilityDescription("Defines credentials for Amazon Web Services
processors.")
@Tags({ "aws", "credentials" })
public class AWSCredentialsProviderServiceImpl ...
{code}
* Naming Conventions - Someone more familiar with NiFi code should chime in,
but don't I think the *Impl convention for a concrete implementation of an
interface is widespread in the NiFi codebase. I looked at the
nifi-couchbase-bundle for comparison, and the interface is
CouchbaseClusterControllerService and the concrete class
CouchbaseClusterService, but I do not really know what the standard is. One
interesting fact is that the class name AWSCredentialsProviderServiceImpl
appears in the Controller Services dialog, so it effectively becomes part of
the user experience. That might explain "CouchbaseClusterService". If there
is a display name, I didn't see any other services using it.
Thanks for putting this together!
> Enhance AWS S3 fetch to access bucket across accounts
> -
>
> Key: NIFI-1325
> URL: https://issues.apache.org/jira/browse/NIFI-1325
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 0.4.1
> Environment: All
>Reporter: Mans Singh
>Assignee: Tony Kurc
>Priority: Minor
> Labels: easyfix
> Fix For: 0.4.1
>
> Attachments: nifi-1325.patch.zip, nifi-1325.patch.zip
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The AWS S3 Fetch Object component does not allow access to bucket across
> accounts. AWS S3 Fetch Object with can be enhanced to provide this
> functionality by using assume role session/credentials
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-786) Add other supporting options for configuring credentials for AWS processors
[ https://issues.apache.org/jira/browse/NIFI-786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089832#comment-15089832 ] James Wing commented on NIFI-786: - The various AWS credential methods should not be thought of as simply different ways to get to a set of API keys, but as different practices for credential configuration management and deployment. For example, * How will the dev team use credentials to develop and test on their own machines? * How can test/stage/production credentials be managed in the runtime environments? * How will cross-account access credentials for customer or partner systems be obtained and controlled? * How will credentials be rotated for long-running systems? * How will credentials be managed across large fleets of servers? AWS has put a great deal of thought into these problems, and their solutions have evolved over time. One unfortunate result is the bewildering array of credential options available now. But current AWS propaganda emphasizes IAM Roles, EC2 instance profiles, and named profiles (from users' ~/.aws/credentials file) over directly providing API keys in application code or config files. * [A New and Standardized Way to Manage Credentials in the AWS SDKs|https://blogs.aws.amazon.com/security/post/Tx3D6U6WSFGOK2H/A-New-and-Standardized-Way-to-Manage-Credentials-in-the-AWS-SDKs] * [Providing AWS Credentials in the AWS SDK for Java|http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/credentials.html] * [Adhere to IAM Best Practices in 2016 (see #10 and #11)|https://blogs.aws.amazon.com/security/post/Tx2OB7YGHMB7WCM/Adhere-to-IAM-Best-Practices-in-2016] The following is a non-exhaustive list of AWS credential methods, and their currently status in NiFi. I recommend NiFi eventually support all of these. ||Credential Method||NiFi Support||Inputs||Notes|| |[Anonymous|http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/AnonymousAWSCredentials.html]|Y|(none)|Default in NiFi. Not widely usable for most AWS services, except S3?| |[Basic|http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/BasicAWSCredentials.html]|Y|Access Key ID, Secret Key|Lowest-common denominator, useful for troubleshooting, but not recommended| |[Properties File|http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/PropertiesCredentials.html]|Y|Path to file with keys| | |[Default|http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html]|N|(none)|The ["Default Credential Provider Chain"|http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/credentials.html#id1] is idiomatic across AWS tools and covers many configuration scenarios, especially EC2 Instance Profile and the default user profile| |[Named Profile|http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/profile/ProfileCredentialsProvider.html]|N|Profile name|Picks a named option out of ~/.aws/credentials file, allowing for multiple methods to be configured on the machine and used in different processors| |[Assumed Role|http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/STSAssumeRoleSessionCredentialsProvider.html]|N|Role ARN, Session Name, ExternalID (opt.)|Two-stage credential, requires one of the above before assuming role. Recommended for cross-account access, see NIFI-1325| To me, the absence of the Default Credential Chain is the most pressing for NiFi. It would be my preference to have the Default Credentials used as the unconfigured default for AWS processors rather than the Anonymous Credential, as is currently the case. This might be a separate issue than a centralized credential service, and maybe I should create a separate ticket. Does anyone know if the Anonymous Credential is currently used out of an affirmative desire to be anonymous or out of a need to return a sensible default matching the interface? Changing this unconfigured behavior would be a theoretically breaking change, but I'm skeptical about how useful Anonymous Credentials are as-is. > Add other supporting options for configuring credentials for AWS processors > --- > > Key: NIFI-786 > URL: https://issues.apache.org/jira/browse/NIFI-786 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 0.3.0 >Reporter: Michael Kobit >Priority: Minor > > I was looking at https://issues.apache.org/jira/browse/NIFI-770 and looked at > how the AWS processors credentials are currently configured. As a NFM you > have a few options with the properties right now: > 1) set basic, static credentials > 2) set a credentials properties filepath > 3) set neither, use anonymous credentials > I think it would be better if ea
[jira] [Commented] (NIFI-1283) Documentation wrong for ControllerStatusReportingTask
[
https://issues.apache.org/jira/browse/NIFI-1283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15086448#comment-15086448
]
James Wing commented on NIFI-1283:
--
I agree there is a discrepancy between the code and the documentation, but I'm
not sure changing the docs is the right fix. It looks like the
ControllerStatusReportTask declares loggers using the getSimpleName() of the
class rather than using the full class name. This is why the logging
configuration works with "ControllerStatusReportTask" and not with
"org.apache.nifi.controller.ControllerStatusReportingTask".
{code:java|title=ControllerStatusReportTask.java|borderStyle=solid}
private static final Logger processorLogger =
LoggerFactory.getLogger(ControllerStatusReportingTask.class.getSimpleName() +
".Processors");
private static final Logger connectionLogger =
LoggerFactory.getLogger(ControllerStatusReportingTask.class.getSimpleName() +
".Connections");
{code}
However, it appears this choice of logger naming is unique to
ControllerStatusReportTask. All of the other loggers I found in casual code
analysis use the full class name, notably similar reporting tasks like
MonitorDiskUsage and MonitorMemory.
{code:java|title=MonitorDiskUsage.java|borderStyle=solid}
private static final Logger logger =
LoggerFactory.getLogger(MonitorDiskUsage.class);
{code}
{code:java|title=MonitorMemory.java|borderStyle=solid}
private static final Logger logger =
LoggerFactory.getLogger(MonitorMemory.class);
{code}
What do you think about changing the code to use the full class name, matching
the documentation and staying consistent with similar reporting functionality?
I would be willing to make the change.
> Documentation wrong for ControllerStatusReportingTask
> -
>
> Key: NIFI-1283
> URL: https://issues.apache.org/jira/browse/NIFI-1283
> Project: Apache NiFi
> Issue Type: Bug
> Components: Documentation & Website
>Affects Versions: 0.4.0
>Reporter: Matthew Clarke
>Priority: Minor
> Labels: beginner, newbie
>
> The documentation explains how to separate the processor and connection
> logging out to a new log file. The logger names provided do not work.
> Original:
> If may be convenient to redirect the logging output of this ReportingTask to
> a separate log file than the typical application log. This can be
> accomplished by modified the logback.xml file in the NiFi conf/ directory
> such that a logger with the name
> org.apache.nifi.controller.ControllerStatusReportingTask is configured to
> write to a separate log.
> Additionally, it may be convenient to disable logging for Processors or for
> Connections or to split them into separate log files. This can be
> accomplished by using the loggers named
> org.apache.nifi.controller.ControllerStatusReportingTask.Processors and
> org.apache.nifi.controller.ControllerStatusReportingTask.Connections,
> respectively.
> The correct loggers to use would be:
> ControllerStatusReportingTask
> ControllerStatusReportingTask.Processors
> ControllerStatusReportingTask.Connections
> If you include the org.apache.nifi.controller protion the logger will not
> work.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
