[jira] [Commented] (NIFI-2585) Add attributes to track where a flow file came from when receiving over site-to-site
[ https://issues.apache.org/jira/browse/NIFI-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15763071#comment-15763071 ] ASF GitHub Bot commented on NIFI-2585: -- Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/1320 Created another PR #1342, to describe my concerns and possible solutions. > Add attributes to track where a flow file came from when receiving over > site-to-site > > > Key: NIFI-2585 > URL: https://issues.apache.org/jira/browse/NIFI-2585 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Bryan Bende >Assignee: Randy Gelhausen >Priority: Minor > > With MiNiFi starting be used to send data to a central NiFi, it would be > helpful if information about the sending host and port was added to each flow > file received over site-to-site. Currently this information is available and > used to generate the transit URI in the RECEIVE event, but this information > isn't available to downstream processors that might want to make routing > decisions. > For reference: > https://github.com/apache/nifi/blob/e23b2356172e128086585fe2c425523c3628d0e7/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-site-to-site/src/main/java/org/apache/nifi/remote/protocol/AbstractFlowFileServerProtocol.java#L452 > A possible approach might be to add two attributes to each flow file, > something like "remote.host" and "remote.address" where remote.host has only > the sending hostname, and remote.address has the sending host and port. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] nifi issue #1320: NIFI-2585 Add attributes to track s2s host and port
Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/1320 Created another PR #1342, to describe my concerns and possible solutions. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1342: NIFI-2585 Add attributes to track s2s host and port
Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/1342 @bbende @randerzander Sorry for taking long, it took time to confirm results and consider how the result should be.. Please take a look on the table in previous comment, and let me know how you think. I added additional commit to address those concerns. In addition to that, I'd like to have attribute key names like `s2s.remote.address` or `s2s.client.address` and `s2s.server.address` to describe which host the value represents more clearly. Also, I'd like to add some unit test cases to protect it from breaking in the future. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-2585) Add attributes to track where a flow file came from when receiving over site-to-site
[ https://issues.apache.org/jira/browse/NIFI-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15763069#comment-15763069 ] ASF GitHub Bot commented on NIFI-2585: -- Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/1342 @bbende @randerzander Sorry for taking long, it took time to confirm results and consider how the result should be.. Please take a look on the table in previous comment, and let me know how you think. I added additional commit to address those concerns. In addition to that, I'd like to have attribute key names like `s2s.remote.address` or `s2s.client.address` and `s2s.server.address` to describe which host the value represents more clearly. Also, I'd like to add some unit test cases to protect it from breaking in the future. Thanks! > Add attributes to track where a flow file came from when receiving over > site-to-site > > > Key: NIFI-2585 > URL: https://issues.apache.org/jira/browse/NIFI-2585 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Bryan Bende >Assignee: Randy Gelhausen >Priority: Minor > > With MiNiFi starting be used to send data to a central NiFi, it would be > helpful if information about the sending host and port was added to each flow > file received over site-to-site. Currently this information is available and > used to generate the transit URI in the RECEIVE event, but this information > isn't available to downstream processors that might want to make routing > decisions. > For reference: > https://github.com/apache/nifi/blob/e23b2356172e128086585fe2c425523c3628d0e7/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-site-to-site/src/main/java/org/apache/nifi/remote/protocol/AbstractFlowFileServerProtocol.java#L452 > A possible approach might be to add two attributes to each flow file, > something like "remote.host" and "remote.address" where remote.host has only > the sending hostname, and remote.address has the sending host and port. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2585) Add attributes to track where a flow file came from when receiving over site-to-site
[ https://issues.apache.org/jira/browse/NIFI-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15763056#comment-15763056 ] ASF GitHub Bot commented on NIFI-2585: -- GitHub user ijokarumawak opened a pull request: https://github.com/apache/nifi/pull/1342 NIFI-2585 Add attributes to track s2s host and port I left the four commits to show the history of how this evolved. We can squash Bryan's two commits when merging, but we wanted Randy to get credit for starting the work on this ticket. While reviewing #1320, I found few concerns as shown in the table below. Let's say there are `client.nifi` and `server.nifi` transferring files each other using S2S. This table shows how `s2s.address` attribute values are set in each case with #1320: | Transfer Protocol | Pulled from a remote OutputPort to S2S client | Received via an InputPort at S2S server | |---|---|| | RAW | Server's hostname and port are the one that the client gets when it received remote NiFi peers. Those values are defined in server's nifi.properties. `nifi.remote.input.host:nifi.remote.input.socket.port` e.x. `server.nifi:8081` | Client's hostname and port is used. But due to the existing logic which parses URL string and extract hostname and port, if the URL string contains illegal character such as underscore, the result becomes null, then converted with `unknown`. This happens with Docker as container hostname often contain underscore. e.x. `client.nifi:58034`, `unknown:unknown` (if hostname contains underscore) **FIX 1** `client_nifi:58034` (even if hostname contains underscore)| | HTTP | Same as RAW protocol, but uses `nifi.web.http(s).port`. e.x. `server.nifi:8080`| Client hostname and port are retrieved from HTTP request object. However, it returns IP address string representation instead of hostname. Probably performance reason. e.x. `192.168.0.33:59946` **FIX 2** `client.nifi:59946`.| ## FIX 1 As reporte in [the previous PR comment](https://github.com/apache/nifi/pull/1307#issuecomment-266442657) I got 'unknown' hostname and port with Docker containers. While this can be handled as a corner case since it's not allowed by the URL specification, I think it'd be better if we can be lenient here to support Docker environment. I confirmed provenance event also failed to resolve hostname and produced provenance details with null hostname. This can be improved by changing Peer's constructor. Currently, it parses peerUrl then use its hostname and port, but the same information should be retrieved from PeerDescriptor without parsing the URL. Also, SocketRemoteSiteListener uses server's hostname and port for PeerDescription, but it seems it's not correct, those should be client's. This commit modifies SocketRemoteSiteListener to use PeerDescriptor instead of parsing URL string. ## FIX 2 This commit modifies DataTransferResource to resolve hostname from IP address using InetAddress when HTTP transport protocol is used, as RAW resolves hostname from socket, using InetAddress. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ijokarumawak/nifi nifi-2585 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1342.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1342 commit c6f0621f985b6deddadc616307f192c1ddf1060d Author: Randy Gelhausen Date: 2016-12-07T07:18:09Z NIFI-2585: Add attributes to track where a flow file came from when receiving over site-to-site commit 841c4c80e42f09e7dc96552c661dde5a81ab7c08 Author: Bryan Bende Date: 2016-12-08T20:17:06Z NIFI-2585 Moving attributes into loop in AbstractFlowFileServerProtocol, and also updating StandardRemoteGroupPort to apply the same attributes when doing a pull-based site-to-site. commit 0e3a29984d1abf94ba86d98fbe283e9671de1671 Author: Bryan Bende Date: 2016-12-12T14:19:22Z NIFI-2585 Adding checks in case host and port are not known commit c206f266c38e537c9434d7ff66ad5e19ceb01e1c Author: Koji Kawamura Date: 2016-12-20T02:19:22Z NIFI-2585: Add attributes to track s2s host and port - Removed host and port field from Peer since the same information is available in PeerDescription - Refactored variable names in SocketRemoteSiteListener to improve readability - Changed how SocketRemoteSiteListener constructs PeerDescription instance. It used to use hard-coded 'localhost' as hostname, and getPort() which returns server's port. Since the peer is a remote peer, i.e the client, it should be
[GitHub] nifi pull request #1342: NIFI-2585 Add attributes to track s2s host and port
GitHub user ijokarumawak opened a pull request: https://github.com/apache/nifi/pull/1342 NIFI-2585 Add attributes to track s2s host and port I left the four commits to show the history of how this evolved. We can squash Bryan's two commits when merging, but we wanted Randy to get credit for starting the work on this ticket. While reviewing #1320, I found few concerns as shown in the table below. Let's say there are `client.nifi` and `server.nifi` transferring files each other using S2S. This table shows how `s2s.address` attribute values are set in each case with #1320: | Transfer Protocol | Pulled from a remote OutputPort to S2S client | Received via an InputPort at S2S server | |---|---|| | RAW | Server's hostname and port are the one that the client gets when it received remote NiFi peers. Those values are defined in server's nifi.properties. `nifi.remote.input.host:nifi.remote.input.socket.port` e.x. `server.nifi:8081` | Client's hostname and port is used. But due to the existing logic which parses URL string and extract hostname and port, if the URL string contains illegal character such as underscore, the result becomes null, then converted with `unknown`. This happens with Docker as container hostname often contain underscore. e.x. `client.nifi:58034`, `unknown:unknown` (if hostname contains underscore) **FIX 1** `client_nifi:58034` (even if hostname contains underscore)| | HTTP | Same as RAW protocol, but uses `nifi.web.http(s).port`. e.x. `server.nifi:8080`| Client hostname and port are retrieved from HTTP request object. However, it returns IP address string representation instead of hostname. Probably performance reason. e.x. `192.168.0.33:59946` **FIX 2** `client.nifi:59946`.| ## FIX 1 As reporte in [the previous PR comment](https://github.com/apache/nifi/pull/1307#issuecomment-266442657) I got 'unknown' hostname and port with Docker containers. While this can be handled as a corner case since it's not allowed by the URL specification, I think it'd be better if we can be lenient here to support Docker environment. I confirmed provenance event also failed to resolve hostname and produced provenance details with null hostname. This can be improved by changing Peer's constructor. Currently, it parses peerUrl then use its hostname and port, but the same information should be retrieved from PeerDescriptor without parsing the URL. Also, SocketRemoteSiteListener uses server's hostname and port for PeerDescription, but it seems it's not correct, those should be client's. This commit modifies SocketRemoteSiteListener to use PeerDescriptor instead of parsing URL string. ## FIX 2 This commit modifies DataTransferResource to resolve hostname from IP address using InetAddress when HTTP transport protocol is used, as RAW resolves hostname from socket, using InetAddress. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ijokarumawak/nifi nifi-2585 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1342.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1342 commit c6f0621f985b6deddadc616307f192c1ddf1060d Author: Randy Gelhausen Date: 2016-12-07T07:18:09Z NIFI-2585: Add attributes to track where a flow file came from when receiving over site-to-site commit 841c4c80e42f09e7dc96552c661dde5a81ab7c08 Author: Bryan Bende Date: 2016-12-08T20:17:06Z NIFI-2585 Moving attributes into loop in AbstractFlowFileServerProtocol, and also updating StandardRemoteGroupPort to apply the same attributes when doing a pull-based site-to-site. commit 0e3a29984d1abf94ba86d98fbe283e9671de1671 Author: Bryan Bende Date: 2016-12-12T14:19:22Z NIFI-2585 Adding checks in case host and port are not known commit c206f266c38e537c9434d7ff66ad5e19ceb01e1c Author: Koji Kawamura Date: 2016-12-20T02:19:22Z NIFI-2585: Add attributes to track s2s host and port - Removed host and port field from Peer since the same information is available in PeerDescription - Refactored variable names in SocketRemoteSiteListener to improve readability - Changed how SocketRemoteSiteListener constructs PeerDescription instance. It used to use hard-coded 'localhost' as hostname, and getPort() which returns server's port. Since the peer is a remote peer, i.e the client, it should be client hostname and port. - Added hostname resolution at DataTransferResource to make s2s.host value consistent with RAW transport. Without this, RAW uses hostname while HTTP uses IP address. It will be hard to be used from downstream
[jira] [Commented] (NIFI-3230) Get/Put JMS broken for simple ActiveMQ SSL URIs
[ https://issues.apache.org/jira/browse/NIFI-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15762478#comment-15762478 ] Michael Moser commented on NIFI-3230: - I have a PR with a fix that I will submit soon. > Get/Put JMS broken for simple ActiveMQ SSL URIs > --- > > Key: NIFI-3230 > URL: https://issues.apache.org/jira/browse/NIFI-3230 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 0.5.1, 0.6.1, 1.1.0, 0.7.1 >Reporter: Michael Moser >Assignee: Michael Moser > > The GetJMS* and PutJMS processors don't work with ActiveMQ SSL using the > simple ssl://127.0.0.1:65189 URI. It does work, though, with composite URIs > like failover:(ssl://127.0.0.1:65189,ssl://127.0.0.1:65188)?randomize=false > Even though we want people to use ConsumeJMS and PublishJMS, we should fix > the GetJMS* and PutJMS processors because they are still in the baseline. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-3230) Get/Put JMS broken for simple ActiveMQ SSL URIs
[ https://issues.apache.org/jira/browse/NIFI-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Moser updated NIFI-3230: Affects Version/s: 0.5.1 0.6.1 > Get/Put JMS broken for simple ActiveMQ SSL URIs > --- > > Key: NIFI-3230 > URL: https://issues.apache.org/jira/browse/NIFI-3230 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 0.5.1, 0.6.1, 1.1.0, 0.7.1 >Reporter: Michael Moser >Assignee: Michael Moser > > The GetJMS* and PutJMS processors don't work with ActiveMQ SSL using the > simple ssl://127.0.0.1:65189 URI. It does work, though, with composite URIs > like failover:(ssl://127.0.0.1:65189,ssl://127.0.0.1:65188)?randomize=false > Even though we want people to use ConsumeJMS and PublishJMS, we should fix > the GetJMS* and PutJMS processors because they are still in the baseline. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-3230) Get/Put JMS broken for simple ActiveMQ SSL URIs
Michael Moser created NIFI-3230: --- Summary: Get/Put JMS broken for simple ActiveMQ SSL URIs Key: NIFI-3230 URL: https://issues.apache.org/jira/browse/NIFI-3230 Project: Apache NiFi Issue Type: Bug Components: Extensions Affects Versions: 0.7.1, 1.1.0 Reporter: Michael Moser Assignee: Michael Moser The GetJMS* and PutJMS processors don't work with ActiveMQ SSL using the simple ssl://127.0.0.1:65189 URI. It does work, though, with composite URIs like failover:(ssl://127.0.0.1:65189,ssl://127.0.0.1:65188)?randomize=false Even though we want people to use ConsumeJMS and PublishJMS, we should fix the GetJMS* and PutJMS processors because they are still in the baseline. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-3225) Abstract Processor type that batches session.get() and session.commit() calls
[ https://issues.apache.org/jira/browse/NIFI-3225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15762429#comment-15762429 ] Mark Payne commented on NIFI-3225: -- [~bryanrosan...@gmail.com] - I'd be very hesitant to modify this. The changes you describe here would have the same affect as the current model if we always call session.commit(). However, any time that the session is rolled back, it would behave a bit differently. Namely, we'd roll back more than we intended because session.checkpoint() hadn't been called. The idea behind the session is that if we call session.rollback(), it should rollback everything since the last time session.commit() or session.rollback() was called but no more. Changing how frequently checkpoint() is called would change that. I'd also recommend digging into this with a profiler, such as VisualVM and/or YourKit. Because checkpoint() is going to be called for every flowfile in the case that you describe, it is reasonable to expect it to show up very frequently when using a sampler instead of a profiler. As you said, session.checkpoint() is quite cheap, but not free. I think it's required for correctness, though. We may be able to improve its efficiency somewhat if its performance is concerning. The majority of its work is transferring data from one HashMap to another. We may be able to use some other type of data structure to be more efficient here, or perhaps tune the HashMap with non-default values for the constructor arguments. > Abstract Processor type that batches session.get() and session.commit() calls > - > > Key: NIFI-3225 > URL: https://issues.apache.org/jira/browse/NIFI-3225 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Bryan Rosander >Assignee: Bryan Rosander >Priority: Minor > Attachments: after.png, before.png > > > For processors that are stateless and support batching, it should be safe to > get and process multiple input FlowFiles for each onTrigger() call. > This should amortize the cost of session.get(), session.checkpoint(), > session.commit() as well as any setup in onTrigger() that isn't dependent on > the FlowFile(s) attributes or content. > An AbstractBatchingProcessor type should reduce boilerplate code in candidate > processors and encourage uniform configurability via a property to control > batch size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-3203) Perform nifi 1.1.1 release duties
[ https://issues.apache.org/jira/browse/NIFI-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15762414#comment-15762414 ] ASF subversion and git services commented on NIFI-3203: --- Commit a92f2e36ed6be695e4dc6f624f6b3a96e6d1a57c in nifi's branch refs/heads/NIFI-3203-rc1 from [~JPercivall] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=a92f2e3 ] NIFI-3203-rc1 prepare release nifi-1.1.1-RC1 > Perform nifi 1.1.1 release duties > - > > Key: NIFI-3203 > URL: https://issues.apache.org/jira/browse/NIFI-3203 > Project: Apache NiFi > Issue Type: Task > Components: Tools and Build >Reporter: Joseph Witt >Assignee: Joseph Percivall > Fix For: 1.1.1 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-3203) Perform nifi 1.1.1 release duties
[ https://issues.apache.org/jira/browse/NIFI-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15762415#comment-15762415 ] ASF subversion and git services commented on NIFI-3203: --- Commit d02d9d30b13dc5d0d014d6797b330a7804f921b4 in nifi's branch refs/heads/NIFI-3203-rc1 from [~JPercivall] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=d02d9d3 ] NIFI-3203-rc1 prepare for next development iteration > Perform nifi 1.1.1 release duties > - > > Key: NIFI-3203 > URL: https://issues.apache.org/jira/browse/NIFI-3203 > Project: Apache NiFi > Issue Type: Task > Components: Tools and Build >Reporter: Joseph Witt >Assignee: Joseph Percivall > Fix For: 1.1.1 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-3153) Update AWS SDK
[ https://issues.apache.org/jira/browse/NIFI-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15762367#comment-15762367 ] ASF GitHub Bot commented on NIFI-3153: -- GitHub user jvwing opened a pull request: https://github.com/apache/nifi/pull/1341 NIFI-3153 Update AWS SDK As part of upgrading the AWS SDK, I ran the suite of AWS integration tests. Getting the integration tests to succeed required a couple of changes detailed below, none of which I believe to be caused by the SDK change: * ITPutS3Object.testGetPropertyDescriptors - bumped up the expected number of Property Descriptors to match the actual number. * ITPutS3Object.testContentType - Ignored. The test checks if the content type was written to the flowfile attributes after writing the content to S3. I did not find any place in the PutS3Object code that attempts this, uses the S3_CONTENT_TYPE constant, or that was submitted with NIFI-2810. I created a separate ticket to investigate, NIFI-3228. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [X] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jvwing/nifi NIFI-3153-update-aws-sdk-1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1341.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1341 commit eb75984af8d1185bab26c12b70da8f8a2568f313 Author: James Wing Date: 2016-12-17T21:30:27Z NIFI-3153 Updating AWS SDK Version > Update AWS SDK > -- > > Key: NIFI-3153 > URL: https://issues.apache.org/jira/browse/NIFI-3153 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: James Wing >Assignee: James Wing >Priority: Minor > > I propose to update NiFi's AWS SDK to v1.11.68 (December 16, 2016) or later > to get support for recent AWS updates: > * New Regions - ap-south-1 (Mumbai), ca-central-1 (Canada), eu-west-2 > (London), us-east-2 (Ohio) > * New AWS services > NiFi's current SDK is v1.11.8 (June 16, 2016). I looked through the [AWS SDK > for Java release notes|https://aws.amazon.com/releasenotes/Java?browse=1] to > search for upgrade complications. Most of the changes look fine -- mostly > new services, regions, and simple bug fixes. I found one item of interest: > * New [Retry > Throttling|https://aws.amazon.com/blogs/developer/introducing-retry-throttling/] > feature has been made the default rather than optional starting in > [v1.11.12|https://aws.amazon.com/releasenotes/Java/2878241577606051]. It > looks very sensible to not flog the system pointlessly if requests are > already failing, and I expect it to be a benefit for most NiFi/AWS use > cases. However, it is the kind of sneaky behavioral change I was looking for. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] nifi pull request #1341: NIFI-3153 Update AWS SDK
GitHub user jvwing opened a pull request: https://github.com/apache/nifi/pull/1341 NIFI-3153 Update AWS SDK As part of upgrading the AWS SDK, I ran the suite of AWS integration tests. Getting the integration tests to succeed required a couple of changes detailed below, none of which I believe to be caused by the SDK change: * ITPutS3Object.testGetPropertyDescriptors - bumped up the expected number of Property Descriptors to match the actual number. * ITPutS3Object.testContentType - Ignored. The test checks if the content type was written to the flowfile attributes after writing the content to S3. I did not find any place in the PutS3Object code that attempts this, uses the S3_CONTENT_TYPE constant, or that was submitted with NIFI-2810. I created a separate ticket to investigate, NIFI-3228. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [X] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jvwing/nifi NIFI-3153-update-aws-sdk-1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1341.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1341 commit eb75984af8d1185bab26c12b70da8f8a2568f313 Author: James Wing Date: 2016-12-17T21:30:27Z NIFI-3153 Updating AWS SDK Version --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (NIFI-2967) Disable "Add" button on new processor dialog if no processors match search query
[ https://issues.apache.org/jira/browse/NIFI-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman updated NIFI-2967: -- Status: Patch Available (was: In Progress) > Disable "Add" button on new processor dialog if no processors match search > query > > > Key: NIFI-2967 > URL: https://issues.apache.org/jira/browse/NIFI-2967 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Affects Versions: 1.0.0 >Reporter: Andy LoPresto >Assignee: Matt Gilman > Labels: beginner, ui, ux, validation > Fix For: 1.2.0 > > Attachments: Screen Shot 2016-10-28 at 9.04.06 PM.png, Screen Shot > 2016-10-28 at 9.04.22 PM.png > > Original Estimate: 1h > Remaining Estimate: 1h > > When adding a processor, if the search term does not match any available > processors, the "Add" button should be disabled. If no processors are > available, the user can still click the "Add" button and then receives an > error message stating "The type of processor to create must be selected." > To reproduce: > # Drag a new processor onto the canvas > # Type "sdfsdf" into the search field > # Click "Add" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2967) Disable "Add" button on new processor dialog if no processors match search query
[ https://issues.apache.org/jira/browse/NIFI-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15762293#comment-15762293 ] ASF GitHub Bot commented on NIFI-2967: -- GitHub user mcgilman opened a pull request: https://github.com/apache/nifi/pull/1340 NIFI-2967: Disabling Add button in new component dialog when appropriate NIFI-2967: - Disabling the Add button in the new Processor, Controller Service, and Reporting Task dialog when no components match the filter. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mcgilman/nifi NIFI-2967 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1340.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1340 commit fb859a454d37727438c4c26ef26c49eb4faadc03 Author: Matt Gilman Date: 2016-12-19T20:59:39Z NIFI-2967: - Disabling the Add button in the new Processor, Controller Service, and Reporting Task dialog when no components match the filter. > Disable "Add" button on new processor dialog if no processors match search > query > > > Key: NIFI-2967 > URL: https://issues.apache.org/jira/browse/NIFI-2967 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Affects Versions: 1.0.0 >Reporter: Andy LoPresto >Assignee: Matt Gilman > Labels: beginner, ui, ux, validation > Fix For: 1.2.0 > > Attachments: Screen Shot 2016-10-28 at 9.04.06 PM.png, Screen Shot > 2016-10-28 at 9.04.22 PM.png > > Original Estimate: 1h > Remaining Estimate: 1h > > When adding a processor, if the search term does not match any available > processors, the "Add" button should be disabled. If no processors are > available, the user can still click the "Add" button and then receives an > error message stating "The type of processor to create must be selected." > To reproduce: > # Drag a new processor onto the canvas > # Type "sdfsdf" into the search field > # Click "Add" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] nifi pull request #1340: NIFI-2967: Disabling Add button in new component di...
GitHub user mcgilman opened a pull request: https://github.com/apache/nifi/pull/1340 NIFI-2967: Disabling Add button in new component dialog when appropriate NIFI-2967: - Disabling the Add button in the new Processor, Controller Service, and Reporting Task dialog when no components match the filter. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mcgilman/nifi NIFI-2967 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1340.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1340 commit fb859a454d37727438c4c26ef26c49eb4faadc03 Author: Matt Gilman Date: 2016-12-19T20:59:39Z NIFI-2967: - Disabling the Add button in the new Processor, Controller Service, and Reporting Task dialog when no components match the filter. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3071) Consider deprecating org.apache.nifi.stream.io.ByteArrayOutputStream in favor of jaba.io.ByteArrayOutputStream
[ https://issues.apache.org/jira/browse/NIFI-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15762269#comment-15762269 ] ASF subversion and git services commented on NIFI-3071: --- Commit 9153fca2172c7a8d9b97a2c848d6a0529f1ad13c in nifi's branch refs/heads/support/nifi-1.1.x from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=9153fca ] NIFI-3071: Deprecated InputStreams & OutputStream sin org.apache.nifi.stream.io package in favor of using their Java counterparts > Consider deprecating org.apache.nifi.stream.io.ByteArrayOutputStream in favor > of jaba.io.ByteArrayOutputStream > -- > > Key: NIFI-3071 > URL: https://issues.apache.org/jira/browse/NIFI-3071 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.0.0 >Reporter: Oleg Zhurakousky >Assignee: Mark Payne >Priority: Blocker > Fix For: 1.2.0, 1.1.1 > > > I believe the "efficiency" statements need to be revisited. For example, > preliminary testing shows that there are no performance difference between > using the one provided with java.io. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-3188) Test corner cases of StreamDemarcator
[ https://issues.apache.org/jira/browse/NIFI-3188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15762270#comment-15762270 ] ASF subversion and git services commented on NIFI-3188: --- Commit 986e716091ade2dec29de579f8be4aa4df0efc4c in nifi's branch refs/heads/support/nifi-1.1.x from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=986e716 ] NIFI-3188: Added unit test to test corner cases This closes #1321. Signed-off-by: Koji Kawamura > Test corner cases of StreamDemarcator > - > > Key: NIFI-3188 > URL: https://issues.apache.org/jira/browse/NIFI-3188 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.2.0, 1.1.1 > > > I've found a corner case that existed in StreamDemarcator in the 1.0.0 > baseline. The corner case has been addressed already in the 1.1.0 baseline. > However, this ticket is to add additional unit tests for StreamDemarcator > that expose the issue in 1.0.0 and show that it is fixed in 1.1.0. This will > be helpful to guard against regressions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-3090) Should add new flow election cluster properties to Admin Guide property tables
[ https://issues.apache.org/jira/browse/NIFI-3090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15762267#comment-15762267 ] ASF subversion and git services commented on NIFI-3090: --- Commit a71d73b75cd803a08ac091ddbede0058fdc184cb in nifi's branch refs/heads/support/nifi-1.1.x from [~andrewmlim] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=a71d73b ] NIFI-3090 Added new flow election cluster properties to Admin Guide property tables This closes #1313. Signed-off-by: Bryan Bende > Should add new flow election cluster properties to Admin Guide property tables > -- > > Key: NIFI-3090 > URL: https://issues.apache.org/jira/browse/NIFI-3090 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation & Website >Affects Versions: 1.1.0 >Reporter: Andrew Lim >Assignee: Andrew Lim >Priority: Minor > Fix For: 1.2.0, 1.1.1 > > > https://issues.apache.org/jira/browse/NIFI-1966 added two new properties: > nifi.cluster.flow.election.max.wait.time > nifi.cluster.flow.election.max.candidates > Documentation was added within the Admin Guide "Flow Election" section, but > these properties should also be added to the "Cluster Node Properties" table > at the end of the doc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-3178) Missing images in User Admin Guide for Operate Palette buttons
[ https://issues.apache.org/jira/browse/NIFI-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15762263#comment-15762263 ] ASF subversion and git services commented on NIFI-3178: --- Commit ed202b6d2a8a597e46836a5a9ae04a6fe71c8056 in nifi's branch refs/heads/support/nifi-1.1.x from [~andrewmlim] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=ed202b6 ] NIFI-3178 Corrected missing Operate Palette button images in User Admin Guide This closes #1314. Signed-off-by: Bryan Bende > Missing images in User Admin Guide for Operate Palette buttons > -- > > Key: NIFI-3178 > URL: https://issues.apache.org/jira/browse/NIFI-3178 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Andrew Lim >Assignee: Andrew Lim >Priority: Minor > Fix For: 1.2.0, 1.1.1 > > > In my PR for NIFI-3143, I updated the User Guide to reference the following > new images: > buttonStart.png > buttonStop.png > buttonDisable.png > buttonEnable.png > However, these images were mistakenly left out of the PR. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-3145) Re-implement NumberParsing double expression
[ https://issues.apache.org/jira/browse/NIFI-3145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15762265#comment-15762265 ] ASF subversion and git services commented on NIFI-3145: --- Commit cd6ed27c86e58d15933cecec81284043be8c in nifi's branch refs/heads/support/nifi-1.1.x from jpercivall [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=cd6ed21 ] NIFI-3145 Rewriting double validation in NumberParsing Adding more tests to TestQuery NIFI-3145 Adding logic to handle lowercase hex values This closes #1296 > Re-implement NumberParsing double expression > > > Key: NIFI-3145 > URL: https://issues.apache.org/jira/browse/NIFI-3145 > Project: Apache NiFi > Issue Type: Bug >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Blocker > Fix For: 1.2.0, 1.1.1 > > > It leverages sample code from the docs here > http://docs.oracle.com/javase/6/docs/api/java/lang/Double.html#valueOf%28java.lang.String%29 > and it is unclear what terms those are under. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-3145) Re-implement NumberParsing double expression
[ https://issues.apache.org/jira/browse/NIFI-3145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15762266#comment-15762266 ] ASF subversion and git services commented on NIFI-3145: --- Commit cd6ed27c86e58d15933cecec81284043be8c in nifi's branch refs/heads/support/nifi-1.1.x from jpercivall [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=cd6ed21 ] NIFI-3145 Rewriting double validation in NumberParsing Adding more tests to TestQuery NIFI-3145 Adding logic to handle lowercase hex values This closes #1296 > Re-implement NumberParsing double expression > > > Key: NIFI-3145 > URL: https://issues.apache.org/jira/browse/NIFI-3145 > Project: Apache NiFi > Issue Type: Bug >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Blocker > Fix For: 1.2.0, 1.1.1 > > > It leverages sample code from the docs here > http://docs.oracle.com/javase/6/docs/api/java/lang/Double.html#valueOf%28java.lang.String%29 > and it is unclear what terms those are under. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-3152) If Provenance Repository runs out of disk space, it may not recover even when disk space is freed up
[
https://issues.apache.org/jira/browse/NIFI-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15762264#comment-15762264
]
ASF subversion and git services commented on NIFI-3152:
---
Commit 070776ebaef9b5c4a0632dd79fdf32617c879f40 in nifi's branch
refs/heads/support/nifi-1.1.x from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=070776e ]
NIFI-3152: Ensure that we always close the IndexWriter when appropriate in
SimpleIndexManager, even if an IOException is thrown when trying to commit the
IndexWriter
This closes #1300.
Signed-off-by: Bryan Bende
> If Provenance Repository runs out of disk space, it may not recover even when
> disk space is freed up
>
>
> Key: NIFI-3152
> URL: https://issues.apache.org/jira/browse/NIFI-3152
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.2.0, 1.1.1
>
>
> If we run out of disk space in the provenance repository, we can sometimes
> get into a situation where the logs show us still waiting for the repo to
> roll over, even after disk space is freed up. A thread dump shows that the
> processors are trying to force the repo to rollover. However, the rollover
> never completes because we can't create an IndexWriter:
> {code}
> "Provenance Repository Rollover Thread-1" Id=128 TIMED_WAITING on null
> at java.lang.Thread.sleep(Native Method)
> at org.apache.lucene.store.Lock.obtain(Lock.java:92)
> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
> at
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:104)
> - waiting on
> org.apache.nifi.provenance.lucene.SimpleIndexManager@22f9da45
> at
> org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1711)
> at
> org.apache.nifi.provenance.PersistentProvenanceRepository$8.run(PersistentProvenanceRepository.java:1311)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Number of Locked Synchronizers: 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@850f87f
> {code}
> The Index Writer is blocking on a lock, waiting to obtain a write lock for
> the Directory.
> Digging around, I believe the issue is that if we call
> SimpleIndexManager.returnIndexWriter, it will call IndexWriter.commit(). But
> if that throws an Exception, we don't properly close the writer. If we are
> running out of disk space, it is likely that we will throw an Exception on
> IndexWriter.commit() so this appears to be the root cause.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-3229) when flowfile penalized the next processor `Tasks/Time` statistics becomes extreamly large
[ https://issues.apache.org/jira/browse/NIFI-3229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Lukyanov updated NIFI-3229: -- Description: fetchfile on `not.found` produces penalized flow file in this case i'm expecting the next processor will do one task execution when flow file penalize time over. but according to stats it executes approximately 1-6 times. i understand that it could be a feature but stats became really unclear... maybe there should be two columns? `All Task/Times` and `Committed Task/Times` was: fetchfile on `not.found` produces penalized flow file in this case i'm expecting the next processor will do one task execution when flow file penalize time over. but according to stats it executes approximately 1-6 times. i understand that it could be a feature but stats became really unclear... maybe there should be two columns? > when flowfile penalized the next processor `Tasks/Time` statistics becomes > extreamly large > -- > > Key: NIFI-3229 > URL: https://issues.apache.org/jira/browse/NIFI-3229 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Dmitry Lukyanov >Priority: Minor > > fetchfile on `not.found` produces penalized flow file > in this case i'm expecting the next processor will do one task execution when > flow file penalize time over. > but according to stats it executes approximately 1-6 times. > i understand that it could be a feature but stats became really unclear... > maybe there should be two columns? > `All Task/Times` and `Committed Task/Times` -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-3229) when flowfile penalized the next processor `Tasks/Time` statistics becomes extreamly large
[ https://issues.apache.org/jira/browse/NIFI-3229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Lukyanov updated NIFI-3229: -- Attachment: flow.xml.gz nifi-stats2.png nifi-stats.png flow file and screenshots attached for one execution > when flowfile penalized the next processor `Tasks/Time` statistics becomes > extreamly large > -- > > Key: NIFI-3229 > URL: https://issues.apache.org/jira/browse/NIFI-3229 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Dmitry Lukyanov >Priority: Minor > Attachments: flow.xml.gz, nifi-stats.png, nifi-stats2.png > > > fetchfile on `not.found` produces penalized flow file > in this case i'm expecting the next processor will do one task execution when > flow file penalize time over. > but according to stats it executes approximately 1-6 times. > i understand that it could be a feature but stats became really unclear... > maybe there should be two columns? > `All Task/Times` and `Committed Task/Times` -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-3188) Test corner cases of StreamDemarcator
[ https://issues.apache.org/jira/browse/NIFI-3188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall updated NIFI-3188: --- Fix Version/s: 1.2.0 > Test corner cases of StreamDemarcator > - > > Key: NIFI-3188 > URL: https://issues.apache.org/jira/browse/NIFI-3188 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.2.0, 1.1.1 > > > I've found a corner case that existed in StreamDemarcator in the 1.0.0 > baseline. The corner case has been addressed already in the 1.1.0 baseline. > However, this ticket is to add additional unit tests for StreamDemarcator > that expose the issue in 1.0.0 and show that it is fixed in 1.1.0. This will > be helpful to guard against regressions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (NIFI-2967) Disable "Add" button on new processor dialog if no processors match search query
[ https://issues.apache.org/jira/browse/NIFI-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman reassigned NIFI-2967: - Assignee: Matt Gilman > Disable "Add" button on new processor dialog if no processors match search > query > > > Key: NIFI-2967 > URL: https://issues.apache.org/jira/browse/NIFI-2967 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Affects Versions: 1.0.0 >Reporter: Andy LoPresto >Assignee: Matt Gilman > Labels: beginner, ui, ux, validation > Fix For: 1.2.0 > > Attachments: Screen Shot 2016-10-28 at 9.04.06 PM.png, Screen Shot > 2016-10-28 at 9.04.22 PM.png > > Original Estimate: 1h > Remaining Estimate: 1h > > When adding a processor, if the search term does not match any available > processors, the "Add" button should be disabled. If no processors are > available, the user can still click the "Add" button and then receives an > error message stating "The type of processor to create must be selected." > To reproduce: > # Drag a new processor onto the canvas > # Type "sdfsdf" into the search field > # Click "Add" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-3229) when flowfile penalized the next processor `Tasks/Time` statistics becomes extreamly large
[ https://issues.apache.org/jira/browse/NIFI-3229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Lukyanov updated NIFI-3229: -- Description: fetchfile on `not.found` produces penalized flow file in this case i'm expecting the next processor will do one task execution when flow file penalize time over. but according to stats it executes approximately 1-6 times. i understand that it could be a feature but stats became really unclear... maybe there should be two columns? > when flowfile penalized the next processor `Tasks/Time` statistics becomes > extreamly large > -- > > Key: NIFI-3229 > URL: https://issues.apache.org/jira/browse/NIFI-3229 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Dmitry Lukyanov >Priority: Minor > > fetchfile on `not.found` produces penalized flow file > in this case i'm expecting the next processor will do one task execution when > flow file penalize time over. > but according to stats it executes approximately 1-6 times. > i understand that it could be a feature but stats became really unclear... > maybe there should be two columns? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-3071) Consider deprecating org.apache.nifi.stream.io.ByteArrayOutputStream in favor of jaba.io.ByteArrayOutputStream
[ https://issues.apache.org/jira/browse/NIFI-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall updated NIFI-3071: --- Fix Version/s: 1.2.0 > Consider deprecating org.apache.nifi.stream.io.ByteArrayOutputStream in favor > of jaba.io.ByteArrayOutputStream > -- > > Key: NIFI-3071 > URL: https://issues.apache.org/jira/browse/NIFI-3071 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.0.0 >Reporter: Oleg Zhurakousky >Assignee: Mark Payne >Priority: Blocker > Fix For: 1.2.0, 1.1.1 > > > I believe the "efficiency" statements need to be revisited. For example, > preliminary testing shows that there are no performance difference between > using the one provided with java.io. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2967) Disable "Add" button on new processor dialog if no processors match search query
[ https://issues.apache.org/jira/browse/NIFI-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman updated NIFI-2967: -- Fix Version/s: 1.2.0 > Disable "Add" button on new processor dialog if no processors match search > query > > > Key: NIFI-2967 > URL: https://issues.apache.org/jira/browse/NIFI-2967 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Affects Versions: 1.0.0 >Reporter: Andy LoPresto > Labels: beginner, ui, ux, validation > Fix For: 1.2.0 > > Attachments: Screen Shot 2016-10-28 at 9.04.06 PM.png, Screen Shot > 2016-10-28 at 9.04.22 PM.png > > Original Estimate: 1h > Remaining Estimate: 1h > > When adding a processor, if the search term does not match any available > processors, the "Add" button should be disabled. If no processors are > available, the user can still click the "Add" button and then receives an > error message stating "The type of processor to create must be selected." > To reproduce: > # Drag a new processor onto the canvas > # Type "sdfsdf" into the search field > # Click "Add" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-3145) Re-implement NumberParsing double expression
[ https://issues.apache.org/jira/browse/NIFI-3145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall updated NIFI-3145: --- Fix Version/s: 1.2.0 > Re-implement NumberParsing double expression > > > Key: NIFI-3145 > URL: https://issues.apache.org/jira/browse/NIFI-3145 > Project: Apache NiFi > Issue Type: Bug >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Blocker > Fix For: 1.2.0, 1.1.1 > > > It leverages sample code from the docs here > http://docs.oracle.com/javase/6/docs/api/java/lang/Double.html#valueOf%28java.lang.String%29 > and it is unclear what terms those are under. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-3152) If Provenance Repository runs out of disk space, it may not recover even when disk space is freed up
[
https://issues.apache.org/jira/browse/NIFI-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Percivall updated NIFI-3152:
---
Fix Version/s: 1.2.0
> If Provenance Repository runs out of disk space, it may not recover even when
> disk space is freed up
>
>
> Key: NIFI-3152
> URL: https://issues.apache.org/jira/browse/NIFI-3152
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.2.0, 1.1.1
>
>
> If we run out of disk space in the provenance repository, we can sometimes
> get into a situation where the logs show us still waiting for the repo to
> roll over, even after disk space is freed up. A thread dump shows that the
> processors are trying to force the repo to rollover. However, the rollover
> never completes because we can't create an IndexWriter:
> {code}
> "Provenance Repository Rollover Thread-1" Id=128 TIMED_WAITING on null
> at java.lang.Thread.sleep(Native Method)
> at org.apache.lucene.store.Lock.obtain(Lock.java:92)
> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
> at
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:104)
> - waiting on
> org.apache.nifi.provenance.lucene.SimpleIndexManager@22f9da45
> at
> org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1711)
> at
> org.apache.nifi.provenance.PersistentProvenanceRepository$8.run(PersistentProvenanceRepository.java:1311)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Number of Locked Synchronizers: 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@850f87f
> {code}
> The Index Writer is blocking on a lock, waiting to obtain a write lock for
> the Directory.
> Digging around, I believe the issue is that if we call
> SimpleIndexManager.returnIndexWriter, it will call IndexWriter.commit(). But
> if that throws an Exception, we don't properly close the writer. If we are
> running out of disk space, it is likely that we will throw an Exception on
> IndexWriter.commit() so this appears to be the root cause.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (NIFI-3229) when flowfile penalized the next processor `Tasks/Time` statistics becomes extreamly large
Dmitry Lukyanov created NIFI-3229: - Summary: when flowfile penalized the next processor `Tasks/Time` statistics becomes extreamly large Key: NIFI-3229 URL: https://issues.apache.org/jira/browse/NIFI-3229 Project: Apache NiFi Issue Type: Improvement Reporter: Dmitry Lukyanov Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-3153) Update AWS SDK
[ https://issues.apache.org/jira/browse/NIFI-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Wing updated NIFI-3153: - Description: I propose to update NiFi's AWS SDK to v1.11.68 (December 16, 2016) or later to get support for recent AWS updates: * New Regions - ap-south-1 (Mumbai), ca-central-1 (Canada), eu-west-2 (London), us-east-2 (Ohio) * New AWS services NiFi's current SDK is v1.11.8 (June 16, 2016). I looked through the [AWS SDK for Java release notes|https://aws.amazon.com/releasenotes/Java?browse=1] to search for upgrade complications. Most of the changes look fine -- mostly new services, regions, and simple bug fixes. I found one item of interest: * New [Retry Throttling|https://aws.amazon.com/blogs/developer/introducing-retry-throttling/] feature has been made the default rather than optional starting in [v1.11.12|https://aws.amazon.com/releasenotes/Java/2878241577606051]. It looks very sensible to not flog the system pointlessly if requests are already failing, and I expect it to be a benefit for most NiFi/AWS use cases. However, it is the kind of sneaky behavioral change I was looking for. was: I propose to update NiFi's AWS SDK to v1.11.63 (December 1, 2016) or later to get support for recent AWS updates: * New region us-east-2 (Ohio) * Updates to S3 (tagging) * New AWS services NiFi's current SDK is v1.11.8 (June 16, 2016). I will update the ticket after analyzing the changes with the expected impact. > Update AWS SDK > -- > > Key: NIFI-3153 > URL: https://issues.apache.org/jira/browse/NIFI-3153 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: James Wing >Assignee: James Wing >Priority: Minor > > I propose to update NiFi's AWS SDK to v1.11.68 (December 16, 2016) or later > to get support for recent AWS updates: > * New Regions - ap-south-1 (Mumbai), ca-central-1 (Canada), eu-west-2 > (London), us-east-2 (Ohio) > * New AWS services > NiFi's current SDK is v1.11.8 (June 16, 2016). I looked through the [AWS SDK > for Java release notes|https://aws.amazon.com/releasenotes/Java?browse=1] to > search for upgrade complications. Most of the changes look fine -- mostly > new services, regions, and simple bug fixes. I found one item of interest: > * New [Retry > Throttling|https://aws.amazon.com/blogs/developer/introducing-retry-throttling/] > feature has been made the default rather than optional starting in > [v1.11.12|https://aws.amazon.com/releasenotes/Java/2878241577606051]. It > looks very sensible to not flog the system pointlessly if requests are > already failing, and I expect it to be a benefit for most NiFi/AWS use > cases. However, it is the kind of sneaky behavioral change I was looking for. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (NIFI-3203) Perform nifi 1.1.1 release duties
[ https://issues.apache.org/jira/browse/NIFI-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall reassigned NIFI-3203: -- Assignee: Joseph Percivall (was: Joseph Witt) > Perform nifi 1.1.1 release duties > - > > Key: NIFI-3203 > URL: https://issues.apache.org/jira/browse/NIFI-3203 > Project: Apache NiFi > Issue Type: Task > Components: Tools and Build >Reporter: Joseph Witt >Assignee: Joseph Percivall > Fix For: 1.1.1 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-3194) PutElasticsearchHttp should route flowfiles to failure on connection errors
[ https://issues.apache.org/jira/browse/NIFI-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall updated NIFI-3194: --- Resolution: Fixed Fix Version/s: 1.1.1 Status: Resolved (was: Patch Available) > PutElasticsearchHttp should route flowfiles to failure on connection errors > --- > > Key: NIFI-3194 > URL: https://issues.apache.org/jira/browse/NIFI-3194 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess > Fix For: 1.2.0, 1.1.1 > > > If PutElasticsearchHttp encounters connection errors with the specified URL > (such as there is no server available at the specified hostname:port), a > ProcessException is thrown and the session is rolled back. > For other processors that accept input flow files and access a URL (such as > PostHttp and InvokeHttp), if such an error occurs, the flow file(s) are > routed to failure. PutElasticsearchHttp should have behavior consistent with > the others, and if connection errors occur, any incoming flow files should be > routed to failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-3194) PutElasticsearchHttp should route flowfiles to failure on connection errors
[ https://issues.apache.org/jira/browse/NIFI-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15762127#comment-15762127 ] ASF subversion and git services commented on NIFI-3194: --- Commit 8e3c4eb9a4246346f9b29b40933e3b7da556eb2c in nifi's branch refs/heads/support/nifi-1.1.x from [~mattyb149] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=8e3c4eb ] NIFI-3194: Fixed error handling in PutElasticsearchHttp Thise closes #1327 Signed-off-by: jpercivall > PutElasticsearchHttp should route flowfiles to failure on connection errors > --- > > Key: NIFI-3194 > URL: https://issues.apache.org/jira/browse/NIFI-3194 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess > Fix For: 1.2.0, 1.1.1 > > > If PutElasticsearchHttp encounters connection errors with the specified URL > (such as there is no server available at the specified hostname:port), a > ProcessException is thrown and the session is rolled back. > For other processors that accept input flow files and access a URL (such as > PostHttp and InvokeHttp), if such an error occurs, the flow file(s) are > routed to failure. PutElasticsearchHttp should have behavior consistent with > the others, and if connection errors occur, any incoming flow files should be > routed to failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-3194) PutElasticsearchHttp should route flowfiles to failure on connection errors
[ https://issues.apache.org/jira/browse/NIFI-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15762125#comment-15762125 ] ASF GitHub Bot commented on NIFI-3194: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/1327 > PutElasticsearchHttp should route flowfiles to failure on connection errors > --- > > Key: NIFI-3194 > URL: https://issues.apache.org/jira/browse/NIFI-3194 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess > Fix For: 1.2.0 > > > If PutElasticsearchHttp encounters connection errors with the specified URL > (such as there is no server available at the specified hostname:port), a > ProcessException is thrown and the session is rolled back. > For other processors that accept input flow files and access a URL (such as > PostHttp and InvokeHttp), if such an error occurs, the flow file(s) are > routed to failure. PutElasticsearchHttp should have behavior consistent with > the others, and if connection errors occur, any incoming flow files should be > routed to failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-3194) PutElasticsearchHttp should route flowfiles to failure on connection errors
[ https://issues.apache.org/jira/browse/NIFI-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15762124#comment-15762124 ] ASF subversion and git services commented on NIFI-3194: --- Commit 3b916353983a547e03ca79f4c7d0f01831c326b4 in nifi's branch refs/heads/master from [~mattyb149] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=3b91635 ] NIFI-3194: Fixed error handling in PutElasticsearchHttp Thise closes #1327 Signed-off-by: jpercivall > PutElasticsearchHttp should route flowfiles to failure on connection errors > --- > > Key: NIFI-3194 > URL: https://issues.apache.org/jira/browse/NIFI-3194 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess > Fix For: 1.2.0 > > > If PutElasticsearchHttp encounters connection errors with the specified URL > (such as there is no server available at the specified hostname:port), a > ProcessException is thrown and the session is rolled back. > For other processors that accept input flow files and access a URL (such as > PostHttp and InvokeHttp), if such an error occurs, the flow file(s) are > routed to failure. PutElasticsearchHttp should have behavior consistent with > the others, and if connection errors occur, any incoming flow files should be > routed to failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] nifi pull request #1327: NIFI-3194: PutElasticsearchHttp should route flowfi...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/1327 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3194) PutElasticsearchHttp should route flowfiles to failure on connection errors
[ https://issues.apache.org/jira/browse/NIFI-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15762117#comment-15762117 ] ASF GitHub Bot commented on NIFI-3194: -- Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/1327 +1 Visually verified code and did a contrib-check build. In a standalone instance, tested many different variations of misconfigured PutES processor hitting ES 2.3.3 and all worked as expected (routing to failure/retry). Thanks @mattyb149 I will merge it in. > PutElasticsearchHttp should route flowfiles to failure on connection errors > --- > > Key: NIFI-3194 > URL: https://issues.apache.org/jira/browse/NIFI-3194 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess > Fix For: 1.2.0 > > > If PutElasticsearchHttp encounters connection errors with the specified URL > (such as there is no server available at the specified hostname:port), a > ProcessException is thrown and the session is rolled back. > For other processors that accept input flow files and access a URL (such as > PostHttp and InvokeHttp), if such an error occurs, the flow file(s) are > routed to failure. PutElasticsearchHttp should have behavior consistent with > the others, and if connection errors occur, any incoming flow files should be > routed to failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] nifi issue #1327: NIFI-3194: PutElasticsearchHttp should route flowfiles to ...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/1327 +1 Visually verified code and did a contrib-check build. In a standalone instance, tested many different variations of misconfigured PutES processor hitting ES 2.3.3 and all worked as expected (routing to failure/retry). Thanks @mattyb149 I will merge it in. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (NIFI-3228) PutS3Object Does Not Write s3.contenttype Attribute
James Wing created NIFI-3228: Summary: PutS3Object Does Not Write s3.contenttype Attribute Key: NIFI-3228 URL: https://issues.apache.org/jira/browse/NIFI-3228 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.0.1, 1.1.0 Reporter: James Wing Priority: Trivial The PutS3Object processor documents that it writes the "s3.contenttype" attribute. There is an integration test for this, ITPutS3Object.testContentType, but it is failing. I did not find code in PutS3Object that attempts to write this attribute. PutS3Object should be fixed to write the attribute, or the documentation and test should be adjusted to match the behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-3225) Abstract Processor type that batches session.get() and session.commit() calls
[ https://issues.apache.org/jira/browse/NIFI-3225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15762053#comment-15762053 ] Bryan Rosander commented on NIFI-3225: -- It's an arbitrary amount of time, sampling is done via thread dump every 50 ms. I've been able to get about 11k/s through my test flow after this and a couple other smaller tweaks. It's about 10k/s max without. > Abstract Processor type that batches session.get() and session.commit() calls > - > > Key: NIFI-3225 > URL: https://issues.apache.org/jira/browse/NIFI-3225 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Bryan Rosander >Assignee: Bryan Rosander >Priority: Minor > Attachments: after.png, before.png > > > For processors that are stateless and support batching, it should be safe to > get and process multiple input FlowFiles for each onTrigger() call. > This should amortize the cost of session.get(), session.checkpoint(), > session.commit() as well as any setup in onTrigger() that isn't dependent on > the FlowFile(s) attributes or content. > An AbstractBatchingProcessor type should reduce boilerplate code in candidate > processors and encourage uniform configurability via a property to control > batch size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (NIFI-3221) ExecuteStreamCommand filters out any double quotes when parsing the "Command Arguments"
[
https://issues.apache.org/jira/browse/NIFI-3221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15762047#comment-15762047
]
Juan C. Sequeiros edited comment on NIFI-3221 at 12/19/16 7:16 PM:
---
If output lets you use a format and I want the format to be like JSON it wont
allow it since I cant use the quotes.
Example: command --format ( { "FIELD 1:" : "%FIELD1", "FIELD 2:" : "%FIELD2" } )
was (Author: digitalplumber):
If output lets you use a format and that format is JSON it wont allow it since
I cant use the quotes.
Example: command --format ( { "FIELD 1:" : "%FIELD1", "FIELD 2:" : "%FIELD2" } )
> ExecuteStreamCommand filters out any double quotes when parsing the "Command
> Arguments"
> ---
>
> Key: NIFI-3221
> URL: https://issues.apache.org/jira/browse/NIFI-3221
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Joseph Percivall
>
> Brought up on the mailing list[1], the ExecuteStreamCommand attempts to
> intelligently parse the "Command Arguments" by automatically detecting double
> quotes (") and treating those within double quotes as a single argument[2].
> This is a nice feature but has the unfortunate side-effect of filtering out
> all double quotes.
> It should instead take into account when the double quote is escaped: \"
> [1] http://mail-archives.apache.org/mod_mbox/nifi-users/201612.mbox/browser
> [2]
> https://github.com/apache/nifi/blob/e12e7a55b75f5e358bdbcea79be9baba77532f94/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/ArgumentUtils.java#L60-L60
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-3221) ExecuteStreamCommand filters out any double quotes when parsing the "Command Arguments"
[
https://issues.apache.org/jira/browse/NIFI-3221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15762047#comment-15762047
]
Juan C. Sequeiros commented on NIFI-3221:
-
If output lets you use a format and that format is JSON it wont allow it since
I cant use the quotes.
Example: command --format ( { "FIELD 1:" : "%FIELD1", "FIELD 2:" : "%FIELD2" } )
> ExecuteStreamCommand filters out any double quotes when parsing the "Command
> Arguments"
> ---
>
> Key: NIFI-3221
> URL: https://issues.apache.org/jira/browse/NIFI-3221
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Joseph Percivall
>
> Brought up on the mailing list[1], the ExecuteStreamCommand attempts to
> intelligently parse the "Command Arguments" by automatically detecting double
> quotes (") and treating those within double quotes as a single argument[2].
> This is a nice feature but has the unfortunate side-effect of filtering out
> all double quotes.
> It should instead take into account when the double quote is escaped: \"
> [1] http://mail-archives.apache.org/mod_mbox/nifi-users/201612.mbox/browser
> [2]
> https://github.com/apache/nifi/blob/e12e7a55b75f5e358bdbcea79be9baba77532f94/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/ArgumentUtils.java#L60-L60
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-3225) Abstract Processor type that batches session.get() and session.commit() calls
[ https://issues.apache.org/jira/browse/NIFI-3225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15762017#comment-15762017 ] Joseph Percivall commented on NIFI-3225: These before/after images, do they run the same amount of data or is just a sample of an arbitrary amount of time/data and just comparing the number of samples in checkpoint seen? > Abstract Processor type that batches session.get() and session.commit() calls > - > > Key: NIFI-3225 > URL: https://issues.apache.org/jira/browse/NIFI-3225 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Bryan Rosander >Assignee: Bryan Rosander >Priority: Minor > Attachments: after.png, before.png > > > For processors that are stateless and support batching, it should be safe to > get and process multiple input FlowFiles for each onTrigger() call. > This should amortize the cost of session.get(), session.checkpoint(), > session.commit() as well as any setup in onTrigger() that isn't dependent on > the FlowFile(s) attributes or content. > An AbstractBatchingProcessor type should reduce boilerplate code in candidate > processors and encourage uniform configurability via a property to control > batch size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-3225) Abstract Processor type that batches session.get() and session.commit() calls
[ https://issues.apache.org/jira/browse/NIFI-3225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Rosander updated NIFI-3225: - Attachment: after.png Attaching image of after get(50) where 1.9% of the time is spent in session.commit() > Abstract Processor type that batches session.get() and session.commit() calls > - > > Key: NIFI-3225 > URL: https://issues.apache.org/jira/browse/NIFI-3225 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Bryan Rosander >Assignee: Bryan Rosander >Priority: Minor > Attachments: after.png, before.png > > > For processors that are stateless and support batching, it should be safe to > get and process multiple input FlowFiles for each onTrigger() call. > This should amortize the cost of session.get(), session.checkpoint(), > session.commit() as well as any setup in onTrigger() that isn't dependent on > the FlowFile(s) attributes or content. > An AbstractBatchingProcessor type should reduce boilerplate code in candidate > processors and encourage uniform configurability via a property to control > batch size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-3225) Abstract Processor type that batches session.get() and session.commit() calls
[ https://issues.apache.org/jira/browse/NIFI-3225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Rosander updated NIFI-3225: - Attachment: before.png Attaching image of before get(50) where 5.9% of the time is spent calling session.commit() > Abstract Processor type that batches session.get() and session.commit() calls > - > > Key: NIFI-3225 > URL: https://issues.apache.org/jira/browse/NIFI-3225 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Bryan Rosander >Assignee: Bryan Rosander >Priority: Minor > Attachments: before.png > > > For processors that are stateless and support batching, it should be safe to > get and process multiple input FlowFiles for each onTrigger() call. > This should amortize the cost of session.get(), session.checkpoint(), > session.commit() as well as any setup in onTrigger() that isn't dependent on > the FlowFile(s) attributes or content. > An AbstractBatchingProcessor type should reduce boilerplate code in candidate > processors and encourage uniform configurability via a property to control > batch size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-3194) PutElasticsearchHttp should route flowfiles to failure on connection errors
[
https://issues.apache.org/jira/browse/NIFI-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15761993#comment-15761993
]
ASF GitHub Bot commented on NIFI-3194:
--
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1327#discussion_r93095656
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java
---
@@ -321,9 +321,16 @@ public void onTrigger(final ProcessContext context,
final ProcessSession session
final Response getResponse;
try {
getResponse = sendRequestToElasticsearch(okHttpClient,
url, username, password, "PUT", requestBody);
-} catch (IllegalStateException | IOException ioe) {
-throw new ProcessException(ioe);
+} catch (final Exception e) {
+logger.error("Routing to {} due to exception: {}", new
Object[]{REL_FAILURE.getName(), e}, e);
+flowFilesToTransfer.forEach((flowFileToTransfer) -> {
+flowFileToTransfer =
session.penalize(flowFileToTransfer);
--- End diff --
Rgr. as long as there is reasoning
> PutElasticsearchHttp should route flowfiles to failure on connection errors
> ---
>
> Key: NIFI-3194
> URL: https://issues.apache.org/jira/browse/NIFI-3194
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.2.0
>
>
> If PutElasticsearchHttp encounters connection errors with the specified URL
> (such as there is no server available at the specified hostname:port), a
> ProcessException is thrown and the session is rolled back.
> For other processors that accept input flow files and access a URL (such as
> PostHttp and InvokeHttp), if such an error occurs, the flow file(s) are
> routed to failure. PutElasticsearchHttp should have behavior consistent with
> the others, and if connection errors occur, any incoming flow files should be
> routed to failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] nifi pull request #1327: NIFI-3194: PutElasticsearchHttp should route flowfi...
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1327#discussion_r93095656
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java
---
@@ -321,9 +321,16 @@ public void onTrigger(final ProcessContext context,
final ProcessSession session
final Response getResponse;
try {
getResponse = sendRequestToElasticsearch(okHttpClient,
url, username, password, "PUT", requestBody);
-} catch (IllegalStateException | IOException ioe) {
-throw new ProcessException(ioe);
+} catch (final Exception e) {
+logger.error("Routing to {} due to exception: {}", new
Object[]{REL_FAILURE.getName(), e}, e);
+flowFilesToTransfer.forEach((flowFileToTransfer) -> {
+flowFileToTransfer =
session.penalize(flowFileToTransfer);
--- End diff --
Rgr. as long as there is reasoning
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (NIFI-3225) Abstract Processor type that batches session.get() and session.commit() calls
[ https://issues.apache.org/jira/browse/NIFI-3225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15761984#comment-15761984 ] Bryan Rosander commented on NIFI-3225: -- [~JPercivall] I wasn't aware of the history of using get(50) before run duration was introduced. I thought to test this after seeing that that was how some other (apparently legacy) processors operate. I've been trying to increase throughput through a flow consisting of SplitJson, EvaluateJsonPath, RouteOnAttribute, UpdateAttribute, and AttributesToJson processors all with a run duration of 25 ms (much higher and things start to choke as the queues fill up and swapping begins). I noticed that about 6% of the AbstractProcessor.onTrigger() time was spend in session.checkpoint(). When changing the get() call to get(50), the time in session.checkpoint() dropped to about 2%. It seemed like a relatively low hanging fruit to get ~= 4% higher throughput and the other points were just a bonus although I agree that OnScheduled makes sense for things that don't change. It seems like the run duration setting and this batch size setting could be complementary for side effect free processors. Even if session.checkpoint() is cheap, it isn't free and for things that aren't modifying external state, the cost can be reduced just by calling session.commit() less frequently. > Abstract Processor type that batches session.get() and session.commit() calls > - > > Key: NIFI-3225 > URL: https://issues.apache.org/jira/browse/NIFI-3225 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Bryan Rosander >Assignee: Bryan Rosander >Priority: Minor > > For processors that are stateless and support batching, it should be safe to > get and process multiple input FlowFiles for each onTrigger() call. > This should amortize the cost of session.get(), session.checkpoint(), > session.commit() as well as any setup in onTrigger() that isn't dependent on > the FlowFile(s) attributes or content. > An AbstractBatchingProcessor type should reduce boilerplate code in candidate > processors and encourage uniform configurability via a property to control > batch size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-3194) PutElasticsearchHttp should route flowfiles to failure on connection errors
[
https://issues.apache.org/jira/browse/NIFI-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15761972#comment-15761972
]
ASF GitHub Bot commented on NIFI-3194:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1327#discussion_r93094070
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java
---
@@ -321,9 +321,16 @@ public void onTrigger(final ProcessContext context,
final ProcessSession session
final Response getResponse;
try {
getResponse = sendRequestToElasticsearch(okHttpClient,
url, username, password, "PUT", requestBody);
-} catch (IllegalStateException | IOException ioe) {
-throw new ProcessException(ioe);
+} catch (final Exception e) {
+logger.error("Routing to {} due to exception: {}", new
Object[]{REL_FAILURE.getName(), e}, e);
+flowFilesToTransfer.forEach((flowFileToTransfer) -> {
+flowFileToTransfer =
session.penalize(flowFileToTransfer);
--- End diff --
I think it was because the processor couldn't tell from the exception
whether it could/should be handled as a retry or failure, so I just penalized
it and sent it to failure. I can change it if that's not the best way to handle
it
> PutElasticsearchHttp should route flowfiles to failure on connection errors
> ---
>
> Key: NIFI-3194
> URL: https://issues.apache.org/jira/browse/NIFI-3194
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.2.0
>
>
> If PutElasticsearchHttp encounters connection errors with the specified URL
> (such as there is no server available at the specified hostname:port), a
> ProcessException is thrown and the session is rolled back.
> For other processors that accept input flow files and access a URL (such as
> PostHttp and InvokeHttp), if such an error occurs, the flow file(s) are
> routed to failure. PutElasticsearchHttp should have behavior consistent with
> the others, and if connection errors occur, any incoming flow files should be
> routed to failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] nifi pull request #1327: NIFI-3194: PutElasticsearchHttp should route flowfi...
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1327#discussion_r93094070
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java
---
@@ -321,9 +321,16 @@ public void onTrigger(final ProcessContext context,
final ProcessSession session
final Response getResponse;
try {
getResponse = sendRequestToElasticsearch(okHttpClient,
url, username, password, "PUT", requestBody);
-} catch (IllegalStateException | IOException ioe) {
-throw new ProcessException(ioe);
+} catch (final Exception e) {
+logger.error("Routing to {} due to exception: {}", new
Object[]{REL_FAILURE.getName(), e}, e);
+flowFilesToTransfer.forEach((flowFileToTransfer) -> {
+flowFileToTransfer =
session.penalize(flowFileToTransfer);
--- End diff --
I think it was because the processor couldn't tell from the exception
whether it could/should be handled as a retry or failure, so I just penalized
it and sent it to failure. I can change it if that's not the best way to handle
it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (NIFI-3194) PutElasticsearchHttp should route flowfiles to failure on connection errors
[
https://issues.apache.org/jira/browse/NIFI-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15761969#comment-15761969
]
ASF GitHub Bot commented on NIFI-3194:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1327#discussion_r93093642
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java
---
@@ -321,9 +321,16 @@ public void onTrigger(final ProcessContext context,
final ProcessSession session
final Response getResponse;
try {
getResponse = sendRequestToElasticsearch(okHttpClient,
url, username, password, "PUT", requestBody);
-} catch (IllegalStateException | IOException ioe) {
-throw new ProcessException(ioe);
+} catch (final Exception e) {
--- End diff --
I couldn't see any in the various calls that are made in that method
> PutElasticsearchHttp should route flowfiles to failure on connection errors
> ---
>
> Key: NIFI-3194
> URL: https://issues.apache.org/jira/browse/NIFI-3194
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.2.0
>
>
> If PutElasticsearchHttp encounters connection errors with the specified URL
> (such as there is no server available at the specified hostname:port), a
> ProcessException is thrown and the session is rolled back.
> For other processors that accept input flow files and access a URL (such as
> PostHttp and InvokeHttp), if such an error occurs, the flow file(s) are
> routed to failure. PutElasticsearchHttp should have behavior consistent with
> the others, and if connection errors occur, any incoming flow files should be
> routed to failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] nifi pull request #1327: NIFI-3194: PutElasticsearchHttp should route flowfi...
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1327#discussion_r93093642
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java
---
@@ -321,9 +321,16 @@ public void onTrigger(final ProcessContext context,
final ProcessSession session
final Response getResponse;
try {
getResponse = sendRequestToElasticsearch(okHttpClient,
url, username, password, "PUT", requestBody);
-} catch (IllegalStateException | IOException ioe) {
-throw new ProcessException(ioe);
+} catch (final Exception e) {
--- End diff --
I couldn't see any in the various calls that are made in that method
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi-minifi-cpp pull request #30: MINIFI-165 Updating licensing terms to inc...
GitHub user apiri opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/30 MINIFI-165 Updating licensing terms to include files bundled with spdlog MINIFI-165 Updating licensing terms to include files bundled with spdlog. mpmc_bounded_q comes from http://www.1024cores.net/home/lock-free-algorithms/queues/bounded-mpmc-queue and has a separate LICENSE file available from http://sites.google.com/site/1024cores/mpmc_bounded.zip You can merge this pull request into a Git repository by running: $ git pull https://github.com/apiri/nifi-minifi-cpp MINIFI-165 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/30.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #30 commit 0288d0d7b7858f6093c87682619cad1e73833546 Author: Aldrin Piri Date: 2016-12-19T18:36:36Z MINIFI-165 Updating licensing terms to include files bundled with spdlog. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (NIFI-3227) Support writing to separate logfiles by Processor Group
[ https://issues.apache.org/jira/browse/NIFI-3227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randy Gelhausen resolved NIFI-3227. --- Resolution: Duplicate Closed as duplicate of [NIFI-3065](https://issues.apache.org/jira/browse/NIFI-3065) > Support writing to separate logfiles by Processor Group > --- > > Key: NIFI-3227 > URL: https://issues.apache.org/jira/browse/NIFI-3227 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Randy Gelhausen > > In my multi-tenant NiFi cluster, projects are "isolated" from one another in > separate Process Groups. > However, all Process Group log messages are written to the same nifi-app.log > file, making it difficult to separate one project's log messages from > another's when debugging. > I would like to be able to see separate logfiles on a per Process Group basis. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-3227) Support writing to separate logfiles by Processor Group
Randy Gelhausen created NIFI-3227: - Summary: Support writing to separate logfiles by Processor Group Key: NIFI-3227 URL: https://issues.apache.org/jira/browse/NIFI-3227 Project: Apache NiFi Issue Type: Improvement Reporter: Randy Gelhausen In my multi-tenant NiFi cluster, projects are "isolated" from one another in separate Process Groups. However, all Process Group log messages are written to the same nifi-app.log file, making it difficult to separate one project's log messages from another's when debugging. I would like to be able to see separate logfiles on a per Process Group basis. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-3226) Elastic search url should support expression language
Frank Maritato created NIFI-3226:
Summary: Elastic search url should support expression language
Key: NIFI-3226
URL: https://issues.apache.org/jira/browse/NIFI-3226
Project: Apache NiFi
Issue Type: Improvement
Affects Versions: 1.1.0
Reporter: Frank Maritato
We have different environments, dev, staging, production, each with different
elastic search servers. Instead of having to manually change the url for each
elastic search processor can you please support expression language for this
property so I can specify ${elasticsearch.url} ?
The following two processors would need this change:
* PutElasticsearchHttp
* FetchElasticsearchHttp
Thanks!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-3225) Abstract Processor type that batches session.get() and session.commit() calls
[ https://issues.apache.org/jira/browse/NIFI-3225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15761716#comment-15761716 ] Joseph Percivall commented on NIFI-3225: Calling "get(X)" with a hardcoded number is how stateless processors used to do things[1] before the ability to set the "Run Duration"[2] was added. The run duration takes care of the batching together the checkpoint and commit. Also if there is setup that isn't dependent on the attributes or content and can be re-used, shouldn't it already be done in the OnScheduled? Is there a specific use-case you have come across recently that warrants a need for this? [1] https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-update-attribute-bundle/nifi-update-attribute-processor/src/main/java/org/apache/nifi/processors/attributes/UpdateAttribute.java#L338 [2] https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#scheduling-tab > Abstract Processor type that batches session.get() and session.commit() calls > - > > Key: NIFI-3225 > URL: https://issues.apache.org/jira/browse/NIFI-3225 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Bryan Rosander >Assignee: Bryan Rosander >Priority: Minor > > For processors that are stateless and support batching, it should be safe to > get and process multiple input FlowFiles for each onTrigger() call. > This should amortize the cost of session.get(), session.checkpoint(), > session.commit() as well as any setup in onTrigger() that isn't dependent on > the FlowFile(s) attributes or content. > An AbstractBatchingProcessor type should reduce boilerplate code in candidate > processors and encourage uniform configurability via a property to control > batch size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (NIFI-3225) Abstract Processor type that batches session.get() and session.commit() calls
[ https://issues.apache.org/jira/browse/NIFI-3225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Rosander reassigned NIFI-3225: Assignee: Bryan Rosander > Abstract Processor type that batches session.get() and session.commit() calls > - > > Key: NIFI-3225 > URL: https://issues.apache.org/jira/browse/NIFI-3225 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Bryan Rosander >Assignee: Bryan Rosander >Priority: Minor > > For processors that are stateless and support batching, it should be safe to > get and process multiple input FlowFiles for each onTrigger() call. > This should amortize the cost of session.get(), session.checkpoint(), > session.commit() as well as any setup in onTrigger() that isn't dependent on > the FlowFile(s) attributes or content. > An AbstractBatchingProcessor type should reduce boilerplate code in candidate > processors and encourage uniform configurability via a property to control > batch size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-3225) Abstract Processor type that batches session.get() and session.commit() calls
Bryan Rosander created NIFI-3225: Summary: Abstract Processor type that batches session.get() and session.commit() calls Key: NIFI-3225 URL: https://issues.apache.org/jira/browse/NIFI-3225 Project: Apache NiFi Issue Type: Improvement Reporter: Bryan Rosander Priority: Minor For processors that are stateless and support batching, it should be safe to get and process multiple input FlowFiles for each onTrigger() call. This should amortize the cost of session.get(), session.checkpoint(), session.commit() as well as any setup in onTrigger() that isn't dependent on the FlowFile(s) attributes or content. An AbstractBatchingProcessor type should reduce boilerplate code in candidate processors and encourage uniform configurability via a property to control batch size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] nifi pull request #1327: NIFI-3194: PutElasticsearchHttp should route flowfi...
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1327#discussion_r93070476
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java
---
@@ -321,9 +321,16 @@ public void onTrigger(final ProcessContext context,
final ProcessSession session
final Response getResponse;
try {
getResponse = sendRequestToElasticsearch(okHttpClient,
url, username, password, "PUT", requestBody);
-} catch (IllegalStateException | IOException ioe) {
-throw new ProcessException(ioe);
+} catch (final Exception e) {
--- End diff --
Will there be any exceptions encountered here where we would want to route
to RETRY instead?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (NIFI-3194) PutElasticsearchHttp should route flowfiles to failure on connection errors
[
https://issues.apache.org/jira/browse/NIFI-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15761672#comment-15761672
]
ASF GitHub Bot commented on NIFI-3194:
--
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1327#discussion_r93070476
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java
---
@@ -321,9 +321,16 @@ public void onTrigger(final ProcessContext context,
final ProcessSession session
final Response getResponse;
try {
getResponse = sendRequestToElasticsearch(okHttpClient,
url, username, password, "PUT", requestBody);
-} catch (IllegalStateException | IOException ioe) {
-throw new ProcessException(ioe);
+} catch (final Exception e) {
--- End diff --
Will there be any exceptions encountered here where we would want to route
to RETRY instead?
> PutElasticsearchHttp should route flowfiles to failure on connection errors
> ---
>
> Key: NIFI-3194
> URL: https://issues.apache.org/jira/browse/NIFI-3194
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.2.0
>
>
> If PutElasticsearchHttp encounters connection errors with the specified URL
> (such as there is no server available at the specified hostname:port), a
> ProcessException is thrown and the session is rolled back.
> For other processors that accept input flow files and access a URL (such as
> PostHttp and InvokeHttp), if such an error occurs, the flow file(s) are
> routed to failure. PutElasticsearchHttp should have behavior consistent with
> the others, and if connection errors occur, any incoming flow files should be
> routed to failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-3194) PutElasticsearchHttp should route flowfiles to failure on connection errors
[
https://issues.apache.org/jira/browse/NIFI-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15761673#comment-15761673
]
ASF GitHub Bot commented on NIFI-3194:
--
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1327#discussion_r93070719
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java
---
@@ -321,9 +321,16 @@ public void onTrigger(final ProcessContext context,
final ProcessSession session
final Response getResponse;
try {
getResponse = sendRequestToElasticsearch(okHttpClient,
url, username, password, "PUT", requestBody);
-} catch (IllegalStateException | IOException ioe) {
-throw new ProcessException(ioe);
+} catch (final Exception e) {
+logger.error("Routing to {} due to exception: {}", new
Object[]{REL_FAILURE.getName(), e}, e);
+flowFilesToTransfer.forEach((flowFileToTransfer) -> {
+flowFileToTransfer =
session.penalize(flowFileToTransfer);
--- End diff --
Why penalize when routing to failure? My view of penalization is that it
should only be used when the flowfile failed to process at this time but if
processed again at a later time it may succeed (ie. a RETRY).
> PutElasticsearchHttp should route flowfiles to failure on connection errors
> ---
>
> Key: NIFI-3194
> URL: https://issues.apache.org/jira/browse/NIFI-3194
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.2.0
>
>
> If PutElasticsearchHttp encounters connection errors with the specified URL
> (such as there is no server available at the specified hostname:port), a
> ProcessException is thrown and the session is rolled back.
> For other processors that accept input flow files and access a URL (such as
> PostHttp and InvokeHttp), if such an error occurs, the flow file(s) are
> routed to failure. PutElasticsearchHttp should have behavior consistent with
> the others, and if connection errors occur, any incoming flow files should be
> routed to failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] nifi pull request #1327: NIFI-3194: PutElasticsearchHttp should route flowfi...
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1327#discussion_r93070719
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java
---
@@ -321,9 +321,16 @@ public void onTrigger(final ProcessContext context,
final ProcessSession session
final Response getResponse;
try {
getResponse = sendRequestToElasticsearch(okHttpClient,
url, username, password, "PUT", requestBody);
-} catch (IllegalStateException | IOException ioe) {
-throw new ProcessException(ioe);
+} catch (final Exception e) {
+logger.error("Routing to {} due to exception: {}", new
Object[]{REL_FAILURE.getName(), e}, e);
+flowFilesToTransfer.forEach((flowFileToTransfer) -> {
+flowFileToTransfer =
session.penalize(flowFileToTransfer);
--- End diff --
Why penalize when routing to failure? My view of penalization is that it
should only be used when the flowfile failed to process at this time but if
processed again at a later time it may succeed (ie. a RETRY).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (NIFI-3194) PutElasticsearchHttp should route flowfiles to failure on connection errors
[ https://issues.apache.org/jira/browse/NIFI-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15761595#comment-15761595 ] ASF GitHub Bot commented on NIFI-3194: -- Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/1327 Reviewing > PutElasticsearchHttp should route flowfiles to failure on connection errors > --- > > Key: NIFI-3194 > URL: https://issues.apache.org/jira/browse/NIFI-3194 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess > Fix For: 1.2.0 > > > If PutElasticsearchHttp encounters connection errors with the specified URL > (such as there is no server available at the specified hostname:port), a > ProcessException is thrown and the session is rolled back. > For other processors that accept input flow files and access a URL (such as > PostHttp and InvokeHttp), if such an error occurs, the flow file(s) are > routed to failure. PutElasticsearchHttp should have behavior consistent with > the others, and if connection errors occur, any incoming flow files should be > routed to failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] nifi issue #1327: NIFI-3194: PutElasticsearchHttp should route flowfiles to ...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/1327 Reviewing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (NIFI-3202) Perform rm duties for nifi 1.0.1
[ https://issues.apache.org/jira/browse/NIFI-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall resolved NIFI-3202. Resolution: Fixed > Perform rm duties for nifi 1.0.1 > > > Key: NIFI-3202 > URL: https://issues.apache.org/jira/browse/NIFI-3202 > Project: Apache NiFi > Issue Type: Task > Components: Tools and Build >Reporter: Joseph Witt >Assignee: Joseph Percivall > Fix For: 1.0.1 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-3170) UI - Unauthorized action in Flow Configuration History
[ https://issues.apache.org/jira/browse/NIFI-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15761405#comment-15761405 ] ASF GitHub Bot commented on NIFI-3170: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/1322 > UI - Unauthorized action in Flow Configuration History > -- > > Key: NIFI-3170 > URL: https://issues.apache.org/jira/browse/NIFI-3170 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Reporter: Matt Gilman >Assignee: Scott Aslan >Priority: Trivial > Fix For: 1.2.0 > > > When a user does not have permissions to an action in the flow configuration > history, we should not be rendering the more info icon. Currently, if you > attempt to click the more info icon there is a javascript error since the > additional details are not returned to the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-3170) UI - Unauthorized action in Flow Configuration History
[ https://issues.apache.org/jira/browse/NIFI-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15761404#comment-15761404 ] ASF GitHub Bot commented on NIFI-3170: -- Github user mcgilman commented on the issue: https://github.com/apache/nifi/pull/1322 Thanks @scottyaslan! This has been merged to master. > UI - Unauthorized action in Flow Configuration History > -- > > Key: NIFI-3170 > URL: https://issues.apache.org/jira/browse/NIFI-3170 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Reporter: Matt Gilman >Assignee: Scott Aslan >Priority: Trivial > Fix For: 1.2.0 > > > When a user does not have permissions to an action in the flow configuration > history, we should not be rendering the more info icon. Currently, if you > attempt to click the more info icon there is a javascript error since the > additional details are not returned to the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-3170) UI - Unauthorized action in Flow Configuration History
[ https://issues.apache.org/jira/browse/NIFI-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman updated NIFI-3170: -- Resolution: Fixed Status: Resolved (was: Patch Available) > UI - Unauthorized action in Flow Configuration History > -- > > Key: NIFI-3170 > URL: https://issues.apache.org/jira/browse/NIFI-3170 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Reporter: Matt Gilman >Assignee: Scott Aslan >Priority: Trivial > Fix For: 1.2.0 > > > When a user does not have permissions to an action in the flow configuration > history, we should not be rendering the more info icon. Currently, if you > attempt to click the more info icon there is a javascript error since the > additional details are not returned to the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] nifi issue #1322: [NIFI-3170] remove action details if user does not have re...
Github user mcgilman commented on the issue: https://github.com/apache/nifi/pull/1322 Thanks @scottyaslan! This has been merged to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3170) UI - Unauthorized action in Flow Configuration History
[ https://issues.apache.org/jira/browse/NIFI-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15761402#comment-15761402 ] ASF subversion and git services commented on NIFI-3170: --- Commit aef17f9a8b5d9ce336f208987655134b59004b9a in nifi's branch refs/heads/master from [~scottyaslan] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=aef17f9 ] [NIFI-3170] remove action details if user does not have read perms, also update action details styles to match other dialogs. This closes #1322 > UI - Unauthorized action in Flow Configuration History > -- > > Key: NIFI-3170 > URL: https://issues.apache.org/jira/browse/NIFI-3170 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Reporter: Matt Gilman >Assignee: Scott Aslan >Priority: Trivial > Fix For: 1.2.0 > > > When a user does not have permissions to an action in the flow configuration > history, we should not be rendering the more info icon. Currently, if you > attempt to click the more info icon there is a javascript error since the > additional details are not returned to the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] nifi pull request #1322: [NIFI-3170] remove action details if user does not ...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/1322 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Assigned] (NIFI-1784) Create a FetchHBase Processor
[ https://issues.apache.org/jira/browse/NIFI-1784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Bende reassigned NIFI-1784: - Assignee: Bryan Bende > Create a FetchHBase Processor > - > > Key: NIFI-1784 > URL: https://issues.apache.org/jira/browse/NIFI-1784 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Minor > > We should provide a processor to fetch a row from HBase. The processor should > support receiving an incoming FlowFile and taking the row id to fetch from an > attribute on the incoming, and should also be able to fetch a static row id. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-3195) 1
[ https://issues.apache.org/jira/browse/NIFI-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] crescent updated NIFI-3195: --- Description: was: NiFi in Cluster mode Application scenarios： 1.get realtime files from ftp server 2.split file to small files 3.put small files data to kafka Version 1.0.0: Avoid duplicate files，The first processor is ListSFTP scheduled :On primary node, the following processors scheduling strategy : Timer driven .After running for several hours，found all the tasks done by primary node，unable to use cluster. Version 1.1.0: Avoid duplicate files，The first processor is ListSFTP scheduled :Timer driven , Execution: Primary Node Only. The following processors scheduling strategy : Timer driven ,Execution: All nodes.After testing for several hours，found that: the following tasks only done by primary node，unable to use cluster. > 1 > - > > Key: NIFI-3195 > URL: https://issues.apache.org/jira/browse/NIFI-3195 > Project: Apache NiFi > Issue Type: Test >Reporter: crescent >Priority: Trivial > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2585) Add attributes to track where a flow file came from when receiving over site-to-site
[ https://issues.apache.org/jira/browse/NIFI-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15760732#comment-15760732 ] ASF GitHub Bot commented on NIFI-2585: -- Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/1320 @bbende @randerzander Thanks for proposing this feature, it'll be useful! Started reviewing this, and noticed few things. 1. Difference between RAW and HTTP protocol for receiving data. RAW resolves hostname from socket, while HTTP does so from HTTP request. Due to this difference, RAW uses hostname as `s2s.host` attribute like `localhost`, while HTTP set `127.0.0.1`. I think this may be difficult when users define routing rules in downstream flow. I wonder if it can produce the same value, hostname or ip address for both transport protocol. 2. Unknown hostname and port due to illegal URL As discussed in the [previous PR](https://github.com/apache/nifi/pull/1307#issuecomment-266442657), I got 'unknown' hostname and port with Docker containers. While this can be handled as a corner case since it's not allowed by the URL specification, I think it'd be better if we can be lenient here to support Docker environment. I confirmed provenance event also failed to resolve hostname and produced provenance details with `null` hostname. I think this can be improved by changing Peer's constructor. It parses peerUrl then use its hostname and port, but the same information should be retrieved from PeerDescriptor without parsing the URL. Currently, SocketRemoteSiteListener uses server's hostname and port for PeerDescription, but it seems it's not correct, those should be client's. Please let me dig deeper comments above. Will update soon. > Add attributes to track where a flow file came from when receiving over > site-to-site > > > Key: NIFI-2585 > URL: https://issues.apache.org/jira/browse/NIFI-2585 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Bryan Bende >Assignee: Randy Gelhausen >Priority: Minor > > With MiNiFi starting be used to send data to a central NiFi, it would be > helpful if information about the sending host and port was added to each flow > file received over site-to-site. Currently this information is available and > used to generate the transit URI in the RECEIVE event, but this information > isn't available to downstream processors that might want to make routing > decisions. > For reference: > https://github.com/apache/nifi/blob/e23b2356172e128086585fe2c425523c3628d0e7/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-site-to-site/src/main/java/org/apache/nifi/remote/protocol/AbstractFlowFileServerProtocol.java#L452 > A possible approach might be to add two attributes to each flow file, > something like "remote.host" and "remote.address" where remote.host has only > the sending hostname, and remote.address has the sending host and port. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] nifi issue #1320: NIFI-2585 Add attributes to track s2s host and port
Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/1320 @bbende @randerzander Thanks for proposing this feature, it'll be useful! Started reviewing this, and noticed few things. 1. Difference between RAW and HTTP protocol for receiving data. RAW resolves hostname from socket, while HTTP does so from HTTP request. Due to this difference, RAW uses hostname as `s2s.host` attribute like `localhost`, while HTTP set `127.0.0.1`. I think this may be difficult when users define routing rules in downstream flow. I wonder if it can produce the same value, hostname or ip address for both transport protocol. 2. Unknown hostname and port due to illegal URL As discussed in the [previous PR](https://github.com/apache/nifi/pull/1307#issuecomment-266442657), I got 'unknown' hostname and port with Docker containers. While this can be handled as a corner case since it's not allowed by the URL specification, I think it'd be better if we can be lenient here to support Docker environment. I confirmed provenance event also failed to resolve hostname and produced provenance details with `null` hostname. I think this can be improved by changing Peer's constructor. It parses peerUrl then use its hostname and port, but the same information should be retrieved from PeerDescriptor without parsing the URL. Currently, SocketRemoteSiteListener uses server's hostname and port for PeerDescription, but it seems it's not correct, those should be client's. Please let me dig deeper comments above. Will update soon. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (NIFI-3224) build on windows,encountered an error,that is frontend-working-directory\node\node.exe is not a valid Win32 application
Lionel created NIFI-3224: Summary: build on windows,encountered an error,that is frontend-working-directory\node\node.exe is not a valid Win32 application Key: NIFI-3224 URL: https://issues.apache.org/jira/browse/NIFI-3224 Project: Apache NiFi Issue Type: Bug Components: Tools and Build Reporter: Lionel when i build nifi by execute mvn -T 2.0C clean install on windows ,then encountered an error,The error is the following: [ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.0:npm (npm install) on project nifi-web-ui: Failed to run task: 'npm --cache-min Infinity install' failed. java.io.IOException: Cannot run program "F:\nifi-1.1.0\nifi-nar-bundles\nifi-framework-bundle\nifi-framework\nifi-web\nifi-web-ui\target\frontend-working-directory\node\node.exe" (in directory "F:\nifi-1.1.0\nifi-nar-bundles\nifi-framework-bundle\nifi-framework\nifi-web\nifi-web-ui\target\frontend-working-directory"): CreateProcess error=193, is not a valid Win32 application -- This message was sent by Atlassian JIRA (v6.3.4#6332)
