[jira] [Commented] (NIFI-5443) Improve cluster configuration for dynamic scaling
[
https://issues.apache.org/jira/browse/NIFI-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559232#comment-16559232
]
Prashanth Venkatesan commented on NIFI-5443:
[~alopresto] There is no blocker... Currently as a workaround in our docker
scripts, we introduced one boolean flag to control updation of authorizers.xml.
So, in case of scale-out cases, we pass that flag as "false" so that
*authorizers.xml* won't be populated. This seems like currently made our
dynamic scaling bit smoother.
> Improve cluster configuration for dynamic scaling
> -
>
> Key: NIFI-5443
> URL: https://issues.apache.org/jira/browse/NIFI-5443
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Andy LoPresto
>Priority: Critical
> Labels: cluster, docker, kubernetes, rkt, scale, security
>
> Currently, NiFi is designed for static clusters, with frequent references in
> configuration files to a priori knowledge of node hostnames, ports, etc.
> Efforts should be taken to make NiFi easier to dynamically scale. This can
> involve containerization improvements via Docker/rkt, deployment improvements
> via Kubernetes, and abstraction of the configuration values needed to stand
> up the cluster. A node should be able to join the cluster, and, given the
> correct keystore and truststore, immediately communicate with other existing
> nodes in the cluster without requiring direct configuration changes to them,
> or a restart of any node.
> * {{authorizers.xml}}
> * node identities
> * permissions ({{RW}} on {{/proxy}})
> * ZooKeeper configuration
> * etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5400) NiFiHostnameVerifier should be replaced
[
https://issues.apache.org/jira/browse/NIFI-5400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559182#comment-16559182
]
ASF GitHub Bot commented on NIFI-5400:
--
Github user alopresto commented on the issue:
https://github.com/apache/nifi/pull/2919
This PR removes the locally-defined inner class `NiFiHostnameVerifier` from
`NiFiClientFactory` and adds unit tests but does not cover the class
[`org.apache.nifi.web.util.NiFiHostnameVerifier`](https://github.com/apache/nifi/blob/master/nifi-commons/nifi-web-utils/src/main/java/org/apache/nifi/web/util/NiFiHostnameVerifier.java#L18)
which is used in
[`org.apache.nifi.web.util.WebUtils#createClientHelper()`](https://github.com/apache/nifi/blob/master/nifi-commons/nifi-web-utils/src/main/java/org/apache/nifi/web/util/WebUtils.java#L98).
Please remove that class entirely and instantiate a
`org.apache.http.conn.ssl.DefaultHostnameVerifier` in
`WebUtils#createClientHelper()` and verify that the calling references (uses of
`WebUtils#createClient(ClientConfig)` and `WebUtils#createClient(ClientConfig,
SSLContext)` listed below) continue to work properly (i.e. add unit tests &
regression tests for edge case hostnames).
```
Method
createClient(ClientConfig, SSLContext)
Found usages (5 usages found)
Unclassified usage (5 usages found)
nifi-confluent-schema-registry-service (1 usage found)
org.apache.nifi.confluent.schemaregistry.client (1 usage found)
RestSchemaRegistryClient (1 usage found)
RestSchemaRegistryClient(List, int, SSLContext,
ComponentLog) (1 usage found)
77 client = WebUtils.createClient(clientConfig,
sslContext);
nifi-framework-core (1 usage found)
org.apache.nifi.remote (1 usage found)
RemoteNiFiUtils (1 usage found)
getClient(SSLContext) (1 usage found)
50 client = WebUtils.createClient(clientConfig,
sslContext);
nifi-web-api (2 usages found)
org.apache.nifi.integration.accesscontrol (1 usage found)
ITAccessTokenEndpoint (1 usage found)
setup() (1 usage found)
88 final Client client =
WebUtils.createClient(null, createTrustContext(props));
org.apache.nifi.integration.util (1 usage found)
NiFiTestServer (1 usage found)
getClient() (1 usage found)
170 return WebUtils.createClient(null,
SslContextFactory.createSslContext(properties));
nifi-web-security (1 usage found)
org.apache.nifi.web.security.x509.ocsp (1 usage found)
OcspCertificateValidator (1 usage found)
OcspCertificateValidator(NiFiProperties) (1 usage
found)
110 client = WebUtils.createClient(clientConfig,
SslContextFactory.createSslContext(properties));
```
```
Method
createClient(ClientConfig)
Found usages (3 usages found)
Unclassified usage (3 usages found)
nifi-framework-core (1 usage found)
org.apache.nifi.remote (1 usage found)
RemoteNiFiUtils (1 usage found)
getClient(SSLContext) (1 usage found)
48 client = WebUtils.createClient(clientConfig);
nifi-web-security (1 usage found)
org.apache.nifi.web.security.x509.ocsp (1 usage found)
OcspCertificateValidator (1 usage found)
OcspCertificateValidator(NiFiProperties) (1 usage
found)
112 client = WebUtils.createClient(clientConfig);
nifi-yandex-processors (1 usage found)
org.apache.nifi.processors.yandex (1 usage found)
YandexTranslate (1 usage found)
onScheduled(ProcessContext) (1 usage found)
201 client = WebUtils.createClient(null);
```
> NiFiHostnameVerifier should be replaced
> ---
>
> Key: NIFI-5400
> URL: https://issues.apache.org/jira/browse/NIFI-5400
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.0
>Reporter: Andy LoPresto
>Priority: Major
> Labels: certificate, hostname, security, tls
>
> The {{NiFiHostnameVerifier}} does not handle wildcard certificates or complex
> {{SubjectAlternativeNames}}. It should be replaced with a more full-featured
> implementation, like {{OkHostnameVerifier}} from {{okhttp}} or
> {{DefaultHostnameVerifier}} from {{http-client}}.
[GitHub] nifi issue #2919: NIFI-5400 - Changed the hostname verifier from the custom ...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2919 This PR removes the locally-defined inner class `NiFiHostnameVerifier` from `NiFiClientFactory` and adds unit tests but does not cover the class [`org.apache.nifi.web.util.NiFiHostnameVerifier`](https://github.com/apache/nifi/blob/master/nifi-commons/nifi-web-utils/src/main/java/org/apache/nifi/web/util/NiFiHostnameVerifier.java#L18) which is used in [`org.apache.nifi.web.util.WebUtils#createClientHelper()`](https://github.com/apache/nifi/blob/master/nifi-commons/nifi-web-utils/src/main/java/org/apache/nifi/web/util/WebUtils.java#L98). Please remove that class entirely and instantiate a `org.apache.http.conn.ssl.DefaultHostnameVerifier` in `WebUtils#createClientHelper()` and verify that the calling references (uses of `WebUtils#createClient(ClientConfig)` and `WebUtils#createClient(ClientConfig, SSLContext)` listed below) continue to work properly (i.e. add unit tests & regression tests for edge case hostnames). ``` Method createClient(ClientConfig, SSLContext) Found usages (5 usages found) Unclassified usage (5 usages found) nifi-confluent-schema-registry-service (1 usage found) org.apache.nifi.confluent.schemaregistry.client (1 usage found) RestSchemaRegistryClient (1 usage found) RestSchemaRegistryClient(List, int, SSLContext, ComponentLog) (1 usage found) 77 client = WebUtils.createClient(clientConfig, sslContext); nifi-framework-core (1 usage found) org.apache.nifi.remote (1 usage found) RemoteNiFiUtils (1 usage found) getClient(SSLContext) (1 usage found) 50 client = WebUtils.createClient(clientConfig, sslContext); nifi-web-api (2 usages found) org.apache.nifi.integration.accesscontrol (1 usage found) ITAccessTokenEndpoint (1 usage found) setup() (1 usage found) 88 final Client client = WebUtils.createClient(null, createTrustContext(props)); org.apache.nifi.integration.util (1 usage found) NiFiTestServer (1 usage found) getClient() (1 usage found) 170 return WebUtils.createClient(null, SslContextFactory.createSslContext(properties)); nifi-web-security (1 usage found) org.apache.nifi.web.security.x509.ocsp (1 usage found) OcspCertificateValidator (1 usage found) OcspCertificateValidator(NiFiProperties) (1 usage found) 110 client = WebUtils.createClient(clientConfig, SslContextFactory.createSslContext(properties)); ``` ``` Method createClient(ClientConfig) Found usages (3 usages found) Unclassified usage (3 usages found) nifi-framework-core (1 usage found) org.apache.nifi.remote (1 usage found) RemoteNiFiUtils (1 usage found) getClient(SSLContext) (1 usage found) 48 client = WebUtils.createClient(clientConfig); nifi-web-security (1 usage found) org.apache.nifi.web.security.x509.ocsp (1 usage found) OcspCertificateValidator (1 usage found) OcspCertificateValidator(NiFiProperties) (1 usage found) 112 client = WebUtils.createClient(clientConfig); nifi-yandex-processors (1 usage found) org.apache.nifi.processors.yandex (1 usage found) YandexTranslate (1 usage found) onScheduled(ProcessContext) (1 usage found) 201 client = WebUtils.createClient(null); ``` ---
[jira] [Commented] (NIFI-5400) NiFiHostnameVerifier should be replaced
[
https://issues.apache.org/jira/browse/NIFI-5400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558976#comment-16558976
]
ASF GitHub Bot commented on NIFI-5400:
--
Github user alopresto commented on the issue:
https://github.com/apache/nifi/pull/2919
Reviewing...
> NiFiHostnameVerifier should be replaced
> ---
>
> Key: NIFI-5400
> URL: https://issues.apache.org/jira/browse/NIFI-5400
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.0
>Reporter: Andy LoPresto
>Priority: Major
> Labels: certificate, hostname, security, tls
>
> The {{NiFiHostnameVerifier}} does not handle wildcard certificates or complex
> {{SubjectAlternativeNames}}. It should be replaced with a more full-featured
> implementation, like {{OkHostnameVerifier}} from {{okhttp}} or
> {{DefaultHostnameVerifier}} from {{http-client}}. Either of these options
> requires introducing a new Maven dependency to {{nifi-commons}} and requires
> further investigation.
> *Note: * the {{sun.net.www.protocol.httpsDefaultHostnameVerifier}} simply
> returns {{false}} on all inputs and is not a valid solution.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #2919: NIFI-5400 - Changed the hostname verifier from the custom ...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2919 Reviewing... ---
[jira] [Commented] (NIFI-5442) Message Page uses raw X-ProxyContextPath
[
https://issues.apache.org/jira/browse/NIFI-5442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558966#comment-16558966
]
ASF GitHub Bot commented on NIFI-5442:
--
Github user alopresto commented on the issue:
https://github.com/apache/nifi/pull/2908
This PR is now ready for final review.
> Message Page uses raw X-ProxyContextPath
>
>
> Key: NIFI-5442
> URL: https://issues.apache.org/jira/browse/NIFI-5442
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.6.0
>Reporter: Dan Fike
>Assignee: Andy LoPresto
>Priority: Major
>
> It looks like {{message-page.jsp}} uses {{X-ProxyContextPath}} verbatim
> without sanitizing it or anything. See
> [https://github.com/apache/nifi/blob/66783c18b24b1c6b1cfd662c58ca9df1e60b866e/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/WEB-INF/pages/message-page.jsp#L21]
>
> I verified this by hitting {{/nifi-api/access/oidc/callback}} on an unsecured
> NiFi host to get the *User authentication/authorization is only supported
> when running over HTTPS* message page.
>
> {code:java}
> $ curl http://hostname/nifi-api/access/oidc/callback
> ...
> type="text/css" />
> ...
> $ curl --header "X-ProxyContextPath: /nifi/assets/reset.css/reset.css\"
> type=\"text/css\" /> type=\"text/javascript\">alert(\"omg\"); href=\"" http://hostname/nifi-api/access/oidc/callback
> ...
> type="text/css" />alert("omg"); rel="stylesheet" href="/nifi/assets/reset.css/reset.css" type="text/css" />
> ...{code}
>
> Presumably we want to do something like this:
> [https://github.com/apache/nifi/commit/5d643edfaba4f5369c94ee1b4eaa5c59e3a9f37a#diff-91119fe15bb6f3b931662093e367b671R20]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #2908: NIFI-5442 Get X-ProxyContextPath value from request attrib...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2908 This PR is now ready for final review. ---
[jira] [Commented] (NIFI-5449) RecordPath Language should allow base64 encode and decode
[ https://issues.apache.org/jira/browse/NIFI-5449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558909#comment-16558909 ] ASF GitHub Bot commented on NIFI-5449: -- GitHub user mattyb149 opened a pull request: https://github.com/apache/nifi/pull/2920 NIFI-5449: Added Base64 Encode/Decode functions to RecordPath Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mattyb149/nifi NIFI-5449 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2920.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2920 commit a4a182de205dca9e8fb6bfc263127c505e9c7ec4 Author: Matthew Burgess Date: 2018-07-26T21:24:14Z NIFI-5449: Added Base64 Encode/Decode functions to RecordPath > RecordPath Language should allow base64 encode and decode > - > > Key: NIFI-5449 > URL: https://issues.apache.org/jira/browse/NIFI-5449 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Dalibor Frívaldský >Assignee: Matt Burgess >Priority: Minor > > Currently there is no way to encode/decode a record field using base64 > encoder/decoder. Without this functionality it is, for example, impossible to > store data into databases that require base64 representation of varbinary > values (e.g. Apache Phoenix). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5449) RecordPath Language should allow base64 encode and decode
[ https://issues.apache.org/jira/browse/NIFI-5449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-5449: --- Status: Patch Available (was: In Progress) > RecordPath Language should allow base64 encode and decode > - > > Key: NIFI-5449 > URL: https://issues.apache.org/jira/browse/NIFI-5449 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Dalibor Frívaldský >Assignee: Matt Burgess >Priority: Minor > > Currently there is no way to encode/decode a record field using base64 > encoder/decoder. Without this functionality it is, for example, impossible to > store data into databases that require base64 representation of varbinary > values (e.g. Apache Phoenix). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2920: NIFI-5449: Added Base64 Encode/Decode functions to ...
GitHub user mattyb149 opened a pull request: https://github.com/apache/nifi/pull/2920 NIFI-5449: Added Base64 Encode/Decode functions to RecordPath Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mattyb149/nifi NIFI-5449 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2920.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2920 commit a4a182de205dca9e8fb6bfc263127c505e9c7ec4 Author: Matthew Burgess Date: 2018-07-26T21:24:14Z NIFI-5449: Added Base64 Encode/Decode functions to RecordPath ---
[jira] [Commented] (NIFI-5400) NiFiHostnameVerifier should be replaced
[
https://issues.apache.org/jira/browse/NIFI-5400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558895#comment-16558895
]
ASF GitHub Bot commented on NIFI-5400:
--
GitHub user thenatog opened a pull request:
https://github.com/apache/nifi/pull/2919
NIFI-5400 - Changed the hostname verifier from the custom NiFi verifi…
…er to the Apache http-client DefaultHostnameVerifier
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [X] Does your PR title start with NIFI- where is the JIRA number
you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [X] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [X] Is your initial contribution a single, squashed commit?
### For code changes:
- [X] Have you ensured that the full suite of tests is executed via mvn
-Pcontrib-check clean install at the root nifi folder?
- [X] Have you written or updated unit tests to verify your changes?
- [X] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [X] If applicable, have you updated the LICENSE file, including the main
LICENSE file under nifi-assembly?
- [X] If applicable, have you updated the NOTICE file, including the main
NOTICE file found under nifi-assembly?
- [X] If adding new Properties, have you added .displayName in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [X] Have you ensured that format looks appropriate for the output in
which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/thenatog/nifi NIFI-5400
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/2919.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2919
commit 325fb220206be3d671cb787e45c8f75107dad44e
Author: thenatog
Date: 2018-07-26T20:33:52Z
NIFI-5400 - Changed the hostname verifier from the custom NiFi verifier to
the Apache http-client DefaultHostnameVerifier
> NiFiHostnameVerifier should be replaced
> ---
>
> Key: NIFI-5400
> URL: https://issues.apache.org/jira/browse/NIFI-5400
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.0
>Reporter: Andy LoPresto
>Priority: Major
> Labels: certificate, hostname, security, tls
>
> The {{NiFiHostnameVerifier}} does not handle wildcard certificates or complex
> {{SubjectAlternativeNames}}. It should be replaced with a more full-featured
> implementation, like {{OkHostnameVerifier}} from {{okhttp}} or
> {{DefaultHostnameVerifier}} from {{http-client}}. Either of these options
> requires introducing a new Maven dependency to {{nifi-commons}} and requires
> further investigation.
> *Note: * the {{sun.net.www.protocol.httpsDefaultHostnameVerifier}} simply
> returns {{false}} on all inputs and is not a valid solution.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2919: NIFI-5400 - Changed the hostname verifier from the ...
GitHub user thenatog opened a pull request: https://github.com/apache/nifi/pull/2919 NIFI-5400 - Changed the hostname verifier from the custom NiFi verifi⦠â¦er to the Apache http-client DefaultHostnameVerifier Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [X] Have you written or updated unit tests to verify your changes? - [X] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [X] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [X] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [X] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/thenatog/nifi NIFI-5400 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2919.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2919 commit 325fb220206be3d671cb787e45c8f75107dad44e Author: thenatog Date: 2018-07-26T20:33:52Z NIFI-5400 - Changed the hostname verifier from the custom NiFi verifier to the Apache http-client DefaultHostnameVerifier ---
[jira] [Commented] (NIFI-5083) Support for Expression Language in Group ID field of Kafka Consumer Processors
[ https://issues.apache.org/jira/browse/NIFI-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558887#comment-16558887 ] ASF GitHub Bot commented on NIFI-5083: -- Github user snagafritz closed the pull request at: https://github.com/apache/nifi/pull/2913 > Support for Expression Language in Group ID field of Kafka Consumer Processors > -- > > Key: NIFI-5083 > URL: https://issues.apache.org/jira/browse/NIFI-5083 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.6.0 >Reporter: Robert Bruno >Priority: Minor > Labels: kafka > Fix For: 1.8.0 > > Attachments: NIFI-5083.mp4 > > > I have the use case where I have two separate NiFi clusters reading data from > the same topics but need to have unique group ids so they each get separate > copies of the data. It would be useful if the group id field supported > expression language so I could use a processor group variable to define the > group id to use. I have a large enough number of kafka consumers to make > this tedious to manually change. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2913: NIFI-5083: enabled EL support for processor configu...
Github user snagafritz closed the pull request at: https://github.com/apache/nifi/pull/2913 ---
[jira] [Commented] (NIFI-5448) Failed EL date parsing live-locks processors without a failure relationship
[
https://issues.apache.org/jira/browse/NIFI-5448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558884#comment-16558884
]
ASF GitHub Bot commented on NIFI-5448:
--
Github user mcgilman commented on the issue:
https://github.com/apache/nifi/pull/2914
Looking at the changeset I see that a property was added to allow the user
to opt-in to the new behavior. That should alleviate my concerns. Sorry for the
confusion.
Only other suggestion would be to change the name of "Stop Processing"
`AllowableValue` to maybe "Penalize" as that is what the current behavior
should be doing.
> Failed EL date parsing live-locks processors without a failure relationship
> ---
>
> Key: NIFI-5448
> URL: https://issues.apache.org/jira/browse/NIFI-5448
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: David Koster
>Assignee: Mike Thomsen
>Priority: Major
>
> Processors that utilize the Expression Language need to always present a
> failure relationship.
> If a processor with only a success relationship, for example UpdateAttribute,
> utilizes the expression language to perform type coercion to a date and
> fails, the processor will be unable to dispose of the FlowFile and remain
> blocked indefinitely.
> Recreation flow:
> GenerateFlowFile -> Update Attribute #1 -> Update Attribute #2 -> Anything
> Update Attribute #1 - test = "Hello World"
> Update Attribute #2 - test = ${test:toDate('-MM-dd')}
>
> Generates an IllegalAttributeException on UpdateAttribute.
>
> The behavior should match numerical type coercion and silently skip the
> processing or offer failure relationships on processors supporting EL
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #2914: NIFI-5448 Added failure relationship to UpdateAttributes t...
Github user mcgilman commented on the issue: https://github.com/apache/nifi/pull/2914 Looking at the changeset I see that a property was added to allow the user to opt-in to the new behavior. That should alleviate my concerns. Sorry for the confusion. Only other suggestion would be to change the name of "Stop Processing" `AllowableValue` to maybe "Penalize" as that is what the current behavior should be doing. ---
[GitHub] nifi issue #2914: NIFI-5448 Added failure relationship to UpdateAttributes t...
Github user mcgilman commented on the issue: https://github.com/apache/nifi/pull/2914 I would be very careful taking that option. Yes, it would ensure that users flows remain valid. However, I would question if that is the behavior that those users actually want. If the new relationship were auto-terminated that would mean that any data routed there would be DROPped. Folks upgrading may not realize this and it could lead to data loss. This concern was brought up in an old JIRA [1]. At that time, the solution was to penalize the flowfile and rollback the session. Since this processor is so widely used, it may warrant a DISCUSS thread so the community can consider the options and when they may be appropriate to introduce. [1] https://issues.apache.org/jira/browse/NIFI-11 ---
[jira] [Commented] (NIFI-5448) Failed EL date parsing live-locks processors without a failure relationship
[
https://issues.apache.org/jira/browse/NIFI-5448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558864#comment-16558864
]
ASF GitHub Bot commented on NIFI-5448:
--
Github user mcgilman commented on the issue:
https://github.com/apache/nifi/pull/2914
I would be very careful taking that option. Yes, it would ensure that users
flows remain valid. However, I would question if that is the behavior that
those users actually want. If the new relationship were auto-terminated that
would mean that any data routed there would be DROPped. Folks upgrading may not
realize this and it could lead to data loss.
This concern was brought up in an old JIRA [1]. At that time, the solution
was to penalize the flowfile and rollback the session. Since this processor is
so widely used, it may warrant a DISCUSS thread so the community can consider
the options and when they may be appropriate to introduce.
[1] https://issues.apache.org/jira/browse/NIFI-11
> Failed EL date parsing live-locks processors without a failure relationship
> ---
>
> Key: NIFI-5448
> URL: https://issues.apache.org/jira/browse/NIFI-5448
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: David Koster
>Assignee: Mike Thomsen
>Priority: Major
>
> Processors that utilize the Expression Language need to always present a
> failure relationship.
> If a processor with only a success relationship, for example UpdateAttribute,
> utilizes the expression language to perform type coercion to a date and
> fails, the processor will be unable to dispose of the FlowFile and remain
> blocked indefinitely.
> Recreation flow:
> GenerateFlowFile -> Update Attribute #1 -> Update Attribute #2 -> Anything
> Update Attribute #1 - test = "Hello World"
> Update Attribute #2 - test = ${test:toDate('-MM-dd')}
>
> Generates an IllegalAttributeException on UpdateAttribute.
>
> The behavior should match numerical type coercion and silently skip the
> processing or offer failure relationships on processors supporting EL
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi-minifi-cpp issue #347: MINIFIPP-514 Incorporated regex property validat...
Github user achristianson commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/347 @apiri This was merged on the basis of "I'll provide an implicit agreement to move forward." MINIFICPP-577 has been created to refactor the Property constructor to reduce the # of arguments. I don't see that it was actually clarified in the conversation as to what the issue was, but that is my best guess. Pleas update ticket if incorrect. ---
[jira] [Created] (MINIFICPP-577) Come up with a better Property construction interface
Andrew Christianson created MINIFICPP-577: - Summary: Come up with a better Property construction interface Key: MINIFICPP-577 URL: https://issues.apache.org/jira/browse/MINIFICPP-577 Project: NiFi MiNiFi C++ Issue Type: Bug Reporter: Andrew Christianson Assignee: Andrew Christianson There are getting to be too many constructor parameters. Refactor/break this down a bit to make it less painful to construct a property having validation options. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-5449) RecordPath Language should allow base64 encode and decode
[ https://issues.apache.org/jira/browse/NIFI-5449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-5449: -- Assignee: Matt Burgess > RecordPath Language should allow base64 encode and decode > - > > Key: NIFI-5449 > URL: https://issues.apache.org/jira/browse/NIFI-5449 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Dalibor Frívaldský >Assignee: Matt Burgess >Priority: Minor > > Currently there is no way to encode/decode a record field using base64 > encoder/decoder. Without this functionality it is, for example, impossible to > store data into databases that require base64 representation of varbinary > values (e.g. Apache Phoenix). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5418) Remove duplicated jackson-databind dependency from nifi-framework-cluster
[
https://issues.apache.org/jira/browse/NIFI-5418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sivaprasanna Sethuraman updated NIFI-5418:
--
Resolution: Fixed
Fix Version/s: 1.8.0
Status: Resolved (was: Patch Available)
> Remove duplicated jackson-databind dependency from nifi-framework-cluster
> -
>
> Key: NIFI-5418
> URL: https://issues.apache.org/jira/browse/NIFI-5418
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Jan Hentschel
>Priority: Trivial
> Fix For: 1.8.0
>
>
> Maven complains a duplicated {{jackson-databind}} dependency in the
> {{nifi-framework-cluster}}, one with the hard-coded version 2.9.4 and one,
> which gets it from the property resulting in the version 2.9.5. The
> dependency with the hard-coded version should be removed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (NIFI-5083) Support for Expression Language in Group ID field of Kafka Consumer Processors
[ https://issues.apache.org/jira/browse/NIFI-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivaprasanna Sethuraman resolved NIFI-5083. --- Resolution: Fixed Fix Version/s: 1.8.0 > Support for Expression Language in Group ID field of Kafka Consumer Processors > -- > > Key: NIFI-5083 > URL: https://issues.apache.org/jira/browse/NIFI-5083 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.6.0 >Reporter: Robert Bruno >Priority: Minor > Labels: kafka > Fix For: 1.8.0 > > Attachments: NIFI-5083.mp4 > > > I have the use case where I have two separate NiFi clusters reading data from > the same topics but need to have unique group ids so they each get separate > copies of the data. It would be useful if the group id field supported > expression language so I could use a processor group variable to define the > group id to use. I have a large enough number of kafka consumers to make > this tedious to manually change. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (NIFI-5440) GetMongo should put database and collection names in flowfile attributes
[ https://issues.apache.org/jira/browse/NIFI-5440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivaprasanna Sethuraman resolved NIFI-5440. --- Resolution: Fixed Fix Version/s: 1.8.0 > GetMongo should put database and collection names in flowfile attributes > > > Key: NIFI-5440 > URL: https://issues.apache.org/jira/browse/NIFI-5440 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Minor > Fix For: 1.8.0 > > > GetMongo has EL support for database and collection names, so the same > GetMongo instance can service every database and collection in a MongoDB > cluster. Therefore, GetMongo should provide the source database and > collection names as attributes on the flowfiles so downstream processors have > access to that information for making decisions. This is particularly > important for RouteOnAttribute. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5440) GetMongo should put database and collection names in flowfile attributes
[ https://issues.apache.org/jira/browse/NIFI-5440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558697#comment-16558697 ] ASF GitHub Bot commented on NIFI-5440: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2906 > GetMongo should put database and collection names in flowfile attributes > > > Key: NIFI-5440 > URL: https://issues.apache.org/jira/browse/NIFI-5440 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Minor > > GetMongo has EL support for database and collection names, so the same > GetMongo instance can service every database and collection in a MongoDB > cluster. Therefore, GetMongo should provide the source database and > collection names as attributes on the flowfiles so downstream processors have > access to that information for making decisions. This is particularly > important for RouteOnAttribute. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5440) GetMongo should put database and collection names in flowfile attributes
[ https://issues.apache.org/jira/browse/NIFI-5440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558696#comment-16558696 ] ASF subversion and git services commented on NIFI-5440: --- Commit b1f78d58a6f5250b4f4e6873b03459a8e987407a in nifi's branch refs/heads/master from [~mike.thomsen] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=b1f78d5 ] NIFI-5440 Added db and collection attributes to GetMongo output. This closes #2906 Signed-off-by: zenfenan > GetMongo should put database and collection names in flowfile attributes > > > Key: NIFI-5440 > URL: https://issues.apache.org/jira/browse/NIFI-5440 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Minor > > GetMongo has EL support for database and collection names, so the same > GetMongo instance can service every database and collection in a MongoDB > cluster. Therefore, GetMongo should provide the source database and > collection names as attributes on the flowfiles so downstream processors have > access to that information for making decisions. This is particularly > important for RouteOnAttribute. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2906: NIFI-5440 Added db and collection attributes to Get...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2906 ---
[jira] [Commented] (NIFI-5418) Remove duplicated jackson-databind dependency from nifi-framework-cluster
[
https://issues.apache.org/jira/browse/NIFI-5418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558675#comment-16558675
]
ASF GitHub Bot commented on NIFI-5418:
--
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/2885
> Remove duplicated jackson-databind dependency from nifi-framework-cluster
> -
>
> Key: NIFI-5418
> URL: https://issues.apache.org/jira/browse/NIFI-5418
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Jan Hentschel
>Priority: Trivial
>
> Maven complains a duplicated {{jackson-databind}} dependency in the
> {{nifi-framework-cluster}}, one with the hard-coded version 2.9.4 and one,
> which gets it from the property resulting in the version 2.9.5. The
> dependency with the hard-coded version should be removed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5418) Remove duplicated jackson-databind dependency from nifi-framework-cluster
[
https://issues.apache.org/jira/browse/NIFI-5418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558674#comment-16558674
]
ASF subversion and git services commented on NIFI-5418:
---
Commit 610cbb66b2e644a782f4714e749da932131d4d9d in nifi's branch
refs/heads/master from [~Jan Hentschel]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=610cbb6 ]
NIFI-5418 Removed duplicated jackson-databind dependency from
nifi-framework-cluster
This closes #2885
Signed-off-by: zenfenan
> Remove duplicated jackson-databind dependency from nifi-framework-cluster
> -
>
> Key: NIFI-5418
> URL: https://issues.apache.org/jira/browse/NIFI-5418
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Jan Hentschel
>Priority: Trivial
>
> Maven complains a duplicated {{jackson-databind}} dependency in the
> {{nifi-framework-cluster}}, one with the hard-coded version 2.9.4 and one,
> which gets it from the property resulting in the version 2.9.5. The
> dependency with the hard-coded version should be removed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2885: NIFI-5418 Removed duplicated jackson-databind depen...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2885 ---
[jira] [Created] (NIFI-5467) Add option to HandleHttpRequest/Response allowing multiple params with the same key
Matt Burgess created NIFI-5467: -- Summary: Add option to HandleHttpRequest/Response allowing multiple params with the same key Key: NIFI-5467 URL: https://issues.apache.org/jira/browse/NIFI-5467 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: Matt Burgess In HttpHandleRequest, the query parameters are added as attributes to the outgoing flow file(s). However, even though there is no standard for allowing multiple parameters with the same key, it is often found in various REST APIs, such as: http://host:port/query?id=a=b In this case there will be a single attribute "http.query.param.id" set to b (the "a" is overwritten). It would be nice to have a property called "Allow multiple parameters" or something, and if true, multiple parameters will result in a comma-delimited list of values rather than a single value. This same approach may be able to be applied to HandleHttpResponse as well, but if not, the title and description of this Jira can be updated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5440) GetMongo should put database and collection names in flowfile attributes
[ https://issues.apache.org/jira/browse/NIFI-5440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558648#comment-16558648 ] ASF GitHub Bot commented on NIFI-5440: -- Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2906 Reviewing.. > GetMongo should put database and collection names in flowfile attributes > > > Key: NIFI-5440 > URL: https://issues.apache.org/jira/browse/NIFI-5440 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Minor > > GetMongo has EL support for database and collection names, so the same > GetMongo instance can service every database and collection in a MongoDB > cluster. Therefore, GetMongo should provide the source database and > collection names as attributes on the flowfiles so downstream processors have > access to that information for making decisions. This is particularly > important for RouteOnAttribute. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2906: NIFI-5440 Added db and collection attributes to GetMongo o...
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2906 Reviewing.. ---
[jira] [Created] (NIFI-5466) Improve efficiency of RingBufferEventRepository
Mark Payne created NIFI-5466:
Summary: Improve efficiency of RingBufferEventRepository
Key: NIFI-5466
URL: https://issues.apache.org/jira/browse/NIFI-5466
Project: Apache NiFi
Issue Type: Improvement
Components: Core Framework
Reporter: Mark Payne
Assignee: Mark Payne
The implementation of the FlowFileEventRepository (i.e., the
RingBufferEventRepository) is implemented in such a way that 300 buckets are
kept for each component. Each bucket holds a set of 16+ stats (bytes read,
bytes written, flowfiles in, flowfiles out, etc. as well as counters)
Each time that a user refreshes stats, we calculate the 5-minute window for
each component by iterating over all 300 buckets, then summing together the
values of the stats. So, for every 1,000 processors, we have 1000 * 300 * 16 =
4.8 MM calculations being performed each time that the user refreshes.
We can improve this significantly. To do this, we can keep a 'running tally'
for each stat. Every time that we update a bucket in the
FlowFileEventRepository, we should similarly update the running tally. Each
time that we 'replace' a bucket in the FlowFileEventRepository, we should
subtract from the running tally the values in the bucket, in addition to adding
in the new value. This would result in amortizing the cost of calculation over
the 5-minute period and means that every time that we get the processor stats
we could do so by simply performing a 'get' operation without any calculations.
But wait, there's more! The EventSumValue class has an {{add(FlowFileEvent)}}
method. If the EventSumValue were to simply keep a boolean value indicating
whether or not it was empty, then when adding to an empty bucket we could avoid
the calculations all together, and when replacing an empty bucket we could
avoid the calculation against the running tally as well.
Similarly, in the VolatileComponentStatusRepository, when we perform a
'capture' we should detect any 'empty' stats objects and use a single static
instance for this, which would significantly reduce the amount of heap that is
used to store metrics.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5413) Write additional documentation for PutMongoRecord
[ https://issues.apache.org/jira/browse/NIFI-5413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558629#comment-16558629 ] ASF GitHub Bot commented on NIFI-5413: -- Github user zenfenan commented on a diff in the pull request: https://github.com/apache/nifi/pull/2881#discussion_r205535512 --- Diff: nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/resources/docs/org.apache.nifi.processors.mongodb.PutMongoRecord/additionalDetails.html --- @@ -0,0 +1,37 @@ + + + + + +PutMongoRecord + + + + + +Description: + +This processor is a record-aware processor for inserting data into MongoDB. It uses a configured record reader and +schema to read an incoming record set from the body of a flowfile and then inserts batches of those records into +a configured MongoDB collection. + + +This processor does not support updates, deletes or upserts. The number of documents to insert at a time is controlled +by the Insert Batch Size configuration property. This value should be set to a reasonable size to ensure --- End diff -- Nitpick: Since these are all text and we are not actually introducing any cool HTML formatting, would it not be enough if we just add it to the processor's `@CapabilityDescription` itself? > Write additional documentation for PutMongoRecord > - > > Key: NIFI-5413 > URL: https://issues.apache.org/jira/browse/NIFI-5413 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2881: NIFI-5413 Added additional documentation for PutMon...
Github user zenfenan commented on a diff in the pull request: https://github.com/apache/nifi/pull/2881#discussion_r205535512 --- Diff: nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/resources/docs/org.apache.nifi.processors.mongodb.PutMongoRecord/additionalDetails.html --- @@ -0,0 +1,37 @@ + + + + + +PutMongoRecord + + + + + +Description: + +This processor is a record-aware processor for inserting data into MongoDB. It uses a configured record reader and +schema to read an incoming record set from the body of a flowfile and then inserts batches of those records into +a configured MongoDB collection. + + +This processor does not support updates, deletes or upserts. The number of documents to insert at a time is controlled +by the Insert Batch Size configuration property. This value should be set to a reasonable size to ensure --- End diff -- Nitpick: Since these are all text and we are not actually introducing any cool HTML formatting, would it not be enough if we just add it to the processor's `@CapabilityDescription` itself? ---
[jira] [Commented] (NIFI-5448) Failed EL date parsing live-locks processors without a failure relationship
[
https://issues.apache.org/jira/browse/NIFI-5448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558617#comment-16558617
]
ASF GitHub Bot commented on NIFI-5448:
--
Github user pvillard31 commented on the issue:
https://github.com/apache/nifi/pull/2914
@MikeThomsen oh nice! completely forgot about this option! my bad
> Failed EL date parsing live-locks processors without a failure relationship
> ---
>
> Key: NIFI-5448
> URL: https://issues.apache.org/jira/browse/NIFI-5448
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: David Koster
>Assignee: Mike Thomsen
>Priority: Major
>
> Processors that utilize the Expression Language need to always present a
> failure relationship.
> If a processor with only a success relationship, for example UpdateAttribute,
> utilizes the expression language to perform type coercion to a date and
> fails, the processor will be unable to dispose of the FlowFile and remain
> blocked indefinitely.
> Recreation flow:
> GenerateFlowFile -> Update Attribute #1 -> Update Attribute #2 -> Anything
> Update Attribute #1 - test = "Hello World"
> Update Attribute #2 - test = ${test:toDate('-MM-dd')}
>
> Generates an IllegalAttributeException on UpdateAttribute.
>
> The behavior should match numerical type coercion and silently skip the
> processing or offer failure relationships on processors supporting EL
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #2914: NIFI-5448 Added failure relationship to UpdateAttributes t...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/2914 @MikeThomsen oh nice! completely forgot about this option! my bad ---
[jira] [Commented] (NIFI-5448) Failed EL date parsing live-locks processors without a failure relationship
[
https://issues.apache.org/jira/browse/NIFI-5448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558612#comment-16558612
]
ASF GitHub Bot commented on NIFI-5448:
--
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2914
@pvillard31 @zenfenan it's an autoterminating relationship by default and
the default behavior is maintain the status quo.
> Failed EL date parsing live-locks processors without a failure relationship
> ---
>
> Key: NIFI-5448
> URL: https://issues.apache.org/jira/browse/NIFI-5448
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: David Koster
>Assignee: Mike Thomsen
>Priority: Major
>
> Processors that utilize the Expression Language need to always present a
> failure relationship.
> If a processor with only a success relationship, for example UpdateAttribute,
> utilizes the expression language to perform type coercion to a date and
> fails, the processor will be unable to dispose of the FlowFile and remain
> blocked indefinitely.
> Recreation flow:
> GenerateFlowFile -> Update Attribute #1 -> Update Attribute #2 -> Anything
> Update Attribute #1 - test = "Hello World"
> Update Attribute #2 - test = ${test:toDate('-MM-dd')}
>
> Generates an IllegalAttributeException on UpdateAttribute.
>
> The behavior should match numerical type coercion and silently skip the
> processing or offer failure relationships on processors supporting EL
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #2914: NIFI-5448 Added failure relationship to UpdateAttributes t...
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2914 @pvillard31 @zenfenan it's an autoterminating relationship by default and the default behavior is maintain the status quo. ---
[jira] [Commented] (NIFI-5413) Write additional documentation for PutMongoRecord
[ https://issues.apache.org/jira/browse/NIFI-5413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558572#comment-16558572 ] ASF GitHub Bot commented on NIFI-5413: -- Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2881 Reviewing.. > Write additional documentation for PutMongoRecord > - > > Key: NIFI-5413 > URL: https://issues.apache.org/jira/browse/NIFI-5413 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2881: NIFI-5413 Added additional documentation for PutMongoRecor...
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2881 Reviewing.. ---
[jira] [Commented] (NIFI-5083) Support for Expression Language in Group ID field of Kafka Consumer Processors
[ https://issues.apache.org/jira/browse/NIFI-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558550#comment-16558550 ] ASF GitHub Bot commented on NIFI-5083: -- Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2913 @snagafritz Hey.. I forgot to close this PR while merging. It is now merged to the master. Can you manually close this PR? > Support for Expression Language in Group ID field of Kafka Consumer Processors > -- > > Key: NIFI-5083 > URL: https://issues.apache.org/jira/browse/NIFI-5083 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.6.0 >Reporter: Robert Bruno >Priority: Minor > Labels: kafka > Attachments: NIFI-5083.mp4 > > > I have the use case where I have two separate NiFi clusters reading data from > the same topics but need to have unique group ids so they each get separate > copies of the data. It would be useful if the group id field supported > expression language so I could use a processor group variable to define the > group id to use. I have a large enough number of kafka consumers to make > this tedious to manually change. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2913: NIFI-5083: enabled EL support for processor configuration ...
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2913 @snagafritz Hey.. I forgot to close this PR while merging. It is now merged to the master. Can you manually close this PR? ---
[jira] [Commented] (NIFI-5083) Support for Expression Language in Group ID field of Kafka Consumer Processors
[ https://issues.apache.org/jira/browse/NIFI-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558537#comment-16558537 ] ASF GitHub Bot commented on NIFI-5083: -- Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2913 +1 LGTM. Merged to master. Thanks @snagafritz > Support for Expression Language in Group ID field of Kafka Consumer Processors > -- > > Key: NIFI-5083 > URL: https://issues.apache.org/jira/browse/NIFI-5083 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.6.0 >Reporter: Robert Bruno >Priority: Minor > Labels: kafka > Attachments: NIFI-5083.mp4 > > > I have the use case where I have two separate NiFi clusters reading data from > the same topics but need to have unique group ids so they each get separate > copies of the data. It would be useful if the group id field supported > expression language so I could use a processor group variable to define the > group id to use. I have a large enough number of kafka consumers to make > this tedious to manually change. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2913: NIFI-5083: enabled EL support for processor configuration ...
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2913 +1 LGTM. Merged to master. Thanks @snagafritz ---
[jira] [Commented] (NIFI-5083) Support for Expression Language in Group ID field of Kafka Consumer Processors
[ https://issues.apache.org/jira/browse/NIFI-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558534#comment-16558534 ] ASF subversion and git services commented on NIFI-5083: --- Commit 4baf35db5ba296d246e3491e8d8c84408306f685 in nifi's branch refs/heads/master from [~snagafritz] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=4baf35d ] NIFI-5083: enabled EL support for processor configuration of Kafka consumer `group.id` property Signed-off-by: zenfenan > Support for Expression Language in Group ID field of Kafka Consumer Processors > -- > > Key: NIFI-5083 > URL: https://issues.apache.org/jira/browse/NIFI-5083 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.6.0 >Reporter: Robert Bruno >Priority: Minor > Labels: kafka > Attachments: NIFI-5083.mp4 > > > I have the use case where I have two separate NiFi clusters reading data from > the same topics but need to have unique group ids so they each get separate > copies of the data. It would be useful if the group id field supported > expression language so I could use a processor group variable to define the > group id to use. I have a large enough number of kafka consumers to make > this tedious to manually change. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5083) Support for Expression Language in Group ID field of Kafka Consumer Processors
[ https://issues.apache.org/jira/browse/NIFI-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558462#comment-16558462 ] ASF GitHub Bot commented on NIFI-5083: -- Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2913 Reviewing.. > Support for Expression Language in Group ID field of Kafka Consumer Processors > -- > > Key: NIFI-5083 > URL: https://issues.apache.org/jira/browse/NIFI-5083 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.6.0 >Reporter: Robert Bruno >Priority: Minor > Labels: kafka > Attachments: NIFI-5083.mp4 > > > I have the use case where I have two separate NiFi clusters reading data from > the same topics but need to have unique group ids so they each get separate > copies of the data. It would be useful if the group id field supported > expression language so I could use a processor group variable to define the > group id to use. I have a large enough number of kafka consumers to make > this tedious to manually change. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2913: NIFI-5083: enabled EL support for processor configuration ...
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2913 Reviewing.. ---
[jira] [Commented] (NIFI-5443) Improve cluster configuration for dynamic scaling
[
https://issues.apache.org/jira/browse/NIFI-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558440#comment-16558440
]
Joseph Witt commented on NIFI-5443:
---
[~alopresto] just to be clear it is designed to make adding nodes easy and this
has proven true over the years. It is also designed to make upgrades easy,
downgrades easy. As new features are added though these aren't always taking
all these aspects into consideration and further sometimes the approaches taken
just dont' work that well. As new authorization/authentication styles are
adding they can strain these as well.
> Improve cluster configuration for dynamic scaling
> -
>
> Key: NIFI-5443
> URL: https://issues.apache.org/jira/browse/NIFI-5443
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Andy LoPresto
>Priority: Critical
> Labels: cluster, docker, kubernetes, rkt, scale, security
>
> Currently, NiFi is designed for static clusters, with frequent references in
> configuration files to a priori knowledge of node hostnames, ports, etc.
> Efforts should be taken to make NiFi easier to dynamically scale. This can
> involve containerization improvements via Docker/rkt, deployment improvements
> via Kubernetes, and abstraction of the configuration values needed to stand
> up the cluster. A node should be able to join the cluster, and, given the
> correct keystore and truststore, immediately communicate with other existing
> nodes in the cluster without requiring direct configuration changes to them,
> or a restart of any node.
> * {{authorizers.xml}}
> * node identities
> * permissions ({{RW}} on {{/proxy}})
> * ZooKeeper configuration
> * etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5443) Improve cluster configuration for dynamic scaling
[
https://issues.apache.org/jira/browse/NIFI-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558424#comment-16558424
]
Andy LoPresto commented on NIFI-5443:
-
[~prashanv] I'm not sure I understand the problem with providing an empty
{{authorizers.xml}} file when adding a node to the cluster. In that case, it
will inherit the contents from the cluster.
This ticket will encompass making this process easier, but what is the blocker
for the current mechanism?
> Improve cluster configuration for dynamic scaling
> -
>
> Key: NIFI-5443
> URL: https://issues.apache.org/jira/browse/NIFI-5443
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Andy LoPresto
>Priority: Critical
> Labels: cluster, docker, kubernetes, rkt, scale, security
>
> Currently, NiFi is designed for static clusters, with frequent references in
> configuration files to a priori knowledge of node hostnames, ports, etc.
> Efforts should be taken to make NiFi easier to dynamically scale. This can
> involve containerization improvements via Docker/rkt, deployment improvements
> via Kubernetes, and abstraction of the configuration values needed to stand
> up the cluster. A node should be able to join the cluster, and, given the
> correct keystore and truststore, immediately communicate with other existing
> nodes in the cluster without requiring direct configuration changes to them,
> or a restart of any node.
> * {{authorizers.xml}}
> * node identities
> * permissions ({{RW}} on {{/proxy}})
> * ZooKeeper configuration
> * etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5465) HandleHttpRequest times out requests if it goes more than 30 seconds without receiving data, regardless of the configured Request Timeout
[
https://issues.apache.org/jira/browse/NIFI-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558418#comment-16558418
]
ASF GitHub Bot commented on NIFI-5465:
--
Github user markap14 commented on the issue:
https://github.com/apache/nifi/pull/2918
An easy way to test the PR is to try running the following code (updating
the URL in the second line of code to point to a running HandleHttpRequest
processor):
```
public static void main(final String[] args) throws Exception {
final URL url = new URL("http://localhost:8153/test;);
final URLConnection conn = url.openConnection();
final HttpURLConnection connection = (HttpURLConnection) conn;
connection.setRequestMethod("POST");
connection.setDoOutput(true);
connection.setFixedLengthStreamingMode(1_000_000);
connection.connect();
final OutputStream out = connection.getOutputStream();
for (int i=0; i < 500_000; i++) {
out.write('A');
}
System.out.println("Sleeping...");
Thread.sleep(35000);
for (int i=0; i < 500_000; i++) {
out.write('A');
}
// read the output from the server
final BufferedReader reader = new BufferedReader(new
InputStreamReader(connection.getInputStream()));
final StringBuilder stringBuilder = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null) {
stringBuilder.append(line + "\n");
}
System.out.println(stringBuilder.toString());
}
```
Before the PR, this caused an Exception after sleeping and NiFi did not
handle the request. After the PR, NiFI handles it properly and returns the
expected response.
> HandleHttpRequest times out requests if it goes more than 30 seconds without
> receiving data, regardless of the configured Request Timeout
> -
>
> Key: NIFI-5465
> URL: https://issues.apache.org/jira/browse/NIFI-5465
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.8.0
>
>
> The HTTP Context Map allows the user to configure the "Request Timeout" but
> if data is written to the HTTP Request, then pauses for more than 30 seconds,
> then writes more data, then the request will timeout. We should allow request
> to go up to the configured "Request Timeout" before throwing a
> TimeoutException. The stack trace seen is:
> {code:java}
> org.apache.nifi.processor.exception.FlowFileAccessException: Failed to import
> data from
> HttpInputOverHTTP@5f73086c[c=16384,q=0,[0]=null,s=ERROR:java.util.concurrent.TimeoutException:
> Idle timeout expired: 30003/3 ms] for
> StandardFlowFileRecord[uuid=2c04929e-cd59-46b6-8612-c437c0230591,claim=,offset=0,name=84593375189600,size=0]
> due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable
> to create ContentClaim due to java.io.IOException:
> java.util.concurrent.TimeoutException: Idle timeout expired: 30003/3 ms;
> rolling back session: {}
> org.apache.nifi.processor.exception.FlowFileAccessException: Failed to import
> data from
> HttpInputOverHTTP@5f73086c[c=16384,q=0,[0]=null,s=ERROR:java.util.concurrent.TimeoutException:
> Idle timeout expired: 30003/3 ms] for
> StandardFlowFileRecord[uuid=2c04929e-cd59-46b6-8612-c437c0230591,claim=,offset=0,name=84593375189600,size=0]
> due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable
> to create ContentClaim due to java.io.IOException:
> java.util.concurrent.TimeoutException: Idle timeout expired: 30003/3 ms
> at
> org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:2942)
>
> at
> org.apache.nifi.processors.standard.HandleHttpRequest.onTrigger(HandleHttpRequest.java:507)
>
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
>
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
>
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
>
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at
[GitHub] nifi issue #2918: NIFI-5465: Set the Idle Timeout on jetty connectors to the...
Github user markap14 commented on the issue:
https://github.com/apache/nifi/pull/2918
An easy way to test the PR is to try running the following code (updating
the URL in the second line of code to point to a running HandleHttpRequest
processor):
```
public static void main(final String[] args) throws Exception {
final URL url = new URL("http://localhost:8153/test;);
final URLConnection conn = url.openConnection();
final HttpURLConnection connection = (HttpURLConnection) conn;
connection.setRequestMethod("POST");
connection.setDoOutput(true);
connection.setFixedLengthStreamingMode(1_000_000);
connection.connect();
final OutputStream out = connection.getOutputStream();
for (int i=0; i < 500_000; i++) {
out.write('A');
}
System.out.println("Sleeping...");
Thread.sleep(35000);
for (int i=0; i < 500_000; i++) {
out.write('A');
}
// read the output from the server
final BufferedReader reader = new BufferedReader(new
InputStreamReader(connection.getInputStream()));
final StringBuilder stringBuilder = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null) {
stringBuilder.append(line + "\n");
}
System.out.println(stringBuilder.toString());
}
```
Before the PR, this caused an Exception after sleeping and NiFi did not
handle the request. After the PR, NiFI handles it properly and returns the
expected response.
---
[jira] [Updated] (NIFI-5465) HandleHttpRequest times out requests if it goes more than 30 seconds without receiving data, regardless of the configured Request Timeout
[
https://issues.apache.org/jira/browse/NIFI-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-5465:
-
Fix Version/s: 1.8.0
Status: Patch Available (was: Open)
> HandleHttpRequest times out requests if it goes more than 30 seconds without
> receiving data, regardless of the configured Request Timeout
> -
>
> Key: NIFI-5465
> URL: https://issues.apache.org/jira/browse/NIFI-5465
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.8.0
>
>
> The HTTP Context Map allows the user to configure the "Request Timeout" but
> if data is written to the HTTP Request, then pauses for more than 30 seconds,
> then writes more data, then the request will timeout. We should allow request
> to go up to the configured "Request Timeout" before throwing a
> TimeoutException. The stack trace seen is:
> {code:java}
> org.apache.nifi.processor.exception.FlowFileAccessException: Failed to import
> data from
> HttpInputOverHTTP@5f73086c[c=16384,q=0,[0]=null,s=ERROR:java.util.concurrent.TimeoutException:
> Idle timeout expired: 30003/3 ms] for
> StandardFlowFileRecord[uuid=2c04929e-cd59-46b6-8612-c437c0230591,claim=,offset=0,name=84593375189600,size=0]
> due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable
> to create ContentClaim due to java.io.IOException:
> java.util.concurrent.TimeoutException: Idle timeout expired: 30003/3 ms;
> rolling back session: {}
> org.apache.nifi.processor.exception.FlowFileAccessException: Failed to import
> data from
> HttpInputOverHTTP@5f73086c[c=16384,q=0,[0]=null,s=ERROR:java.util.concurrent.TimeoutException:
> Idle timeout expired: 30003/3 ms] for
> StandardFlowFileRecord[uuid=2c04929e-cd59-46b6-8612-c437c0230591,claim=,offset=0,name=84593375189600,size=0]
> due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable
> to create ContentClaim due to java.io.IOException:
> java.util.concurrent.TimeoutException: Idle timeout expired: 30003/3 ms
> at
> org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:2942)
>
> at
> org.apache.nifi.processors.standard.HandleHttpRequest.onTrigger(HandleHttpRequest.java:507)
>
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
>
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
>
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
>
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.nifi.processor.exception.FlowFileAccessException:
> Unable to create ContentClaim due to java.io.IOException:
> java.util.concurrent.TimeoutException: Idle timeout expired: 30003/3 ms
> at
> org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:2935)
>
> ... 13 common frames omitted
> Caused by: java.io.IOException: java.util.concurrent.TimeoutException: Idle
> timeout expired: 30003/3 ms
> at
> org.eclipse.jetty.server.HttpInput$ErrorState.noContent(HttpInput.java:1047)
> at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:307)
> at java.io.InputStream.read(Unknown Source)
> at org.apache.nifi.stream.io.StreamUtils.copy(StreamUtils.java:35)
> at
> org.apache.nifi.controller.repository.FileSystemRepository.importFrom(FileSystemRepository.java:734)
>
> at
> org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:2932)
>
> ... 13 common frames omitted
> Caused by: java.util.concurrent.TimeoutException: Idle timeout expired:
> 30003/3 ms
> at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166)
> at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50)
> at
[jira] [Commented] (NIFI-5465) HandleHttpRequest times out requests if it goes more than 30 seconds without receiving data, regardless of the configured Request Timeout
[
https://issues.apache.org/jira/browse/NIFI-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558416#comment-16558416
]
ASF GitHub Bot commented on NIFI-5465:
--
GitHub user markap14 opened a pull request:
https://github.com/apache/nifi/pull/2918
NIFI-5465: Set the Idle Timeout on jetty connectors to the same as th…
…e Request Timeout.
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with NIFI- where is the JIRA number
you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/markap14/nifi NIFI-5465
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/2918.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2918
commit 68e3b481ccd810555a9798cd3e8985f23001737a
Author: Mark Payne
Date: 2018-07-26T15:17:28Z
NIFI-5465: Set the Idle Timeout on jetty connectors to the same as the
Request Timeout.
> HandleHttpRequest times out requests if it goes more than 30 seconds without
> receiving data, regardless of the configured Request Timeout
> -
>
> Key: NIFI-5465
> URL: https://issues.apache.org/jira/browse/NIFI-5465
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
>
> The HTTP Context Map allows the user to configure the "Request Timeout" but
> if data is written to the HTTP Request, then pauses for more than 30 seconds,
> then writes more data, then the request will timeout. We should allow request
> to go up to the configured "Request Timeout" before throwing a
> TimeoutException. The stack trace seen is:
> {code:java}
> org.apache.nifi.processor.exception.FlowFileAccessException: Failed to import
> data from
> HttpInputOverHTTP@5f73086c[c=16384,q=0,[0]=null,s=ERROR:java.util.concurrent.TimeoutException:
> Idle timeout expired: 30003/3 ms] for
> StandardFlowFileRecord[uuid=2c04929e-cd59-46b6-8612-c437c0230591,claim=,offset=0,name=84593375189600,size=0]
> due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable
> to create ContentClaim due to java.io.IOException:
> java.util.concurrent.TimeoutException: Idle timeout expired: 30003/3 ms;
> rolling back session: {}
> org.apache.nifi.processor.exception.FlowFileAccessException: Failed to import
> data from
> HttpInputOverHTTP@5f73086c[c=16384,q=0,[0]=null,s=ERROR:java.util.concurrent.TimeoutException:
> Idle timeout expired: 30003/3 ms] for
> StandardFlowFileRecord[uuid=2c04929e-cd59-46b6-8612-c437c0230591,claim=,offset=0,name=84593375189600,size=0]
> due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable
> to create ContentClaim due to java.io.IOException:
> java.util.concurrent.TimeoutException: Idle timeout expired: 30003/3 ms
> at
>
[GitHub] nifi pull request #2918: NIFI-5465: Set the Idle Timeout on jetty connectors...
GitHub user markap14 opened a pull request: https://github.com/apache/nifi/pull/2918 NIFI-5465: Set the Idle Timeout on jetty connectors to the same as th⦠â¦e Request Timeout. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/markap14/nifi NIFI-5465 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2918.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2918 commit 68e3b481ccd810555a9798cd3e8985f23001737a Author: Mark Payne Date: 2018-07-26T15:17:28Z NIFI-5465: Set the Idle Timeout on jetty connectors to the same as the Request Timeout. ---
[jira] [Created] (NIFI-5465) HandleHttpRequest times out requests if it goes more than 30 seconds without receiving data, regardless of the configured Request Timeout
Mark Payne created NIFI-5465:
Summary: HandleHttpRequest times out requests if it goes more than
30 seconds without receiving data, regardless of the configured Request Timeout
Key: NIFI-5465
URL: https://issues.apache.org/jira/browse/NIFI-5465
Project: Apache NiFi
Issue Type: Bug
Components: Extensions
Reporter: Mark Payne
Assignee: Mark Payne
The HTTP Context Map allows the user to configure the "Request Timeout" but if
data is written to the HTTP Request, then pauses for more than 30 seconds, then
writes more data, then the request will timeout. We should allow request to go
up to the configured "Request Timeout" before throwing a TimeoutException. The
stack trace seen is:
{code:java}
org.apache.nifi.processor.exception.FlowFileAccessException: Failed to import
data from
HttpInputOverHTTP@5f73086c[c=16384,q=0,[0]=null,s=ERROR:java.util.concurrent.TimeoutException:
Idle timeout expired: 30003/3 ms] for
StandardFlowFileRecord[uuid=2c04929e-cd59-46b6-8612-c437c0230591,claim=,offset=0,name=84593375189600,size=0]
due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable to
create ContentClaim due to java.io.IOException:
java.util.concurrent.TimeoutException: Idle timeout expired: 30003/3 ms;
rolling back session: {}
org.apache.nifi.processor.exception.FlowFileAccessException: Failed to import
data from
HttpInputOverHTTP@5f73086c[c=16384,q=0,[0]=null,s=ERROR:java.util.concurrent.TimeoutException:
Idle timeout expired: 30003/3 ms] for
StandardFlowFileRecord[uuid=2c04929e-cd59-46b6-8612-c437c0230591,claim=,offset=0,name=84593375189600,size=0]
due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable to
create ContentClaim due to java.io.IOException:
java.util.concurrent.TimeoutException: Idle timeout expired: 30003/3 ms
at
org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:2942)
at
org.apache.nifi.processors.standard.HandleHttpRequest.onTrigger(HandleHttpRequest.java:507)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.nifi.processor.exception.FlowFileAccessException: Unable
to create ContentClaim due to java.io.IOException:
java.util.concurrent.TimeoutException: Idle timeout expired: 30003/3 ms
at
org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:2935)
... 13 common frames omitted
Caused by: java.io.IOException: java.util.concurrent.TimeoutException: Idle
timeout expired: 30003/3 ms
at org.eclipse.jetty.server.HttpInput$ErrorState.noContent(HttpInput.java:1047)
at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:307)
at java.io.InputStream.read(Unknown Source)
at org.apache.nifi.stream.io.StreamUtils.copy(StreamUtils.java:35)
at
org.apache.nifi.controller.repository.FileSystemRepository.importFrom(FileSystemRepository.java:734)
at
org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:2932)
... 13 common frames omitted
Caused by: java.util.concurrent.TimeoutException: Idle timeout expired:
30003/3 ms
at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166)
at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown
Source)
... 4 common frames omitted{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-5464) Local Ports are able to be started without incoming connections, then use a lot of CPU
Mark Payne created NIFI-5464: Summary: Local Ports are able to be started without incoming connections, then use a lot of CPU Key: NIFI-5464 URL: https://issues.apache.org/jira/browse/NIFI-5464 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Mark Payne Currently, validation of a LocalPort simply checks that it has an outgoing connection. This means that it's possible to create an outgoing connection for a port but not have any incoming connections. When this happens, if the Port is started, the framework determines that it is a "source component" because it has no incoming connections. As such, it is never yielded via the "bored yield duration". This results in high CPU usage. The fix in this case, I believe, is to address the validation. We should not allow a Local Port to be enabled if it has no incoming connections. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFIREG-188) Login by hitting Enter in the password field in web UI
Julian Gimbel created NIFIREG-188: - Summary: Login by hitting Enter in the password field in web UI Key: NIFIREG-188 URL: https://issues.apache.org/jira/browse/NIFIREG-188 Project: NiFi Registry Issue Type: Improvement Affects Versions: 0.2.0, 0.1.0 Reporter: Julian Gimbel While logging in to Nifi Registry it should be possible to hit enter in the password field to login instead of using the mouse or tab on to the login button. Unfortunately I am not experienced enough with angular to do it myself. It is not as simple as just adding a tag around the dialog fields and buttons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFIREG-187) Add to Migration Guidance: Copy flow_storage directory
Julian Gimbel created NIFIREG-187:
-
Summary: Add to Migration Guidance: Copy flow_storage directory
Key: NIFIREG-187
URL: https://issues.apache.org/jira/browse/NIFIREG-187
Project: NiFi Registry
Issue Type: Improvement
Affects Versions: 0.2.0, 0.1.0
Reporter: Julian Gimbel
When we tried to migrate H2 Database from 0.1.0 to 0.2.0 we got the problem
that the nifi-registry-app.log showed that all our buckets flows and flow
snapshots were successfully copied from legacy to new database.
However when we wanted to import or refresh the repository we could select the
flows from the dropdown list, but got an error when we clicked on import.
The log showed the following:
{code:java}
2018-07-26 07:51:21,859 WARN [NiFi Registry Web Server-36]
o.a.n.registry.service.RegistryService The specified flow id
[d7d8ed4b-68e4-4d88-932c-486863a9f194] does not exist.
2018-07-26 07:51:21,860 INFO [NiFi Registry Web Server-36]
o.a.n.r.w.m.ResourceNotFoundExceptionMapper
org.apache.nifi.registry.exception.ResourceNotFoundException: The specified
flow ID does not exist in this bucket.. Returning Not Found response.{code}
Only after we found this conversation in the [Upgrade Mailing List
|http://apache-nifi-developer-list.39713.n7.nabble.com/Upgrading-NiFi-Registry-tp19098p19215.html]
we figured out that we also need to copy HOME/flow_storage directory as the
database only contains the metadata but the flow_storage contains the actual
data.
It might help people if this peace of information would be added to the
[migration
guidance|https://cwiki.apache.org/confluence/display/NIFIREG/Migration+Guidance].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5448) Failed EL date parsing live-locks processors without a failure relationship
[
https://issues.apache.org/jira/browse/NIFI-5448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558243#comment-16558243
]
ASF GitHub Bot commented on NIFI-5448:
--
Github user pvillard31 commented on the issue:
https://github.com/apache/nifi/pull/2914
@MikeThomsen - unfortunately, I believe this is a breaking change that will
need to wait for NiFi 2.0.0. Adding a relationship will make the processor
invalid after an upgrade and users will have to define what they want to do
with this new relationship.
> Failed EL date parsing live-locks processors without a failure relationship
> ---
>
> Key: NIFI-5448
> URL: https://issues.apache.org/jira/browse/NIFI-5448
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: David Koster
>Assignee: Mike Thomsen
>Priority: Major
>
> Processors that utilize the Expression Language need to always present a
> failure relationship.
> If a processor with only a success relationship, for example UpdateAttribute,
> utilizes the expression language to perform type coercion to a date and
> fails, the processor will be unable to dispose of the FlowFile and remain
> blocked indefinitely.
> Recreation flow:
> GenerateFlowFile -> Update Attribute #1 -> Update Attribute #2 -> Anything
> Update Attribute #1 - test = "Hello World"
> Update Attribute #2 - test = ${test:toDate('-MM-dd')}
>
> Generates an IllegalAttributeException on UpdateAttribute.
>
> The behavior should match numerical type coercion and silently skip the
> processing or offer failure relationships on processors supporting EL
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #2914: NIFI-5448 Added failure relationship to UpdateAttributes t...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/2914 @MikeThomsen - unfortunately, I believe this is a breaking change that will need to wait for NiFi 2.0.0. Adding a relationship will make the processor invalid after an upgrade and users will have to define what they want to do with this new relationship. ---
[jira] [Updated] (NIFI-5454) Add ability to specify per-flowfile values via DuplicateFlowFile
[ https://issues.apache.org/jira/browse/NIFI-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-5454: - Fix Version/s: 1.8.0 > Add ability to specify per-flowfile values via DuplicateFlowFile > > > Key: NIFI-5454 > URL: https://issues.apache.org/jira/browse/NIFI-5454 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Minor > Fix For: 1.8.0 > > > DuplicateFlowFile is currently useful mostly for load testing, as it > generates a fixed number of flow files as copies of the incoming flow file. > However there is no way to distinguish the flow files from each other, so for > non-test purposes, it would be nice to have a way to treat copies > differently, either for routing purposes or to access other values, such as > attributes with delimited fields, etc. > This Jira proposes to add the following improvements to DuplicateFlowFile: > - add a "copy.index" attribute to the outgoing flow files, setting the > attribute to zero (0) on the original flow file, and incrementally setting it > on the copies. > - Add support for Expression Language to the Number of Copies property -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5454) Add ability to specify per-flowfile values via DuplicateFlowFile
[ https://issues.apache.org/jira/browse/NIFI-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558239#comment-16558239 ] ASF GitHub Bot commented on NIFI-5454: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2917 > Add ability to specify per-flowfile values via DuplicateFlowFile > > > Key: NIFI-5454 > URL: https://issues.apache.org/jira/browse/NIFI-5454 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Minor > Fix For: 1.8.0 > > > DuplicateFlowFile is currently useful mostly for load testing, as it > generates a fixed number of flow files as copies of the incoming flow file. > However there is no way to distinguish the flow files from each other, so for > non-test purposes, it would be nice to have a way to treat copies > differently, either for routing purposes or to access other values, such as > attributes with delimited fields, etc. > This Jira proposes to add the following improvements to DuplicateFlowFile: > - add a "copy.index" attribute to the outgoing flow files, setting the > attribute to zero (0) on the original flow file, and incrementally setting it > on the copies. > - Add support for Expression Language to the Number of Copies property -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5454) Add ability to specify per-flowfile values via DuplicateFlowFile
[ https://issues.apache.org/jira/browse/NIFI-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-5454: - Resolution: Fixed Status: Resolved (was: Patch Available) > Add ability to specify per-flowfile values via DuplicateFlowFile > > > Key: NIFI-5454 > URL: https://issues.apache.org/jira/browse/NIFI-5454 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Minor > > DuplicateFlowFile is currently useful mostly for load testing, as it > generates a fixed number of flow files as copies of the incoming flow file. > However there is no way to distinguish the flow files from each other, so for > non-test purposes, it would be nice to have a way to treat copies > differently, either for routing purposes or to access other values, such as > attributes with delimited fields, etc. > This Jira proposes to add the following improvements to DuplicateFlowFile: > - add a "copy.index" attribute to the outgoing flow files, setting the > attribute to zero (0) on the original flow file, and incrementally setting it > on the copies. > - Add support for Expression Language to the Number of Copies property -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2917: NIFI-5454: Added EL support and copy.index attribut...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2917 ---
[jira] [Commented] (NIFI-5454) Add ability to specify per-flowfile values via DuplicateFlowFile
[ https://issues.apache.org/jira/browse/NIFI-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558238#comment-16558238 ] ASF subversion and git services commented on NIFI-5454: --- Commit 692943f016b19a5b8ff915c5921194081418e9ea in nifi's branch refs/heads/master from [~ca9mbu] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=692943f ] NIFI-5454: Added EL support and copy.index attribute to DuplicateFlowFile Signed-off-by: Pierre Villard This closes #2917. > Add ability to specify per-flowfile values via DuplicateFlowFile > > > Key: NIFI-5454 > URL: https://issues.apache.org/jira/browse/NIFI-5454 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Minor > > DuplicateFlowFile is currently useful mostly for load testing, as it > generates a fixed number of flow files as copies of the incoming flow file. > However there is no way to distinguish the flow files from each other, so for > non-test purposes, it would be nice to have a way to treat copies > differently, either for routing purposes or to access other values, such as > attributes with delimited fields, etc. > This Jira proposes to add the following improvements to DuplicateFlowFile: > - add a "copy.index" attribute to the outgoing flow files, setting the > attribute to zero (0) on the original flow file, and incrementally setting it > on the copies. > - Add support for Expression Language to the Number of Copies property -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5454) Add ability to specify per-flowfile values via DuplicateFlowFile
[ https://issues.apache.org/jira/browse/NIFI-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558237#comment-16558237 ] ASF GitHub Bot commented on NIFI-5454: -- Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/2917 +1, merging to master, thanks @mattyb149 > Add ability to specify per-flowfile values via DuplicateFlowFile > > > Key: NIFI-5454 > URL: https://issues.apache.org/jira/browse/NIFI-5454 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Minor > > DuplicateFlowFile is currently useful mostly for load testing, as it > generates a fixed number of flow files as copies of the incoming flow file. > However there is no way to distinguish the flow files from each other, so for > non-test purposes, it would be nice to have a way to treat copies > differently, either for routing purposes or to access other values, such as > attributes with delimited fields, etc. > This Jira proposes to add the following improvements to DuplicateFlowFile: > - add a "copy.index" attribute to the outgoing flow files, setting the > attribute to zero (0) on the original flow file, and incrementally setting it > on the copies. > - Add support for Expression Language to the Number of Copies property -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2917: NIFI-5454: Added EL support and copy.index attribute to Du...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/2917 +1, merging to master, thanks @mattyb149 ---
