[jira] [Commented] (NIFI-5746) The SEND and RECEIVE provenance events for load balancing do not have the same transit uri syntax
[ https://issues.apache.org/jira/browse/NIFI-5746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663196#comment-16663196 ] ASF GitHub Bot commented on NIFI-5746: -- Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/3109 Reviewing... > The SEND and RECEIVE provenance events for load balancing do not have the > same transit uri syntax > - > > Key: NIFI-5746 > URL: https://issues.apache.org/jira/browse/NIFI-5746 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.9.0 > > > The SEND event has a transit uri like nifi:connection: > The RECEIVE event has a transit uri like nifi:// address>/loadbalance/ > The RECEIVE event is much preferred, as it indicates not only that the > transfer was via load balance but also includes the address of the node and > the UUID of the connection. The SEND Transit URI should be changed to > nifi:///loadbalance/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #3109: NIFI-5746: Use Node Identifier's node address instead of g...
Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/3109 Reviewing... ---
[jira] [Assigned] (MINIFICPP-655) Issues building on OSX 10.14 (Mojave)
[
https://issues.apache.org/jira/browse/MINIFICPP-655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dustin Rodrigues reassigned MINIFICPP-655:
--
Assignee: Dustin Rodrigues
> Issues building on OSX 10.14 (Mojave)
> -
>
> Key: MINIFICPP-655
> URL: https://issues.apache.org/jira/browse/MINIFICPP-655
> Project: NiFi MiNiFi C++
> Issue Type: Bug
>Reporter: Dustin Rodrigues
>Assignee: Dustin Rodrigues
>Priority: Major
>
> There are issues building on OSX Mojave when attempting to use homebrew
> dependencies. Homebrew refuses to link macOS-provided software into
> /usr/local/{bin,lib,include} such as bison, flex, and curl and possibly
> others. cmake needs to be aware of the homebrew-installed locations of
> /usr/local/opt/PROGRAMNAME .
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (MINIFICPP-655) Issues building on OSX 10.14 (Mojave)
Dustin Rodrigues created MINIFICPP-655:
--
Summary: Issues building on OSX 10.14 (Mojave)
Key: MINIFICPP-655
URL: https://issues.apache.org/jira/browse/MINIFICPP-655
Project: NiFi MiNiFi C++
Issue Type: Bug
Reporter: Dustin Rodrigues
There are issues building on OSX Mojave when attempting to use homebrew
dependencies. Homebrew refuses to link macOS-provided software into
/usr/local/{bin,lib,include} such as bison, flex, and curl and possibly others.
cmake needs to be aware of the homebrew-installed locations of
/usr/local/opt/PROGRAMNAME .
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #3068: NIFI-5693 add support for multiple attachments in PutEmail...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3068 No rush, but am ready to review it whenever your ready. ---
[jira] [Commented] (NIFI-5693) PutEmail should support multiple attachments
[ https://issues.apache.org/jira/browse/NIFI-5693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663094#comment-16663094 ] ASF GitHub Bot commented on NIFI-5693: -- Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3068 No rush, but am ready to review it whenever your ready. > PutEmail should support multiple attachments > > > Key: NIFI-5693 > URL: https://issues.apache.org/jira/browse/NIFI-5693 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.7.1 >Reporter: David Savage >Priority: Minor > > *As a* NiFi User > *I want* to create multi part emails with many attachments > *So that* I can interface NiFi flows with legacy email systems > For this use case I reviewed this comment here [1] unfortunately the legacy > system requires individual attachments and is out of my control to upgrade so > the simple MergeFile based approach will not work. > I can see three potential solutions to this: > # Update PutEmail to extend BinFiles and use a similar mechanism to > MergeFiles to collect many files from the flow. Complexity - Hard, > Flexibility - High > # Update PutEmail with an optional attribute to unpack files generated by > MergeFile and add each entry as a new attachment. Complexity - Medium, > Hackiness - Medium > # Ignore PutEmail and use a custom script based on Groovy/Jython etc. > Complexity - Simple, Supportability Low > I've created a patch for 2 that I'll submit as a merge request soon. > Interested in feedback and happy to try other approaches if this is something > that the community is interested in supporting going forwards. > [1] > https://mail-archives.apache.org/mod_mbox/nifi-users/201709.mbox/%3CCALJK9a6oSdy5PXaKE0GxY1M3xQqYW8%2B0UUgsQmtbkO1qMK-MQA%40mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5747) InvokeHTTP Processor NPE
[
https://issues.apache.org/jira/browse/NIFI-5747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Sniffen updated NIFI-5747:
-
Description:
When using the InvokeHTTP Processor. If the requested URL redirects from HTTPS
to HTTP, the convertAttributesFromHeaders function attempts to call
{code:java}
responseHttp.handshake().peerPrincipal().getName()
{code}
when
{code:java}
responseHttp.handshake()
{code}
returns null. This throws a NPE.
This can be fixed by checking the responseHttp object for SSL rather than the
original URL.
was:
When using the InvokeHTTP Processor. If the requested URL redirects from HTTPS
to HTTP, the convertAttributesFromHeaders function attempts to call
{code:java}
responseHttp.handshake().peerPrincipal().getName()
{code}
when
{code:java}
responseHttp.handshake()
{code}
returns null. This throws a NPE.
> InvokeHTTP Processor NPE
>
>
> Key: NIFI-5747
> URL: https://issues.apache.org/jira/browse/NIFI-5747
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Julian Sniffen
>Priority: Major
> Attachments: nifi-5747.patch.zip
>
>
> When using the InvokeHTTP Processor. If the requested URL redirects from
> HTTPS to HTTP, the convertAttributesFromHeaders function attempts to call
> {code:java}
> responseHttp.handshake().peerPrincipal().getName()
> {code}
> when
> {code:java}
> responseHttp.handshake()
> {code}
> returns null. This throws a NPE.
> This can be fixed by checking the responseHttp object for SSL rather than the
> original URL.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5747) InvokeHTTP Processor NPE
[
https://issues.apache.org/jira/browse/NIFI-5747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Sniffen updated NIFI-5747:
-
Status: Patch Available (was: Open)
> InvokeHTTP Processor NPE
>
>
> Key: NIFI-5747
> URL: https://issues.apache.org/jira/browse/NIFI-5747
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Julian Sniffen
>Priority: Major
> Attachments: nifi-5747.patch.zip
>
>
> When using the InvokeHTTP Processor. If the requested URL redirects from
> HTTPS to HTTP, the convertAttributesFromHeaders function attempts to call
> {code:java}
> responseHttp.handshake().peerPrincipal().getName()
> {code}
> when
> {code:java}
> responseHttp.handshake()
> {code}
> returns null. This throws a NPE.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5747) InvokeHTTP Processor NPE
[
https://issues.apache.org/jira/browse/NIFI-5747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Sniffen updated NIFI-5747:
-
Attachment: nifi-5747.patch.zip
> InvokeHTTP Processor NPE
>
>
> Key: NIFI-5747
> URL: https://issues.apache.org/jira/browse/NIFI-5747
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Julian Sniffen
>Priority: Major
> Attachments: nifi-5747.patch.zip
>
>
> When using the InvokeHTTP Processor. If the requested URL redirects from
> HTTPS to HTTP, the convertAttributesFromHeaders function attempts to call
> {code:java}
> responseHttp.handshake().peerPrincipal().getName()
> {code}
> when
> {code:java}
> responseHttp.handshake()
> {code}
> returns null. This throws a NPE.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #422: Minificpp 644 - C API: add support to reg...
Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/422 ---
[jira] [Commented] (MINIFICPP-653) Log message will segfault client if no content produced
[ https://issues.apache.org/jira/browse/MINIFICPP-653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662947#comment-16662947 ] ASF GitHub Bot commented on MINIFICPP-653: -- GitHub user phrocker opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/427 MINIFICPP-653: Check if empty content, if so don't produce log messag… …e that can segfault client Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/phrocker/nifi-minifi-cpp MINIFICPP-653 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/427.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #427 commit b7b05b3fafeb7a23f6d5f1017ca2b9a6fea4bfb1 Author: Marc Parisi Date: 2018-10-23T15:51:19Z MINIFICPP-653: Check if empty content, if so don't produce log message that can segfault client > Log message will segfault client if no content produced > --- > > Key: MINIFICPP-653 > URL: https://issues.apache.org/jira/browse/MINIFICPP-653 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Blocker > > Log message will segfault client if no content produced -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #427: MINIFICPP-653: Check if empty content, if...
GitHub user phrocker opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/427 MINIFICPP-653: Check if empty content, if so don't produce log messag⦠â¦e that can segfault client Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/phrocker/nifi-minifi-cpp MINIFICPP-653 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/427.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #427 commit b7b05b3fafeb7a23f6d5f1017ca2b9a6fea4bfb1 Author: Marc Parisi Date: 2018-10-23T15:51:19Z MINIFICPP-653: Check if empty content, if so don't produce log message that can segfault client ---
[jira] [Created] (NIFI-5748) Improve handling of X-Forwarded-* headers in URI Rewriting
Kevin Doran created NIFI-5748:
-
Summary: Improve handling of X-Forwarded-* headers in URI Rewriting
Key: NIFI-5748
URL: https://issues.apache.org/jira/browse/NIFI-5748
Project: Apache NiFi
Issue Type: Improvement
Reporter: Kevin Doran
This ticket is to improve the handling of headers used by popular proxies when
rewriting URIs in NiFI. Currently, NiFi checks the following headers when
determining how to re-write URLs it returns clients:
>From
>[ApplicationResource|https://github.com/apache/nifi/blob/2201f7746fd16874aefbd12d546565f5d105ab04/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/ApplicationResource.java#L110]:
{code:java}
public static final String PROXY_SCHEME_HTTP_HEADER = "X-ProxyScheme";
public static final String PROXY_HOST_HTTP_HEADER = "X-ProxyHost";
public static final String PROXY_PORT_HTTP_HEADER = "X-ProxyPort";
public static final String PROXY_CONTEXT_PATH_HTTP_HEADER =
"X-ProxyContextPath";
public static final String FORWARDED_PROTO_HTTP_HEADER = "X-Forwarded-Proto";
public static final String FORWARDED_HOST_HTTP_HEADER = "X-Forwarded-Server";
public static final String FORWARDED_PORT_HTTP_HEADER = "X-Forwarded-Port";
public static final String FORWARDED_CONTEXT_HTTP_HEADER =
"X-Forwarded-Context";
// ...
final String scheme = getFirstHeaderValue(PROXY_SCHEME_HTTP_HEADER,
FORWARDED_PROTO_HTTP_HEADER);
final String host = getFirstHeaderValue(PROXY_HOST_HTTP_HEADER,
FORWARDED_HOST_HTTP_HEADER);
final String port = getFirstHeaderValue(PROXY_PORT_HTTP_HEADER,
FORWARDED_PORT_HTTP_HEADER);
{code}
Based on this, it looks like if both {{X-Forwarded-Server}} and
{{X-Forwarded-Host}} are set, that {{-Host}} will contain the hostname the
user/client requested, and {{-Server}} will contain the hostname of the proxy
server (ie, what the proxy server is able to determine from inspecting the
hostname of the instance it is on). See this for more details:
https://stackoverflow.com/questions/43689625/x-forwarded-host-vs-x-forwarded-server
Here is a demo based on docker containers and a reverse-proxy called Traefik
that shows where the current NiFi logic can break:
https://gist.github.com/kevdoran/2892004ccbfbb856115c8a756d9d4538
To use this gist, you can run the following:
{noformat}
wget -qO-
https://gist.githubusercontent.com/kevdoran/2892004ccbfbb856115c8a756d9d4538/raw/fb72151900d4d8fdcf4919fe5c8a94805fbb8401/docker-compose.yml
| docker-compose -f - up
{noformat}
Once the environment is up. Go to http://nifi.docker.localhost/nifi in Chrome
and try adding/configuring/moving a processor. This will reproduce the issue.
For this task, the following improvement is recommended:
Change the Header (string literal) for FORWARDED_HOST_HTTP_HEADER from
"X-Forwarded-Server" to "X-Forwarded-Host"
Additionally, some proxies use a different header for the context path prefix.
Traefik uses {{X-Forwarded-Prefix}}. There does not appear to be a universal
standard. In the future, we could make this header configurable, but for now,
perhaps we should add {{X-Forwarded-Prefix}} to the headers checked by NiFi so
that Traefik is supported out-of-the-box.
*Important:* After making this change, verify that proxying NiFi via Knox still
works, i.e., Knox should be sending the X-Forwarded-Host header that matches
what the user requested in the browser.
This change applies to NiFi Registry as well.
/cc [~jtstorck]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5747) InvokeHTTP Processor NPE
[
https://issues.apache.org/jira/browse/NIFI-5747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Sniffen updated NIFI-5747:
-
Description:
When using the InvokeHTTP Processor. If the requested URL redirects from HTTPS
to HTTP, the convertAttributesFromHeaders function attempts to call
{code:java}
responseHttp.handshake().peerPrincipal().getName()
{code}
when
{code:java}
responseHttp.handshake()
{code}
returns null. This throws a NPE.
was:
When using the InvokeHTTP Processor. If the requested URL redirects from HTTPS
to HTTP, the convertAttributesFromHeaders function attempts to call
`responseHttp.handshake().peerPrincipal().getName()` when
`responseHttp.handshake()` returns null. This throws a NPE.
> InvokeHTTP Processor NPE
>
>
> Key: NIFI-5747
> URL: https://issues.apache.org/jira/browse/NIFI-5747
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Julian Sniffen
>Priority: Major
>
> When using the InvokeHTTP Processor. If the requested URL redirects from
> HTTPS to HTTP, the convertAttributesFromHeaders function attempts to call
> {code:java}
> responseHttp.handshake().peerPrincipal().getName()
> {code}
> when
> {code:java}
> responseHttp.handshake()
> {code}
> returns null. This throws a NPE.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-5747) InvokeHTTP Processor NPE
Julian Sniffen created NIFI-5747: Summary: InvokeHTTP Processor NPE Key: NIFI-5747 URL: https://issues.apache.org/jira/browse/NIFI-5747 Project: Apache NiFi Issue Type: Bug Reporter: Julian Sniffen When using the InvokeHTTP Processor. If the requested URL redirects from HTTPS to HTTP, the convertAttributesFromHeaders function attempts to call `responseHttp.handshake().peerPrincipal().getName()` when `responseHttp.handshake()` returns null. This throws a NPE. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5746) The SEND and RECEIVE provenance events for load balancing do not have the same transit uri syntax
[ https://issues.apache.org/jira/browse/NIFI-5746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-5746: - Fix Version/s: 1.9.0 Status: Patch Available (was: Open) > The SEND and RECEIVE provenance events for load balancing do not have the > same transit uri syntax > - > > Key: NIFI-5746 > URL: https://issues.apache.org/jira/browse/NIFI-5746 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.9.0 > > > The SEND event has a transit uri like nifi:connection: > The RECEIVE event has a transit uri like nifi:// address>/loadbalance/ > The RECEIVE event is much preferred, as it indicates not only that the > transfer was via load balance but also includes the address of the node and > the UUID of the connection. The SEND Transit URI should be changed to > nifi:///loadbalance/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5746) The SEND and RECEIVE provenance events for load balancing do not have the same transit uri syntax
[ https://issues.apache.org/jira/browse/NIFI-5746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662736#comment-16662736 ] ASF GitHub Bot commented on NIFI-5746: -- GitHub user markap14 opened a pull request: https://github.com/apache/nifi/pull/3109 NIFI-5746: Use Node Identifier's node address instead of getting from… … socket for RECEIVE prov events; make SEND prov events match syntax of RECEIVE prov events Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/markap14/nifi NIFI-5746 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3109.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3109 commit c5e79da4449db81119ab898f15ab7c2aa64b9c91 Author: Mark Payne Date: 2018-10-24T17:45:55Z NIFI-5746: Use Node Identifier's node address instead of getting from socket for RECEIVE prov events; make SEND prov events match syntax of RECEIVE prov events > The SEND and RECEIVE provenance events for load balancing do not have the > same transit uri syntax > - > > Key: NIFI-5746 > URL: https://issues.apache.org/jira/browse/NIFI-5746 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.9.0 > > > The SEND event has a transit uri like nifi:connection: > The RECEIVE event has a transit uri like nifi:// address>/loadbalance/ > The RECEIVE event is much preferred, as it indicates not only that the > transfer was via load balance but also includes the address of the node and > the UUID of the connection. The SEND Transit URI should be changed to > nifi:///loadbalance/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5745) Load Balanced Connections can stop balancing across the cluster if all nodes report backpressure and all data destined for another node
[ https://issues.apache.org/jira/browse/NIFI-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-5745: - Fix Version/s: 1.9.0 Status: Patch Available (was: Open) > Load Balanced Connections can stop balancing across the cluster if all nodes > report backpressure and all data destined for another node > --- > > Key: NIFI-5745 > URL: https://issues.apache.org/jira/browse/NIFI-5745 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.9.0 > > > If a connection is load balanced, and backpressure is configured, we can > possibly get into a situation where all nodes want to send data to other > nodes, but those nodes are applying backpressure. If this backpressure is > applied only due to flowfiles existing in the connection, waiting to be > distributed, we can reach a live lock scenario. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3109: NIFI-5746: Use Node Identifier's node address inste...
GitHub user markap14 opened a pull request: https://github.com/apache/nifi/pull/3109 NIFI-5746: Use Node Identifier's node address instead of getting from⦠⦠socket for RECEIVE prov events; make SEND prov events match syntax of RECEIVE prov events Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/markap14/nifi NIFI-5746 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3109.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3109 commit c5e79da4449db81119ab898f15ab7c2aa64b9c91 Author: Mark Payne Date: 2018-10-24T17:45:55Z NIFI-5746: Use Node Identifier's node address instead of getting from socket for RECEIVE prov events; make SEND prov events match syntax of RECEIVE prov events ---
[jira] [Updated] (NIFI-5714) Hive[3]ConnectionPool - Kerberos Authentication issue/misleading
[
https://issues.apache.org/jira/browse/NIFI-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-5714:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Hive[3]ConnectionPool - Kerberos Authentication issue/misleading
>
>
> Key: NIFI-5714
> URL: https://issues.apache.org/jira/browse/NIFI-5714
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
> Fix For: 1.9.0
>
>
> In {{HiveConnectionPool}} and {{Hive3ConnectionPool}}, in the {{@OnEnabled}}
> method, we have:
> {code:java}
> log.info("Hive Security Enabled, logging in as principal {} with keytab {}",
> new Object[] {resolvedPrincipal, resolvedKeytab});
> try {
> ugi = hiveConfigurator.authenticate(hiveConfig, resolvedPrincipal,
> resolvedKeytab);
> } catch (AuthenticationFailedException ae) {
> log.error(ae.getMessage(), ae);
> }
> getLogger().info("Successfully logged in as principal {} with keytab {}", new
> Object[] {resolvedPrincipal, resolvedKeytab});{code}
> Which causes two issues:
> * we're logging the successful message even though the authentication failed
> * the Hive connection is created using the NiFi user identity (this would
> need to be confirmed but that's what I observed during a test - it could be
> due to the environment though)
> In my opinion, an {{InitializationException}} should be thrown so that the
> controller service is not enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5714) Hive[3]ConnectionPool - Kerberos Authentication issue/misleading
[
https://issues.apache.org/jira/browse/NIFI-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662601#comment-16662601
]
ASF GitHub Bot commented on NIFI-5714:
--
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/3086
> Hive[3]ConnectionPool - Kerberos Authentication issue/misleading
>
>
> Key: NIFI-5714
> URL: https://issues.apache.org/jira/browse/NIFI-5714
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
> Fix For: 1.9.0
>
>
> In {{HiveConnectionPool}} and {{Hive3ConnectionPool}}, in the {{@OnEnabled}}
> method, we have:
> {code:java}
> log.info("Hive Security Enabled, logging in as principal {} with keytab {}",
> new Object[] {resolvedPrincipal, resolvedKeytab});
> try {
> ugi = hiveConfigurator.authenticate(hiveConfig, resolvedPrincipal,
> resolvedKeytab);
> } catch (AuthenticationFailedException ae) {
> log.error(ae.getMessage(), ae);
> }
> getLogger().info("Successfully logged in as principal {} with keytab {}", new
> Object[] {resolvedPrincipal, resolvedKeytab});{code}
> Which causes two issues:
> * we're logging the successful message even though the authentication failed
> * the Hive connection is created using the NiFi user identity (this would
> need to be confirmed but that's what I observed during a test - it could be
> due to the environment though)
> In my opinion, an {{InitializationException}} should be thrown so that the
> controller service is not enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5714) Hive[3]ConnectionPool - Kerberos Authentication issue/misleading
[
https://issues.apache.org/jira/browse/NIFI-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662599#comment-16662599
]
ASF subversion and git services commented on NIFI-5714:
---
Commit 2201f7746fd16874aefbd12d546565f5d105ab04 in nifi's branch
refs/heads/master from [~pvillard]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=2201f77 ]
NIFI-5714 - Hive[3]ConnectionPool - Kerberos Authentication issue/misleading
add @Ignore on unit test...
Signed-off-by: Matthew Burgess
This closes #3086
> Hive[3]ConnectionPool - Kerberos Authentication issue/misleading
>
>
> Key: NIFI-5714
> URL: https://issues.apache.org/jira/browse/NIFI-5714
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
> Fix For: 1.9.0
>
>
> In {{HiveConnectionPool}} and {{Hive3ConnectionPool}}, in the {{@OnEnabled}}
> method, we have:
> {code:java}
> log.info("Hive Security Enabled, logging in as principal {} with keytab {}",
> new Object[] {resolvedPrincipal, resolvedKeytab});
> try {
> ugi = hiveConfigurator.authenticate(hiveConfig, resolvedPrincipal,
> resolvedKeytab);
> } catch (AuthenticationFailedException ae) {
> log.error(ae.getMessage(), ae);
> }
> getLogger().info("Successfully logged in as principal {} with keytab {}", new
> Object[] {resolvedPrincipal, resolvedKeytab});{code}
> Which causes two issues:
> * we're logging the successful message even though the authentication failed
> * the Hive connection is created using the NiFi user identity (this would
> need to be confirmed but that's what I observed during a test - it could be
> due to the environment though)
> In my opinion, an {{InitializationException}} should be thrown so that the
> controller service is not enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5714) Hive[3]ConnectionPool - Kerberos Authentication issue/misleading
[
https://issues.apache.org/jira/browse/NIFI-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-5714:
---
Fix Version/s: 1.9.0
> Hive[3]ConnectionPool - Kerberos Authentication issue/misleading
>
>
> Key: NIFI-5714
> URL: https://issues.apache.org/jira/browse/NIFI-5714
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
> Fix For: 1.9.0
>
>
> In {{HiveConnectionPool}} and {{Hive3ConnectionPool}}, in the {{@OnEnabled}}
> method, we have:
> {code:java}
> log.info("Hive Security Enabled, logging in as principal {} with keytab {}",
> new Object[] {resolvedPrincipal, resolvedKeytab});
> try {
> ugi = hiveConfigurator.authenticate(hiveConfig, resolvedPrincipal,
> resolvedKeytab);
> } catch (AuthenticationFailedException ae) {
> log.error(ae.getMessage(), ae);
> }
> getLogger().info("Successfully logged in as principal {} with keytab {}", new
> Object[] {resolvedPrincipal, resolvedKeytab});{code}
> Which causes two issues:
> * we're logging the successful message even though the authentication failed
> * the Hive connection is created using the NiFi user identity (this would
> need to be confirmed but that's what I observed during a test - it could be
> due to the environment though)
> In my opinion, an {{InitializationException}} should be thrown so that the
> controller service is not enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3086: NIFI-5714 - Hive[3]ConnectionPool - Kerberos Authen...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3086 ---
[jira] [Commented] (NIFI-5714) Hive[3]ConnectionPool - Kerberos Authentication issue/misleading
[
https://issues.apache.org/jira/browse/NIFI-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662598#comment-16662598
]
ASF GitHub Bot commented on NIFI-5714:
--
Github user mattyb149 commented on the issue:
https://github.com/apache/nifi/pull/3086
+1 LGTM, tested successfully. Thanks for the fix! Merging to master
> Hive[3]ConnectionPool - Kerberos Authentication issue/misleading
>
>
> Key: NIFI-5714
> URL: https://issues.apache.org/jira/browse/NIFI-5714
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
> Fix For: 1.9.0
>
>
> In {{HiveConnectionPool}} and {{Hive3ConnectionPool}}, in the {{@OnEnabled}}
> method, we have:
> {code:java}
> log.info("Hive Security Enabled, logging in as principal {} with keytab {}",
> new Object[] {resolvedPrincipal, resolvedKeytab});
> try {
> ugi = hiveConfigurator.authenticate(hiveConfig, resolvedPrincipal,
> resolvedKeytab);
> } catch (AuthenticationFailedException ae) {
> log.error(ae.getMessage(), ae);
> }
> getLogger().info("Successfully logged in as principal {} with keytab {}", new
> Object[] {resolvedPrincipal, resolvedKeytab});{code}
> Which causes two issues:
> * we're logging the successful message even though the authentication failed
> * the Hive connection is created using the NiFi user identity (this would
> need to be confirmed but that's what I observed during a test - it could be
> due to the environment though)
> In my opinion, an {{InitializationException}} should be thrown so that the
> controller service is not enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #3086: NIFI-5714 - Hive[3]ConnectionPool - Kerberos Authenticatio...
Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/3086 +1 LGTM, tested successfully. Thanks for the fix! Merging to master ---
[jira] [Created] (NIFI-5746) The SEND and RECEIVE provenance events for load balancing do not have the same transit uri syntax
Mark Payne created NIFI-5746: Summary: The SEND and RECEIVE provenance events for load balancing do not have the same transit uri syntax Key: NIFI-5746 URL: https://issues.apache.org/jira/browse/NIFI-5746 Project: Apache NiFi Issue Type: Bug Components: Core Framework Affects Versions: 1.8.0 Reporter: Mark Payne Assignee: Mark Payne The SEND event has a transit uri like nifi:connection: The RECEIVE event has a transit uri like nifi:///loadbalance/ The RECEIVE event is much preferred, as it indicates not only that the transfer was via load balance but also includes the address of the node and the UUID of the connection. The SEND Transit URI should be changed to nifi:///loadbalance/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5745) Load Balanced Connections can stop balancing across the cluster if all nodes report backpressure and all data destined for another node
[ https://issues.apache.org/jira/browse/NIFI-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662556#comment-16662556 ] ASF GitHub Bot commented on NIFI-5745: -- GitHub user markap14 opened a pull request: https://github.com/apache/nifi/pull/3108 NIFI-5745: When determining if backpressure should be applied across … …nodes for load balancing, only consider if the local partition has reached the threshold limits instead of considering the size of the entire queue Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/markap14/nifi NIFI-5745 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3108.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3108 commit 1ee39847e6ecc0860bb1e272a4aba0ebf23bde16 Author: Mark Payne Date: 2018-10-24T17:06:57Z NIFI-5745: When determining if backpressure should be applied across nodes for load balancing, only consider if the local partition has reached the threshold limits instead of considering the size of the entire queue > Load Balanced Connections can stop balancing across the cluster if all nodes > report backpressure and all data destined for another node > --- > > Key: NIFI-5745 > URL: https://issues.apache.org/jira/browse/NIFI-5745 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > > If a connection is load balanced, and backpressure is configured, we can > possibly get into a situation where all nodes want to send data to other > nodes, but those nodes are applying backpressure. If this backpressure is > applied only due to flowfiles existing in the connection, waiting to be > distributed, we can reach a live lock scenario. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3108: NIFI-5745: When determining if backpressure should ...
GitHub user markap14 opened a pull request: https://github.com/apache/nifi/pull/3108 NIFI-5745: When determining if backpressure should be applied across ⦠â¦nodes for load balancing, only consider if the local partition has reached the threshold limits instead of considering the size of the entire queue Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/markap14/nifi NIFI-5745 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3108.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3108 commit 1ee39847e6ecc0860bb1e272a4aba0ebf23bde16 Author: Mark Payne Date: 2018-10-24T17:06:57Z NIFI-5745: When determining if backpressure should be applied across nodes for load balancing, only consider if the local partition has reached the threshold limits instead of considering the size of the entire queue ---
[jira] [Created] (NIFI-5745) Load Balanced Connections can stop balancing across the cluster if all nodes report backpressure and all data destined for another node
Mark Payne created NIFI-5745: Summary: Load Balanced Connections can stop balancing across the cluster if all nodes report backpressure and all data destined for another node Key: NIFI-5745 URL: https://issues.apache.org/jira/browse/NIFI-5745 Project: Apache NiFi Issue Type: Bug Components: Core Framework Affects Versions: 1.8.0 Reporter: Mark Payne Assignee: Mark Payne If a connection is load balanced, and backpressure is configured, we can possibly get into a situation where all nodes want to send data to other nodes, but those nodes are applying backpressure. If this backpressure is applied only due to flowfiles existing in the connection, waiting to be distributed, we can reach a live lock scenario. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp issue #422: Minificpp 644 - C API: add support to register t...
Github user phrocker commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/422 new travis failure appears to be unrelated to this commit, so will merge. thanks! ---
[jira] [Commented] (NIFI-5079) Add support for logical types to Hive DDL creation in PutORC
[ https://issues.apache.org/jira/browse/NIFI-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662425#comment-16662425 ] Matt Burgess commented on NIFI-5079: We considered this, but we don't know the implications of evicting the version of Avro that Hive 1.2.1 wants (1.7.7) for a newer one (1.8.x). It seems to fix this issue fine and supports logical types, but it may mess something else up in the Hive client, so we thought it too dangerous. We can certainly re-open the discussion here and/or on the mailing list if you like. Another reason was that Hive 1.2.1 is quite out of date, hopefully folks will get to Hive 3 sooner than later, and the logical types are all supported in the Hive 3 versions of the processors and controller services. > Add support for logical types to Hive DDL creation in PutORC > > > Key: NIFI-5079 > URL: https://issues.apache.org/jira/browse/NIFI-5079 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > While other processors have been updated to support Avro logical types (such > as decimal, timestamp, etc.), ConvertAvroToORC does not currently support > them. We should update ConvertAvroToORC to handle Avro logical types, not > just in the conversion to ORC but also in the generated Hive DDL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5744) Put exception message to attribute while ExecuteSQL fail
[
https://issues.apache.org/jira/browse/NIFI-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662423#comment-16662423
]
ASF GitHub Bot commented on NIFI-5744:
--
Github user mattyb149 commented on the issue:
https://github.com/apache/nifi/pull/3107
Reviewing...
> Put exception message to attribute while ExecuteSQL fail
>
>
> Key: NIFI-5744
> URL: https://issues.apache.org/jira/browse/NIFI-5744
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.7.1
>Reporter: Deon Huang
>Assignee: Deon Huang
>Priority: Minor
>
> In some scenario, it would be great if we could have different behavior based
> on exception.
> Better error tracking afterwards in attribute format instead of tracking in
> log.
> For example, if it’s connection refused exception due to wrong url.
> We won’t want to retry and error message attribute would be helpful to keep
> track of.
> While it’s other scenario that database temporary unavailable, we should
> retry it based on should retry exception.
> Should be a quick fix at AbstractExecuteSQL before transfer flowfile to
> failure relationship
> {code:java}
> session.transfer(fileToProcess, REL_FAILURE);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #3107: NIFI-5744: Put exception message to attribute while Execut...
Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/3107 Reviewing... ---
[jira] [Resolved] (NIFI-5181) Update default Provenance Repository to WriteAheadProvenanceRepository
[ https://issues.apache.org/jira/browse/NIFI-5181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Moser resolved NIFI-5181. - Resolution: Fixed Fix Version/s: 1.8.0 Resolved by NIFI-5482 > Update default Provenance Repository to WriteAheadProvenanceRepository > -- > > Key: NIFI-5181 > URL: https://issues.apache.org/jira/browse/NIFI-5181 > Project: Apache NiFi > Issue Type: Improvement > Components: Tools and Build >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.8.0 > > > The WriteAheadProvenanceRepository was written several releases ago in order > to replace the PersistentProvenanceRepository. The > PersistentProvenanceRepository remained the default in nifi.properties to > ensure that the WriteAheadProvenanceRepository was stable enough before > jumping right in. Time has shown that the WriteAheadProvenanceRepository is > not only far faster than the old implementation but that it is also more > stable. We should update the default now to the > WriteAheadProvenanceRepository. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-1364) Audit OCSP certificate validation
[ https://issues.apache.org/jira/browse/NIFI-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Gough reassigned NIFI-1364: -- Assignee: Nathan Gough (was: Andy LoPresto) > Audit OCSP certificate validation > - > > Key: NIFI-1364 > URL: https://issues.apache.org/jira/browse/NIFI-1364 > Project: Apache NiFi > Issue Type: Task > Components: Core Framework >Affects Versions: 0.4.1 >Reporter: Andy LoPresto >Assignee: Nathan Gough >Priority: Major > Labels: certificate, ocsp, security > > While upgrading the version of BouncyCastle libraries used, I had to re-write > the OCSP certificate validation code because BC split the PKIX code into a > separate module and renamed many classes & methods. During this re-write, I > made the code compile using the new logic, but I am unsure that OCSP > validation needs to occur outside of the SSL/TLS negotiation, or that the > current mechanism is correct. > Questions: > * Can we use Java's built-in OCSP validation? [1][2] > * Is the current mechanism correct, where a local cache is used with custom > internal classes representing OCSP requests and statuses, and it queries a > pre-specified OCSP responder as opposed to the per-certificate OCSP responder > listed in each certificate's Authority Information Access OCSP URI [3]? I > think this design decision stems from a legacy environment which may not > apply to current use cases. > More information: [4] > [1] https://blogs.oracle.com/xuelei/entry/enable_ocsp_checking > [2] > https://stackoverflow.com/questions/8506661/check-x509-certificate-revocation-status-in-spring-security-before-authenticatin > [3] > https://blog.ivanristic.com/2014/02/checking-ocsp-revocation-using-openssl.html > [4] > https://raymii.org/s/articles/OpenSSL_Manually_Verify_a_certificate_against_an_OCSP.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5370) Cluster request replication failing with wildcard certs
[
https://issues.apache.org/jira/browse/NIFI-5370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662388#comment-16662388
]
alfredo commented on NIFI-5370:
---
hi all
i am deploying a nifi cluster secured using the docker image for nifi version
1.7.1 and still facing same problem, maybe im forgetting to do some
configuration?
kind regards,
> Cluster request replication failing with wildcard certs
> ---
>
> Key: NIFI-5370
> URL: https://issues.apache.org/jira/browse/NIFI-5370
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.7.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
> Labels: certificate, cluster, security, tls, wildcard
> Fix For: 1.8.0, 1.7.1
>
>
> From the users mailing list:
> {quote}
> Team,
>
> NiFi secured cluster throws below error with wildcarded self-signed
> standalone certificate. Just a brief background, we are deploying nifi in
> Kubernetes where we have to use wildcarded certificates. Till nifi 1.6.0, it
> was working fine.
> Also I tried bringing up NiFi in linux VM in secured cluster mode with
> wildcarded certs, I am getting same error.
>
> Toolkit command to generate certs:
> bin/tls-toolkit.sh standalone -n
> '*.mynifi-nifi-headless.default.svc.cluster.local’ -C 'CN=admin, OU=NIFI' -o
>
>
> Logs:
> 2018-07-02 12:40:32,369 WARN [Replicate Request Thread-1]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET
> /nifi-api/flow/current-user to
> mynifi-nifi-1.mynifi-nifi-headless.default.svc.cluster.local:8443 due to
> javax.net.ssl.SSLPeerUnverifiedException: Hostname
> mynifi-nifi-1.mynifi-nifi-headless.default.svc.cluster.local not verified:
> certificate: sha256/
> DN: CN=*.mynifi-nifi-headless.default.svc.cluster.local, OU=NIFI
> subjectAltNames: [*.mynifi-nifi-headless.default.svc.cluster.local]
> 2018-07-02 12:40:32,370 WARN [Replicate Request Thread-1]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator
> javax.net.ssl.SSLPeerUnverifiedException: Hostname
> mynifi-nifi-1.mynifi-nifi-headless.default.svc.cluster.local not verified:
> certificate: sha256/
> DN: CN=*.mynifi-nifi-headless.default.svc.cluster.local, OU=NIFI
> subjectAltNames: [*.mynifi-nifi-headless.default.svc.cluster.local]
> at
> okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:316)
>
> Please help me in resolving this.
>
> Note: Same certificates is working for single mode setup.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi-minifi pull request #139: MINIFI-477 Upgrade dependencies to 1.8.0
GitHub user apiri opened a pull request: https://github.com/apache/nifi-minifi/pull/139 MINIFI-477 Upgrade dependencies to 1.8.0 This upgrades dependencies to 1.8.0. This should not be merged in until NiFi 1.8.0 release voting completes and artifacts are generally available, but was done to ensure there were no overarching issues that required more extensive changes. --- Thank you for submitting a contribution to Apache NiFi - MiNiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi-minifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under minifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under minifi-assembly? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apiri/nifi-minifi minifi-477 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi/pull/139.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #139 commit 206e0be64c2be011a86c2024aef5eb847115a917 Author: Aldrin Piri Date: 2018-10-24T13:34:31Z MINIFI-477 Upgrade dependencies to 1.8.0 ---
[GitHub] nifi pull request #3107: NIFI-5744: Put exception message to attribute while...
GitHub user yjhyjhyjh0 opened a pull request: https://github.com/apache/nifi/pull/3107 NIFI-5744: Put exception message to attribute while ExecuteSQL fail Put exception message to flowfile attribute if exception occur. Add unit test to ExecuteSQL, ExecuteSQLRecord. Update document annotation. Similar exception catching also implement in other processors like InvokeHTTP, GenerateTableFetch. ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/yjhyjhyjh0/nifi NIFI-5744 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3107.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3107 ---
[jira] [Commented] (NIFI-5744) Put exception message to attribute while ExecuteSQL fail
[
https://issues.apache.org/jira/browse/NIFI-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662291#comment-16662291
]
ASF GitHub Bot commented on NIFI-5744:
--
GitHub user yjhyjhyjh0 opened a pull request:
https://github.com/apache/nifi/pull/3107
NIFI-5744: Put exception message to attribute while ExecuteSQL fail
Put exception message to flowfile attribute if exception occur.
Add unit test to ExecuteSQL, ExecuteSQLRecord.
Update document annotation.
Similar exception catching also implement in other processors like
InvokeHTTP, GenerateTableFetch.
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with NIFI- where is the JIRA number
you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [x] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yjhyjhyjh0/nifi NIFI-5744
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/3107.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3107
> Put exception message to attribute while ExecuteSQL fail
>
>
> Key: NIFI-5744
> URL: https://issues.apache.org/jira/browse/NIFI-5744
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.7.1
>Reporter: Deon Huang
>Assignee: Deon Huang
>Priority: Minor
>
> In some scenario, it would be great if we could have different behavior based
> on exception.
> Better error tracking afterwards in attribute format instead of tracking in
> log.
> For example, if it’s connection refused exception due to wrong url.
> We won’t want to retry and error message attribute would be helpful to keep
> track of.
> While it’s other scenario that database temporary unavailable, we should
> retry it based on should retry exception.
> Should be a quick fix at AbstractExecuteSQL before transfer flowfile to
> failure relationship
> {code:java}
> session.transfer(fileToProcess, REL_FAILURE);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (MINIFICPP-654) C API: failure callback improvements
Arpad Boda created MINIFICPP-654: Summary: C API: failure callback improvements Key: MINIFICPP-654 URL: https://issues.apache.org/jira/browse/MINIFICPP-654 Project: NiFi MiNiFi C++ Issue Type: Improvement Reporter: Arpad Boda Assignee: Arpad Boda Fix For: 0.6.0 Improvements and further discussion of failure callbacks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #421: Minificpp 641 - C API: add support to reg...
Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/421 ---
[jira] [Commented] (NIFI-3641) gRPC processor
[ https://issues.apache.org/jira/browse/NIFI-3641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662167#comment-16662167 ] David Balamber commented on NIFI-3641: -- Are there any plans to improve the current InvokeGRPC and ListenGRPC processors, so that they could take any .proto file to communicate with a third-party gRPC server that uses a specific Protobuf message definition? > gRPC processor > -- > > Key: NIFI-3641 > URL: https://issues.apache.org/jira/browse/NIFI-3641 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Bryan Rosander >Priority: Major > > "gRPC is a modern open source high performance RPC framework that can run in > any environment. It can efficiently connect services in and across data > centers with pluggable support for load balancing, tracing, health checking > and authentication. It is also applicable in last mile of distributed > computing to connect devices, mobile applications and browsers to backend > services." [1] > It would be useful to be able to interact with gRPC services in NiFi. A > first step would be to have a processor that can take a .proto file or files > and then use the generated code to talk to a gRPC server. Having a > Site-To-Site gRPC implementation could also be useful though as it should be > very easy to port to any of the supported platforms. [2] > This should allow us to interact with Tensor Flow among other things. > [1] http://www.grpc.io/about/ > [2] http://www.grpc.io/about/#osp -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2743: NIFI-5226: Implement a Record API based PutInfluxDB proces...
Github user ivankudibal commented on the issue: https://github.com/apache/nifi/pull/2743 @MikeThomsen ---
[jira] [Commented] (NIFI-5226) Implement a Record API based PutInfluxDB processor
[ https://issues.apache.org/jira/browse/NIFI-5226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662133#comment-16662133 ] ASF GitHub Bot commented on NIFI-5226: -- Github user ivankudibal commented on the issue: https://github.com/apache/nifi/pull/2743 @MikeThomsen > Implement a Record API based PutInfluxDB processor > -- > > Key: NIFI-5226 > URL: https://issues.apache.org/jira/browse/NIFI-5226 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Bonitoo4Influxdata Engineering >Priority: Major > Labels: influx, processor, timeseries > > Implement a new processor that uses the Record API to put data into InfluxDB. > PutInfluxDBRecord features: > * Input can be any built-in or custom implemented NiFi RecordReader (json, > avro, csv, InfluxLineProtocolReader...) > * Configurable mapping between NiFi Records and InfluxDB measurement, field > and tags > * Configurable timestamp precision > * Reusable connection settings (InfluxDB url, password) for more processors > via InfluxDBService controller > * Advanced InfluxDB client settings > * > ** Gzip compression > ** Batching, jitter, flush settings > InfluxDBService features: > * InfluxDBService allows sharing connection configuration among more NiFi > processors. > * SSL support > InfluxLineProtocolReader features > * InfluxLineProtocolReader parses the InfluxDB Line Protocol into NiFi > Record. This allows processing, filtering and partitioning data in NiFi > obtained from Telegraf agents, IoT devices, InfluxDB subscriptions and other > InfluxDB Line protocol devices. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-5744) Put exception message to attribute while ExecuteSQL fail
[
https://issues.apache.org/jira/browse/NIFI-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deon Huang reassigned NIFI-5744:
Assignee: Deon Huang
> Put exception message to attribute while ExecuteSQL fail
>
>
> Key: NIFI-5744
> URL: https://issues.apache.org/jira/browse/NIFI-5744
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.7.1
>Reporter: Deon Huang
>Assignee: Deon Huang
>Priority: Minor
>
> In some scenario, it would be great if we could have different behavior based
> on exception.
> Better error tracking afterwards in attribute format instead of tracking in
> log.
> For example, if it’s connection refused exception due to wrong url.
> We won’t want to retry and error message attribute would be helpful to keep
> track of.
> While it’s other scenario that database temporary unavailable, we should
> retry it based on should retry exception.
> Should be a quick fix at AbstractExecuteSQL before transfer flowfile to
> failure relationship
> {code:java}
> session.transfer(fileToProcess, REL_FAILURE);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5744) Put exception message to attribute while ExecuteSQL fail
[
https://issues.apache.org/jira/browse/NIFI-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deon Huang updated NIFI-5744:
-
Issue Type: Improvement (was: Bug)
> Put exception message to attribute while ExecuteSQL fail
>
>
> Key: NIFI-5744
> URL: https://issues.apache.org/jira/browse/NIFI-5744
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.7.1
>Reporter: Deon Huang
>Priority: Minor
>
> In some scenario, it would be great if we could have different behavior based
> on exception.
> Better error tracking afterwards in attribute format instead of tracking in
> log.
> For example, if it’s connection refused exception due to wrong url.
> We won’t want to retry and error message attribute would be helpful to keep
> track of.
> While it’s other scenario that database temporary unavailable, we should
> retry it based on should retry exception.
> Should be a quick fix at AbstractExecuteSQL before transfer flowfile to
> failure relationship
> {code:java}
> session.transfer(fileToProcess, REL_FAILURE);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-5744) Put exception message to attribute while ExecuteSQL fail
Deon Huang created NIFI-5744:
Summary: Put exception message to attribute while ExecuteSQL fail
Key: NIFI-5744
URL: https://issues.apache.org/jira/browse/NIFI-5744
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 1.7.1
Reporter: Deon Huang
In some scenario, it would be great if we could have different behavior based
on exception.
Better error tracking afterwards in attribute format instead of tracking in
log.
For example, if it’s connection refused exception due to wrong url.
We won’t want to retry and error message attribute would be helpful to keep
track of.
While it’s other scenario that database temporary unavailable, we should retry
it based on should retry exception.
Should be a quick fix at AbstractExecuteSQL before transfer flowfile to failure
relationship
{code:java}
session.transfer(fileToProcess, REL_FAILURE);
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5079) Add support for logical types to Hive DDL creation in PutORC
[
https://issues.apache.org/jira/browse/NIFI-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661929#comment-16661929
]
Bartłomiej Tartanus commented on NIFI-5079:
---
Does it mean that ConvertAvroToOrc is going to be depracated? Currently putting
records with decimal field into Hive causes this error during select query:
{{java.lang.RuntimeException:
com.cloudera.hiveserver2.support.exceptions.GeneralException:
[Cloudera][HiveJDBCDriver](500312) Error in fetching data rows:
*org.apache.hive.service.cli.HiveSQLException:java.io.IOException:
java.io.IOException: ORC does not support type conversion from file type binary
(14) to reader type decimal(10,2) (14):25:24;}}
Currently I am using NiFi 1.6 with Avro 1.8.2 and had to add logical types
support without updating NiFi version, so I've prepared patch that fixes this
error. I can submit pull request if you're interested. Only NiFiOrcUtils file
needs to be changed - avro version is still 1.7.7.
> Add support for logical types to Hive DDL creation in PutORC
>
>
> Key: NIFI-5079
> URL: https://issues.apache.org/jira/browse/NIFI-5079
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> While other processors have been updated to support Avro logical types (such
> as decimal, timestamp, etc.), ConvertAvroToORC does not currently support
> them. We should update ConvertAvroToORC to handle Avro logical types, not
> just in the conversion to ORC but also in the generated Hive DDL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (NIFI-4273) incorrect data access urls when using site to site when connecting through reverse proxies. Inability for site to site data access urls to utilize proxied user identites
[
https://issues.apache.org/jira/browse/NIFI-4273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Kawamura resolved NIFI-4273.
-
Resolution: Fixed
Assignee: Koji Kawamura
Fix Version/s: 1.7.0
This use-case is now supported by NIFI-4932.
[~WaterslideKid] Please refer NiFi Admin guid for how to configure NiFi
hostname/port for S2S behind a reverse proxy.
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#site_to_site_reverse_proxy_properties
> incorrect data access urls when using site to site when connecting through
> reverse proxies. Inability for site to site data access urls to utilize
> proxied user identites.
> ---
>
> Key: NIFI-4273
> URL: https://issues.apache.org/jira/browse/NIFI-4273
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.3.0
> Environment: ubuntu 16.04 64bit, openjdk version "1.8.0_131"
> OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-2ubuntu1.16.04.3-b11)
> OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)
>Reporter: WaterslideKid
>Assignee: Koji Kawamura
>Priority: Minor
> Fix For: 1.7.0
>
>
> Issue:
>Data access using site-to-site binding is based on local environment and
> does not easily support a reverse proxy configuration. The UI works great
> under a reverse proxy configuration, but a site to site configuration fails.
> It appears that the underlying site to site data access URLs are built from
> local environment information rather than configuration and do not honor the
> SSL X-ProxiedEntitiesChain or other reverse proxy context information to
> build proper URLs.
> My configuration:
> I have two nifi systems that I want to connect to each other via NIFI site to
> site. One is monitoring logs and intending to publish these logs by offering
> a site to site web service/server in order to be collected by a client nifi.
> My desire is to have the client nifi contact the server to pull logs by
> connecting through a nginx reverse proxy. All of this is done over SSL using
> mutual SSL. I'm able to get these systems talking to each over directly but
> having issues talking through a reverse proxy.
> In my preferred configuration, the nifi server behind the reverse proxy is
> bound to localhost on port 8443. The reverse proxy is bound to an external
> ip address on port 4443. The client is set up to talk to the server using
> site to iste over the reverse proxy on the external ip on port 4443. In the
> UI, the configuration is able to view the flow ports for site to site but
> data refuses to flow. The client complains that it is unable to talk with
> the data channel via site to site to the server because it is unable to
> connect to :8443 - which is incorrect. It appears that the site to
> site data channel gets its configuration by choosing the hostname and port of
> the nifi service rather than obeying the reverse proxy configuration of the
> UI. There is no way to configure the client or server to use an alternative
> port or hostname for site to site data.
>I have a successful configuration of a mutual SSL enabled NIFI reverse
> proxy that allows UI clients to access a NIFI server and pass on client SSL
> information.
>Here is my site to site configuration
> {code:java}
> # --- the following is a snipplet from an nginx configuration
> listen 4443 ssl default_server;
> ssl_certificate /etc/nginx/cert/nifi/server.pem;
> ssl_certificate_key /etc/nginx/cert/nifi/server.key;
> #require clients to authenticate with certs that are part of CA
> #force mutual SSL
> ssl_client_certificate /etc/nginx/cert/nifi/ca.pem;
> ssl_verify_client on;
> ssl on;
>location / {
> #give the reverse proxy server a client certificate for nifi
> proxy_ssl_certificate /etc/nginx/cert/nifi/rproxy.pem;
> proxy_ssl_certificate_key /etc/nginx/cert/nifi/rproxy.key;
>
> #note that you have to give this client the "proxy user
> requests"
> #access policy so that it can pass on clients using the
> # X-ProxiedEntitiesChain header -- only valid under mutual ssl
> #do some other proxy optimization
> proxy_buffering off;
> #the following are special headers needed by nifi to build
> proper urls
> proxy_set_header X-ProxyScheme https;
> proxy_set_header X-ProxyHost $host;
> proxy_set_header X-ProxyPort $server_port;
> proxy_set_header
[jira] [Updated] (NIFI-5739) CaptureChangeMySQL should maintain its JDBC connection automatically
[
https://issues.apache.org/jira/browse/NIFI-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Villard updated NIFI-5739:
-
Resolution: Fixed
Fix Version/s: 1.9.0
Status: Resolved (was: Patch Available)
> CaptureChangeMySQL should maintain its JDBC connection automatically
>
>
> Key: NIFI-5739
> URL: https://issues.apache.org/jira/browse/NIFI-5739
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.2.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>Priority: Major
> Fix For: 1.9.0
>
>
> CaptureChangeMySQL uses two different connections to the target MySQL server.
> A) binlog client, B) JDBC client. The JDBC client is used to query table
> column definition, which is not returned by binlog messages.
> CaptureChangeMySQL establishes the JDBC connection when it starts. If the
> connection is not used for the next configured wait_timeout period (defaults
> to 28800 secs), the connection will be thought as an idle connection and
> closed by MySQL server. That happens if no new table gets created, and no DML
> is executed for the new table. The binlog client can stay connected in this
> situation.
> After the connection got closed, if a DML event is received, and if its table
> column info is not known yet, then CaptureChangeMySQL will get the table
> column info by executing a query using the connection. As a result, following
> error occurs:
> {code:java}
> org.apache.nifi.processor.exception.ProcessException: java.io.IOException:
> The last packet successfully received from the server was 55,237 milliseconds
> ago. The last packet sent successfully to the server was 55,238 milliseconds
> ago. is longer than the server configured value of 'wait_timeout'. You should
> consider either expiring and/or testing connection validity before use in
> your application, increasing the server configured values for client
> timeouts, or using the Connector/J connection property 'autoReconnect=true'
> to avoid this problem.
> at
> org.apache.nifi.cdc.mysql.processors.CaptureChangeMySQL.onTrigger(CaptureChangeMySQL.java:593)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: The last packet successfully received from
> the server was 55,237 milliseconds ago. The last packet sent successfully to
> the server was 55,238 milliseconds ago. is longer than the server configured
> value of 'wait_timeout'. You should consider either expiring and/or testing
> connection validity before use in your application, increasing the server
> configured values for client timeouts, or using the Connector/J connection
> property 'autoReconnect=true' to avoid this problem.
> at
> org.apache.nifi.cdc.mysql.processors.CaptureChangeMySQL.outputEvents(CaptureChangeMySQL.java:772)
> at
> org.apache.nifi.cdc.mysql.processors.CaptureChangeMySQL.onTrigger(CaptureChangeMySQL.java:571)
> ... 10 common frames omitted
> {code}
> CaptureChangeMySQL should maintain underlying JDBC connection automatically,
> so that the processor can keep running.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5739) CaptureChangeMySQL should maintain its JDBC connection automatically
[
https://issues.apache.org/jira/browse/NIFI-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661861#comment-16661861
]
ASF GitHub Bot commented on NIFI-5739:
--
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/3103
> CaptureChangeMySQL should maintain its JDBC connection automatically
>
>
> Key: NIFI-5739
> URL: https://issues.apache.org/jira/browse/NIFI-5739
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.2.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>Priority: Major
>
> CaptureChangeMySQL uses two different connections to the target MySQL server.
> A) binlog client, B) JDBC client. The JDBC client is used to query table
> column definition, which is not returned by binlog messages.
> CaptureChangeMySQL establishes the JDBC connection when it starts. If the
> connection is not used for the next configured wait_timeout period (defaults
> to 28800 secs), the connection will be thought as an idle connection and
> closed by MySQL server. That happens if no new table gets created, and no DML
> is executed for the new table. The binlog client can stay connected in this
> situation.
> After the connection got closed, if a DML event is received, and if its table
> column info is not known yet, then CaptureChangeMySQL will get the table
> column info by executing a query using the connection. As a result, following
> error occurs:
> {code:java}
> org.apache.nifi.processor.exception.ProcessException: java.io.IOException:
> The last packet successfully received from the server was 55,237 milliseconds
> ago. The last packet sent successfully to the server was 55,238 milliseconds
> ago. is longer than the server configured value of 'wait_timeout'. You should
> consider either expiring and/or testing connection validity before use in
> your application, increasing the server configured values for client
> timeouts, or using the Connector/J connection property 'autoReconnect=true'
> to avoid this problem.
> at
> org.apache.nifi.cdc.mysql.processors.CaptureChangeMySQL.onTrigger(CaptureChangeMySQL.java:593)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: The last packet successfully received from
> the server was 55,237 milliseconds ago. The last packet sent successfully to
> the server was 55,238 milliseconds ago. is longer than the server configured
> value of 'wait_timeout'. You should consider either expiring and/or testing
> connection validity before use in your application, increasing the server
> configured values for client timeouts, or using the Connector/J connection
> property 'autoReconnect=true' to avoid this problem.
> at
> org.apache.nifi.cdc.mysql.processors.CaptureChangeMySQL.outputEvents(CaptureChangeMySQL.java:772)
> at
> org.apache.nifi.cdc.mysql.processors.CaptureChangeMySQL.onTrigger(CaptureChangeMySQL.java:571)
> ... 10 common frames omitted
> {code}
> CaptureChangeMySQL should maintain underlying JDBC connection automatically,
> so that the processor can keep running.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5739) CaptureChangeMySQL should maintain its JDBC connection automatically
[
https://issues.apache.org/jira/browse/NIFI-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661859#comment-16661859
]
ASF subversion and git services commented on NIFI-5739:
---
Commit d28b1172db974cb1bd6aeba479b7655ce89c42db in nifi's branch
refs/heads/master from [~ijokarumawak]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=d28b117 ]
NIFI-5739: Maintain CaptureChangeMySQL JDBC connection automatically
Signed-off-by: Pierre Villard
This closes #3103.
> CaptureChangeMySQL should maintain its JDBC connection automatically
>
>
> Key: NIFI-5739
> URL: https://issues.apache.org/jira/browse/NIFI-5739
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.2.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>Priority: Major
>
> CaptureChangeMySQL uses two different connections to the target MySQL server.
> A) binlog client, B) JDBC client. The JDBC client is used to query table
> column definition, which is not returned by binlog messages.
> CaptureChangeMySQL establishes the JDBC connection when it starts. If the
> connection is not used for the next configured wait_timeout period (defaults
> to 28800 secs), the connection will be thought as an idle connection and
> closed by MySQL server. That happens if no new table gets created, and no DML
> is executed for the new table. The binlog client can stay connected in this
> situation.
> After the connection got closed, if a DML event is received, and if its table
> column info is not known yet, then CaptureChangeMySQL will get the table
> column info by executing a query using the connection. As a result, following
> error occurs:
> {code:java}
> org.apache.nifi.processor.exception.ProcessException: java.io.IOException:
> The last packet successfully received from the server was 55,237 milliseconds
> ago. The last packet sent successfully to the server was 55,238 milliseconds
> ago. is longer than the server configured value of 'wait_timeout'. You should
> consider either expiring and/or testing connection validity before use in
> your application, increasing the server configured values for client
> timeouts, or using the Connector/J connection property 'autoReconnect=true'
> to avoid this problem.
> at
> org.apache.nifi.cdc.mysql.processors.CaptureChangeMySQL.onTrigger(CaptureChangeMySQL.java:593)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: The last packet successfully received from
> the server was 55,237 milliseconds ago. The last packet sent successfully to
> the server was 55,238 milliseconds ago. is longer than the server configured
> value of 'wait_timeout'. You should consider either expiring and/or testing
> connection validity before use in your application, increasing the server
> configured values for client timeouts, or using the Connector/J connection
> property 'autoReconnect=true' to avoid this problem.
> at
> org.apache.nifi.cdc.mysql.processors.CaptureChangeMySQL.outputEvents(CaptureChangeMySQL.java:772)
> at
> org.apache.nifi.cdc.mysql.processors.CaptureChangeMySQL.onTrigger(CaptureChangeMySQL.java:571)
> ... 10 common frames omitted
> {code}
> CaptureChangeMySQL should maintain underlying JDBC connection automatically,
> so that the processor can keep running.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3103: NIFI-5739: Maintain CaptureChangeMySQL JDBC connect...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3103 ---
[jira] [Commented] (NIFI-5739) CaptureChangeMySQL should maintain its JDBC connection automatically
[
https://issues.apache.org/jira/browse/NIFI-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661856#comment-16661856
]
ASF GitHub Bot commented on NIFI-5739:
--
Github user pvillard31 commented on the issue:
https://github.com/apache/nifi/pull/3103
+1, merging to master, thanks @ijokarumawak
> CaptureChangeMySQL should maintain its JDBC connection automatically
>
>
> Key: NIFI-5739
> URL: https://issues.apache.org/jira/browse/NIFI-5739
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.2.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>Priority: Major
>
> CaptureChangeMySQL uses two different connections to the target MySQL server.
> A) binlog client, B) JDBC client. The JDBC client is used to query table
> column definition, which is not returned by binlog messages.
> CaptureChangeMySQL establishes the JDBC connection when it starts. If the
> connection is not used for the next configured wait_timeout period (defaults
> to 28800 secs), the connection will be thought as an idle connection and
> closed by MySQL server. That happens if no new table gets created, and no DML
> is executed for the new table. The binlog client can stay connected in this
> situation.
> After the connection got closed, if a DML event is received, and if its table
> column info is not known yet, then CaptureChangeMySQL will get the table
> column info by executing a query using the connection. As a result, following
> error occurs:
> {code:java}
> org.apache.nifi.processor.exception.ProcessException: java.io.IOException:
> The last packet successfully received from the server was 55,237 milliseconds
> ago. The last packet sent successfully to the server was 55,238 milliseconds
> ago. is longer than the server configured value of 'wait_timeout'. You should
> consider either expiring and/or testing connection validity before use in
> your application, increasing the server configured values for client
> timeouts, or using the Connector/J connection property 'autoReconnect=true'
> to avoid this problem.
> at
> org.apache.nifi.cdc.mysql.processors.CaptureChangeMySQL.onTrigger(CaptureChangeMySQL.java:593)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: The last packet successfully received from
> the server was 55,237 milliseconds ago. The last packet sent successfully to
> the server was 55,238 milliseconds ago. is longer than the server configured
> value of 'wait_timeout'. You should consider either expiring and/or testing
> connection validity before use in your application, increasing the server
> configured values for client timeouts, or using the Connector/J connection
> property 'autoReconnect=true' to avoid this problem.
> at
> org.apache.nifi.cdc.mysql.processors.CaptureChangeMySQL.outputEvents(CaptureChangeMySQL.java:772)
> at
> org.apache.nifi.cdc.mysql.processors.CaptureChangeMySQL.onTrigger(CaptureChangeMySQL.java:571)
> ... 10 common frames omitted
> {code}
> CaptureChangeMySQL should maintain underlying JDBC connection automatically,
> so that the processor can keep running.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #3103: NIFI-5739: Maintain CaptureChangeMySQL JDBC connection aut...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3103 +1, merging to master, thanks @ijokarumawak ---
[jira] [Commented] (NIFI-5714) Hive[3]ConnectionPool - Kerberos Authentication issue/misleading
[
https://issues.apache.org/jira/browse/NIFI-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661848#comment-16661848
]
ASF GitHub Bot commented on NIFI-5714:
--
Github user pvillard31 commented on the issue:
https://github.com/apache/nifi/pull/3086
Hey @mattyb149 - I gave up on the unit test and added the Ignore
annotation. I confirmed in my environment that the controller service won't be
enabled if Kerberos authentication fails and that we are not logging the
"authentication successful" message anymore.
> Hive[3]ConnectionPool - Kerberos Authentication issue/misleading
>
>
> Key: NIFI-5714
> URL: https://issues.apache.org/jira/browse/NIFI-5714
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
>
> In {{HiveConnectionPool}} and {{Hive3ConnectionPool}}, in the {{@OnEnabled}}
> method, we have:
> {code:java}
> log.info("Hive Security Enabled, logging in as principal {} with keytab {}",
> new Object[] {resolvedPrincipal, resolvedKeytab});
> try {
> ugi = hiveConfigurator.authenticate(hiveConfig, resolvedPrincipal,
> resolvedKeytab);
> } catch (AuthenticationFailedException ae) {
> log.error(ae.getMessage(), ae);
> }
> getLogger().info("Successfully logged in as principal {} with keytab {}", new
> Object[] {resolvedPrincipal, resolvedKeytab});{code}
> Which causes two issues:
> * we're logging the successful message even though the authentication failed
> * the Hive connection is created using the NiFi user identity (this would
> need to be confirmed but that's what I observed during a test - it could be
> due to the environment though)
> In my opinion, an {{InitializationException}} should be thrown so that the
> controller service is not enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #3086: NIFI-5714 - Hive[3]ConnectionPool - Kerberos Authenticatio...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3086 Hey @mattyb149 - I gave up on the unit test and added the Ignore annotation. I confirmed in my environment that the controller service won't be enabled if Kerberos authentication fails and that we are not logging the "authentication successful" message anymore. ---
[jira] [Commented] (NIFI-5742) StopWatch.getDuration produces incorrect results
[ https://issues.apache.org/jira/browse/NIFI-5742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661809#comment-16661809 ] Koji Kawamura commented on NIFI-5742: - [~harschware] Thanks for reporting the issue. I think returning duration if getElapsed() is called when duration > -1 is the safe and most meaningful way to fix this. Do you want to submit a patch for this? > StopWatch.getDuration produces incorrect results > > > Key: NIFI-5742 > URL: https://issues.apache.org/jira/browse/NIFI-5742 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 0.8.0 >Reporter: Tim Harsch >Priority: Major > > getDuration will produce unexpected results when the StopWatch has been > stopped. > See: > https://github.com/apache/nifi/blob/02261311b3b3f765ebb394f8f101b0373a7fb3ab/nifi-commons/nifi-utils/src/main/java/org/apache/nifi/util/StopWatch.java#L77 > Currently, if the stopwatch has been stopped it returns the current nanos > since epoch - (-1), which is meaningless at best and likely to cause issues > for applications using this method. > It should either return an exception because the elapsed time has no meaning > after being stopped, or return the elapsed time when the watch was stopped, > or just keep going and return the current value. > The following gist demonstrates the problem: > https://gist.github.com/harschware/6974b61837618574bae6ab75369ead30 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
