[jira] [Commented] (NIFI-2525) HTTP Site-to-Site fails with java.nio.channels.AsynchronousCloseException when sending through proxy that requires authentication
[ https://issues.apache.org/jira/browse/NIFI-2525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15436132#comment-15436132 ] ASF GitHub Bot commented on NIFI-2525: -- Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/915 @markap14 Thanks for reviewing the code! @YolandaMDavis Thanks for the functional test and detailed test result report! > HTTP Site-to-Site fails with java.nio.channels.AsynchronousCloseException > when sending through proxy that requires authentication > - > > Key: NIFI-2525 > URL: https://issues.apache.org/jira/browse/NIFI-2525 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.0.0 >Reporter: Koji Kawamura >Assignee: Koji Kawamura > Fix For: 1.0.0 > > > Pulling data using Remote Process Group from output port works. > However, pushing data using Remote Process Group to input port fails with > AsynchronousCloseException. > A RPG sends data via POST, then a proxy server returns 407: proxy auth > required. After this, the RPG should resend the request with credential, but > the data channel is already closed. > Currently, it uses chunked encoding so that it can stream data to send. > Sending actual data twice won't be efficient. We need to do the > authentication before start reading flow-file stream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2525) HTTP Site-to-Site fails with java.nio.channels.AsynchronousCloseException when sending through proxy that requires authentication
[ https://issues.apache.org/jira/browse/NIFI-2525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15436040#comment-15436040 ] ASF GitHub Bot commented on NIFI-2525: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/915 > HTTP Site-to-Site fails with java.nio.channels.AsynchronousCloseException > when sending through proxy that requires authentication > - > > Key: NIFI-2525 > URL: https://issues.apache.org/jira/browse/NIFI-2525 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.0.0 >Reporter: Koji Kawamura > Fix For: 1.0.0 > > > Pulling data using Remote Process Group from output port works. > However, pushing data using Remote Process Group to input port fails with > AsynchronousCloseException. > A RPG sends data via POST, then a proxy server returns 407: proxy auth > required. After this, the RPG should resend the request with credential, but > the data channel is already closed. > Currently, it uses chunked encoding so that it can stream data to send. > Sending actual data twice won't be efficient. We need to do the > authentication before start reading flow-file stream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2525) HTTP Site-to-Site fails with java.nio.channels.AsynchronousCloseException when sending through proxy that requires authentication
[ https://issues.apache.org/jira/browse/NIFI-2525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15436038#comment-15436038 ] ASF subversion and git services commented on NIFI-2525: --- Commit 671301193bf5b05a42d0e19aaa9503424ec04bc3 in nifi's branch refs/heads/master from [~ijokarumawak] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=6713011 ] NIFI-2525: Fix Proxy auth issue with async send. Without this fix, NiFi fails to send data via HTTP Site-to-Site through Proxy which requires authentication due to AsynchronousCloseException. It happens when async client replays producing contents in order to re-send the request with auth credential for the proxy server, however the connection is already closed. This fix makes NiFi to send actual data only at the second round of requests, so that flow-file contents can be sent without reading it twice. Signed-off-by: Yolanda M. Davis This closes #915 > HTTP Site-to-Site fails with java.nio.channels.AsynchronousCloseException > when sending through proxy that requires authentication > - > > Key: NIFI-2525 > URL: https://issues.apache.org/jira/browse/NIFI-2525 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.0.0 >Reporter: Koji Kawamura > Fix For: 1.0.0 > > > Pulling data using Remote Process Group from output port works. > However, pushing data using Remote Process Group to input port fails with > AsynchronousCloseException. > A RPG sends data via POST, then a proxy server returns 407: proxy auth > required. After this, the RPG should resend the request with credential, but > the data channel is already closed. > Currently, it uses chunked encoding so that it can stream data to send. > Sending actual data twice won't be efficient. We need to do the > authentication before start reading flow-file stream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2525) HTTP Site-to-Site fails with java.nio.channels.AsynchronousCloseException when sending through proxy that requires authentication
[ https://issues.apache.org/jira/browse/NIFI-2525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15435987#comment-15435987 ] ASF GitHub Bot commented on NIFI-2525: -- Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/915 I setup 2 standalone nifi instances on separate hosts and created a 3rd host to serve as proxy server using Apache Web. NiFi server A was set as the push server which generated 10MB Flow Files every 10 seconds. Server A referred to server B via a RPG. Server B received input flowfiles and saved them to disk via PutFile. The following test configurations were performed: 1) Unsecured NiFi instances with no Proxy - Data was exchanged and saved on Server B as expected 2) Unsecured NiFi instance with Proxy (Basic/Digest Auth) - when attempting to post via Proxy on Server B endpoint nifi-api/data-transfer/input-ports, proxy logged connection attempt with Status 407 415 (authentication required, post method). Proxy logged subsequent responses of 201 or 202 for these attempts. For deletes, 407/415 were logged and subsequent 200 was received (OK). No exception seen in log on either server. Data exchange completed successfully 4) Secured NiFi Instances with no Proxy - After required policy updates in NiFi data was exchanged and performed as expected 5) Secured NiFi Instances with Proxy (Basic/Digest Auth) - Results same as test #2 however proxy logs did not detail resource endpoints accessed (I believe due to SSL enabled) 6) Secured NiFi instances with Proxy (No User/Pass Provided) - NiFi logged error indicating : Unable to refresh Remote Group's peers due to response code 407:Proxy Authentication Required with explanation: null. Error Message was expected and resolved once user/pass provided for proxy. Given the above tests I believe that proxy authentication is working as expected with this patch +1 Will merge into master shortly > HTTP Site-to-Site fails with java.nio.channels.AsynchronousCloseException > when sending through proxy that requires authentication > - > > Key: NIFI-2525 > URL: https://issues.apache.org/jira/browse/NIFI-2525 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.0.0 >Reporter: Koji Kawamura > Fix For: 1.0.0 > > > Pulling data using Remote Process Group from output port works. > However, pushing data using Remote Process Group to input port fails with > AsynchronousCloseException. > A RPG sends data via POST, then a proxy server returns 407: proxy auth > required. After this, the RPG should resend the request with credential, but > the data channel is already closed. > Currently, it uses chunked encoding so that it can stream data to send. > Sending actual data twice won't be efficient. We need to do the > authentication before start reading flow-file stream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2525) HTTP Site-to-Site fails with java.nio.channels.AsynchronousCloseException when sending through proxy that requires authentication
[ https://issues.apache.org/jira/browse/NIFI-2525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15434968#comment-15434968 ] ASF GitHub Bot commented on NIFI-2525: -- Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/915 @markap14 I can test functionally as well > HTTP Site-to-Site fails with java.nio.channels.AsynchronousCloseException > when sending through proxy that requires authentication > - > > Key: NIFI-2525 > URL: https://issues.apache.org/jira/browse/NIFI-2525 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.0.0 >Reporter: Koji Kawamura > Fix For: 1.0.0 > > > Pulling data using Remote Process Group from output port works. > However, pushing data using Remote Process Group to input port fails with > AsynchronousCloseException. > A RPG sends data via POST, then a proxy server returns 407: proxy auth > required. After this, the RPG should resend the request with credential, but > the data channel is already closed. > Currently, it uses chunked encoding so that it can stream data to send. > Sending actual data twice won't be efficient. We need to do the > authentication before start reading flow-file stream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2525) HTTP Site-to-Site fails with java.nio.channels.AsynchronousCloseException when sending through proxy that requires authentication
[ https://issues.apache.org/jira/browse/NIFI-2525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15434953#comment-15434953 ] ASF GitHub Bot commented on NIFI-2525: -- Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/915 I've reviewed the code here, and from the perspective of a code review I am a +1. I have not yet had a chance to test this, though. Will try to get to that today. If anyone else has a chance to review functionality, that would be great, as well. > HTTP Site-to-Site fails with java.nio.channels.AsynchronousCloseException > when sending through proxy that requires authentication > - > > Key: NIFI-2525 > URL: https://issues.apache.org/jira/browse/NIFI-2525 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.0.0 >Reporter: Koji Kawamura > Fix For: 1.0.0 > > > Pulling data using Remote Process Group from output port works. > However, pushing data using Remote Process Group to input port fails with > AsynchronousCloseException. > A RPG sends data via POST, then a proxy server returns 407: proxy auth > required. After this, the RPG should resend the request with credential, but > the data channel is already closed. > Currently, it uses chunked encoding so that it can stream data to send. > Sending actual data twice won't be efficient. We need to do the > authentication before start reading flow-file stream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2525) HTTP Site-to-Site fails with java.nio.channels.AsynchronousCloseException when sending through proxy that requires authentication
[ https://issues.apache.org/jira/browse/NIFI-2525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15433834#comment-15433834 ] ASF GitHub Bot commented on NIFI-2525: -- Github user ijokarumawak commented on a diff in the pull request: https://github.com/apache/nifi/pull/915#discussion_r75969046 --- Diff: nifi-assembly/NOTICE --- @@ -909,6 +909,11 @@ The following binary components are provided under the Apache Software License v Expert Group and released to the public domain, as explained at http://creativecommons.org/publicdomain/zero/1.0/ + (ASLv2) LittleProxy --- End diff -- @markap14 Thanks for pointing that out. Removed it. > HTTP Site-to-Site fails with java.nio.channels.AsynchronousCloseException > when sending through proxy that requires authentication > - > > Key: NIFI-2525 > URL: https://issues.apache.org/jira/browse/NIFI-2525 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.0.0 >Reporter: Koji Kawamura > Fix For: 1.0.0 > > > Pulling data using Remote Process Group from output port works. > However, pushing data using Remote Process Group to input port fails with > AsynchronousCloseException. > A RPG sends data via POST, then a proxy server returns 407: proxy auth > required. After this, the RPG should resend the request with credential, but > the data channel is already closed. > Currently, it uses chunked encoding so that it can stream data to send. > Sending actual data twice won't be efficient. We need to do the > authentication before start reading flow-file stream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2525) HTTP Site-to-Site fails with java.nio.channels.AsynchronousCloseException when sending through proxy that requires authentication
[ https://issues.apache.org/jira/browse/NIFI-2525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15432713#comment-15432713 ] ASF GitHub Bot commented on NIFI-2525: -- Github user markap14 commented on a diff in the pull request: https://github.com/apache/nifi/pull/915#discussion_r75856424 --- Diff: nifi-assembly/NOTICE --- @@ -909,6 +909,11 @@ The following binary components are provided under the Apache Software License v Expert Group and released to the public domain, as explained at http://creativecommons.org/publicdomain/zero/1.0/ + (ASLv2) LittleProxy --- End diff -- @ijokarumawak This should not need to be added to the NOTICE, as it is a test dependency only. > HTTP Site-to-Site fails with java.nio.channels.AsynchronousCloseException > when sending through proxy that requires authentication > - > > Key: NIFI-2525 > URL: https://issues.apache.org/jira/browse/NIFI-2525 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.0.0 >Reporter: Koji Kawamura > Fix For: 1.0.0 > > > Pulling data using Remote Process Group from output port works. > However, pushing data using Remote Process Group to input port fails with > AsynchronousCloseException. > A RPG sends data via POST, then a proxy server returns 407: proxy auth > required. After this, the RPG should resend the request with credential, but > the data channel is already closed. > Currently, it uses chunked encoding so that it can stream data to send. > Sending actual data twice won't be efficient. We need to do the > authentication before start reading flow-file stream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2525) HTTP Site-to-Site fails with java.nio.channels.AsynchronousCloseException when sending through proxy that requires authentication
[ https://issues.apache.org/jira/browse/NIFI-2525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15432132#comment-15432132 ] Joseph Witt commented on NIFI-2525: --- nice Koji! Let's def have this on the 1.0 fix list. > HTTP Site-to-Site fails with java.nio.channels.AsynchronousCloseException > when sending through proxy that requires authentication > - > > Key: NIFI-2525 > URL: https://issues.apache.org/jira/browse/NIFI-2525 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.0.0 >Reporter: Koji Kawamura > Fix For: 1.0.0 > > > Pulling data using Remote Process Group from output port works. > However, pushing data using Remote Process Group to input port fails with > AsynchronousCloseException. > A RPG sends data via POST, then a proxy server returns 407: proxy auth > required. After this, the RPG should resend the request with credential, but > the data channel is already closed. > Currently, it uses chunked encoding so that it can stream data to send. > Sending actual data twice won't be efficient. We need to do the > authentication before start reading flow-file stream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2525) HTTP Site-to-Site fails with java.nio.channels.AsynchronousCloseException when sending through proxy that requires authentication
[ https://issues.apache.org/jira/browse/NIFI-2525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15432127#comment-15432127 ] ASF GitHub Bot commented on NIFI-2525: -- GitHub user ijokarumawak opened a pull request: https://github.com/apache/nifi/pull/915 NIFI-2525: Fix Proxy auth issue with async send. Without this fix, NiFi fails to send data via HTTP Site-to-Site through Proxy which requires authentication due to AsynchronousCloseException. It happens when async client replays producing contents in order to re-send the request with auth credential for the proxy server, however the connection is already closed. This fix makes NiFi to send actual data only at the second round of requests, so that flow-file contents can be sent without reading it twice. Unit test cases using LittleProxy are also added to confirm HTTP Site-to-Site works with Proxy with/without authentication. I've tested with Apache mod_proxy and Squid proxy servers with basic/digest auth. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ijokarumawak/nifi nifi-2525 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/915.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #915 commit 12cf7ef186440ef5de07b04ecaa33d93846ece6f Author: Koji Kawamura Date: 2016-08-23T01:57:57Z NIFI-2525: Fix Proxy auth issue with async send. Without this fix, NiFi fails to send data via HTTP Site-to-Site through Proxy which requires authentication due to AsynchronousCloseException. It happens when async client replays producing contents in order to re-send the request with auth credential for the proxy server, however the connection is already closed. This fix makes NiFi to send actual data only at the second round of requests, so that flow-file contents can be sent without reading it twice. > HTTP Site-to-Site fails with java.nio.channels.AsynchronousCloseException > when sending through proxy that requires authentication > - > > Key: NIFI-2525 > URL: https://issues.apache.org/jira/browse/NIFI-2525 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.0.0 >Reporter: Koji Kawamura > > Pulling data using Remote Process Group from output port works. > However, pushing data using Remote Process Group to input port fails with > AsynchronousCloseException. > A RPG sends data via POST, then a proxy server returns 407: proxy auth > required. After this, the RPG should resend the request with credential, but > the data channel is already closed. > Currently, it uses chunked encoding so that it can stream data to send. > Sending actual data twice won't be efficient. We need to do the > authentication before start reading flow-file stream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)