GitHub user ijokarumawak opened a pull request: https://github.com/apache/nifi/pull/656
Issue HTTP Site-to-Site with lots of queued flow-file This PR is not an actual fix for NIFI-2259. I just want to share it with @markap14. @markap14 I've been trying to reproduce the issue about PipedInputStream.awaitSpace stops working when there're lots of small flow-files to be sent, like 10,000 files, 1kb each. But couldn't reproduce it so far. So, I would like to investigate the issue differently. While I was trying to reproduce it, I found another issue related to the amount of queued flow-files. That is NIFI-2259. With this fix, I was able to queue up 60,000 1kb flow files at Site-to-Site client, then successfully transferred it to remote input-port using HTTP Site-to-Site. (I think it can hold more than that, but stopped increasing at that point) I also tested by adding RateControl after remote input-port, simulate the slow data consumption. It worked, too, without stopping or losing data. Would you pull this branch and test it again? I am not thinking this will fix your problem, but certainly make HTTP Site-to-Site works better in a case with lots of flow-files. Also, if you can reproduce the issue: - Please modify org.apache.nifi.remote log level to DEBUG, and share nifi-app.log - Please share whole thread dump. I'd like to see the counter part thread which is draining data from PipedInputStream. - Also, flow template would be appreciated, too Thanks! You can merge this pull request into a Git repository by running: $ git pull https://github.com/ijokarumawak/nifi pr-530-with-NIFI-2259 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/656.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #656 ---- commit 8a2f03e46ef86f213507980f6f24f76588bbcc5e Author: Mark Payne <marka...@hotmail.com> Date: 2016-06-14T18:47:24Z NIFI-1992: Updated site-to-site client and server to support clustered nifi instances commit 715906a12608c17c770759769b834151add30af5 Author: Koji Kawamura <ijokaruma...@gmail.com> Date: 2016-07-15T06:33:36Z NIFI-2259: HTTP Site-to-Site can't handle DEST_FULL HTTP Site-to-Site can't handle TRANSACTION_FINISHED_BUT_DESTINATION_FULL scenario as expected. That happens if the remote NiFi's input port destination relationship becomes full during Site-to-Site client sends data. The data which has already sent to the remote NiFi has to be committed successfully, however, the remote NiFi returns 503 as a response of commit HTTP request. Because it does check port availability. The port availability check shouldn't be done with commit request, since the session at source NiFi has already been committed. The remote NiFi should commit its session as well, and return TRANSACTION_FINISHED_BUT_DESTINATION_FULL response. This fix make remote NiFi keep the handshaken properties when it holds transaction to be committed. Then if a transaction already has handshaken properties, then use it, instead of doing handshake process again. ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---