[GitHub] nifi pull request #656: Issue HTTP Site-to-Site with lots of queued flow-fil...

[ijokarumawak]

Github user ijokarumawak closed the pull request at:

https://github.com/apache/nifi/pull/656


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #656: Issue HTTP Site-to-Site with lots of queued flow-fil...

[ijokarumawak]

GitHub user ijokarumawak opened a pull request:

https://github.com/apache/nifi/pull/656

Issue HTTP Site-to-Site with lots of queued flow-file

This PR is not an actual fix for NIFI-2259. I just want to share it with 
@markap14.

@markap14 I've been trying to reproduce the issue about 
PipedInputStream.awaitSpace stops working when there're lots of small 
flow-files to be sent, like 10,000 files, 1kb each. But couldn't reproduce it 
so far.

So, I would like to investigate the issue differently. While I was trying 
to reproduce it, I found another issue related to the amount of queued 
flow-files. That is NIFI-2259.

With this fix, I was able to queue up 60,000 1kb flow files at Site-to-Site 
client, then successfully transferred it to remote input-port using HTTP 
Site-to-Site. (I think it can hold more than that, but stopped increasing at 
that point)

I also tested by adding RateControl after remote input-port, simulate the 
slow data consumption. It worked, too, without stopping or losing data.

Would you pull this branch and test it again? I am not thinking this will 
fix your problem, but certainly make HTTP Site-to-Site works better in a case 
with lots of flow-files.

Also, if you can reproduce the issue:
- Please modify org.apache.nifi.remote log level to DEBUG, and share 
nifi-app.log
- Please share whole thread dump. I'd like to see the counter part thread 
which is draining data from PipedInputStream.
- Also, flow template would be appreciated, too

Thanks!

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijokarumawak/nifi pr-530-with-NIFI-2259

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/656.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #656


commit 8a2f03e46ef86f213507980f6f24f76588bbcc5e
Author: Mark Payne 
Date:   2016-06-14T18:47:24Z

NIFI-1992: Updated site-to-site client and server to support clustered nifi 
instances

commit 715906a12608c17c770759769b834151add30af5
Author: Koji Kawamura 
Date:   2016-07-15T06:33:36Z

NIFI-2259: HTTP Site-to-Site can't handle DEST_FULL

HTTP Site-to-Site can't handle TRANSACTION_FINISHED_BUT_DESTINATION_FULL
scenario as expected.

That happens if the remote NiFi's input port destination relationship
becomes full during Site-to-Site client sends data. The data which has
already sent to the remote NiFi has to be committed successfully,
however, the remote NiFi returns 503 as a response of commit HTTP
request. Because it does check port availability.

The port availability check shouldn't be done with commit request, since
the session at source NiFi has already been committed. The remote NiFi
should commit its session as well, and return
TRANSACTION_FINISHED_BUT_DESTINATION_FULL response.

This fix make remote NiFi keep the handshaken properties when it holds
transaction to be committed. Then if a transaction already has
handshaken properties, then use it, instead of doing handshake process
again.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---