[jira] [Comment Edited] (NIFI-8968) Improve throughput performance for InvokeHTTP
[
https://issues.apache.org/jira/browse/NIFI-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17808964#comment-17808964
]
Michael W Moser edited comment on NIFI-8968 at 1/20/24 10:20 PM:
-
{{Thanks for trying this out again [~markbean] . I think your results are very
close to my results on row 3,2,1 of my table. Do you have time to try a
configuration that matches row 4 of my table? Essentially this:}}
{{GenerateFlowFile(s) -> RouteOnAttribute}}
{{ | (fileSize < 1_000_000) -> PackageFlowFile (batch
size 500) |}}
{{ | (fileSize >= 1_000_000) -> PackageFlowFile (batch
size 1) -> InvokeHTTP}}
was (Author: mosermw):
{{Thanks for trying this out again [~markbean] . I think your results are very
close to my results on row 3,2,1 of my table. Do you have time to try a
configuration that matches row 4 of my table? Essentially this:}}
{{GenerateFlowFile(s) -> RouteOnAttribute}}
{{ | }}{{(fileSize < 1_000_000) -> PackageFlowFile
(batch size 500) |}}
{{ | (fileSize >= 1_000_000) -> PackageFlowFile (batch
size 1) -> InvokeHTTP}}
> Improve throughput performance for InvokeHTTP
> -
>
> Key: NIFI-8968
> URL: https://issues.apache.org/jira/browse/NIFI-8968
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.14.0
>Reporter: Mark Bean
>Priority: Major
> Attachments: ListenHTTP-BytesOut.PNG, ListenHTTP-FFOut.PNG,
> PostHTTP_vs_InvokeHTTP.json, PostHTTP_vs_InvokeHTTP.xml
>
>
> InvokeHTTP is the preferred processor to use over the deprecated PostHTTP.
> However, PostHTTP outperforms InvokeHTTP (at least in POST mode). A template
> and a JSON file have been attached to this ticket for benchmarking the two
> processors. Using this flow, PostHTTP was observed to have a throughput of
> approximately 5 times greater than InvokeHTTP.
> In addition, it was noted that InvokeHTTP had approximately 5 times as many
> tasks and 5 times the task duration for a given 5 minute stats window. And,
> the statistics of Bytes Read and Bytes Transferred remain at zero for
> InvokeHTTP; this area accurate statistics also needs to be addressed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-8968) Improve throughput performance for InvokeHTTP
[
https://issues.apache.org/jira/browse/NIFI-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17808964#comment-17808964
]
Michael W Moser commented on NIFI-8968:
---
{{Thanks for trying this out again [~markbean] . I think your results are very
close to my results on row 3,2,1 of my table. Do you have time to try a
configuration that matches row 4 of my table? Essentially this:}}
{{GenerateFlowFile(s) -> RouteOnAttribute}}
{{ | }}{{(fileSize < 1_000_000) -> PackageFlowFile
(batch size 500) |}}
{{ | (fileSize >= 1_000_000) -> PackageFlowFile (batch
size 1) -> InvokeHTTP}}
> Improve throughput performance for InvokeHTTP
> -
>
> Key: NIFI-8968
> URL: https://issues.apache.org/jira/browse/NIFI-8968
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.14.0
>Reporter: Mark Bean
>Priority: Major
> Attachments: ListenHTTP-BytesOut.PNG, ListenHTTP-FFOut.PNG,
> PostHTTP_vs_InvokeHTTP.json, PostHTTP_vs_InvokeHTTP.xml
>
>
> InvokeHTTP is the preferred processor to use over the deprecated PostHTTP.
> However, PostHTTP outperforms InvokeHTTP (at least in POST mode). A template
> and a JSON file have been attached to this ticket for benchmarking the two
> processors. Using this flow, PostHTTP was observed to have a throughput of
> approximately 5 times greater than InvokeHTTP.
> In addition, it was noted that InvokeHTTP had approximately 5 times as many
> tasks and 5 times the task duration for a given 5 minute stats window. And,
> the statistics of Bytes Read and Bytes Transferred remain at zero for
> InvokeHTTP; this area accurate statistics also needs to be addressed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-8968) Improve throughput performance for InvokeHTTP
[ https://issues.apache.org/jira/browse/NIFI-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17808962#comment-17808962 ] Mark Bean commented on NIFI-8968: - I performed a similar test by using PackageFlowFile and InvokeHTTP. The flow consisted of 3 GenerateFlowFile processors running simultaneously generating files of sizes 1 KB, 1 MB and 100 MB. Three configurations where run: 1) GenerateFlowFile -> PackageFlowFile -> InvokeHTTP 2) GenerateFlowFile -> InvokeHTTP 3) GenerateFlowFile -> PostHTTP Each configuration was run for approximately 15 minutes. The status history graph for the corresponding ListenHTTP is attached. Bottom line: PostHTTP still shows best throughput. However, PackageFlowFile does improve the overall throughput for InvokeHTTP. > Improve throughput performance for InvokeHTTP > - > > Key: NIFI-8968 > URL: https://issues.apache.org/jira/browse/NIFI-8968 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.14.0 >Reporter: Mark Bean >Priority: Major > Attachments: ListenHTTP-BytesOut.PNG, ListenHTTP-FFOut.PNG, > PostHTTP_vs_InvokeHTTP.json, PostHTTP_vs_InvokeHTTP.xml > > > InvokeHTTP is the preferred processor to use over the deprecated PostHTTP. > However, PostHTTP outperforms InvokeHTTP (at least in POST mode). A template > and a JSON file have been attached to this ticket for benchmarking the two > processors. Using this flow, PostHTTP was observed to have a throughput of > approximately 5 times greater than InvokeHTTP. > In addition, it was noted that InvokeHTTP had approximately 5 times as many > tasks and 5 times the task duration for a given 5 minute stats window. And, > the statistics of Bytes Read and Bytes Transferred remain at zero for > InvokeHTTP; this area accurate statistics also needs to be addressed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-8968) Improve throughput performance for InvokeHTTP
[ https://issues.apache.org/jira/browse/NIFI-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Bean updated NIFI-8968: Attachment: ListenHTTP-BytesOut.PNG ListenHTTP-FFOut.PNG > Improve throughput performance for InvokeHTTP > - > > Key: NIFI-8968 > URL: https://issues.apache.org/jira/browse/NIFI-8968 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.14.0 >Reporter: Mark Bean >Priority: Major > Attachments: ListenHTTP-BytesOut.PNG, ListenHTTP-FFOut.PNG, > PostHTTP_vs_InvokeHTTP.json, PostHTTP_vs_InvokeHTTP.xml > > > InvokeHTTP is the preferred processor to use over the deprecated PostHTTP. > However, PostHTTP outperforms InvokeHTTP (at least in POST mode). A template > and a JSON file have been attached to this ticket for benchmarking the two > processors. Using this flow, PostHTTP was observed to have a throughput of > approximately 5 times greater than InvokeHTTP. > In addition, it was noted that InvokeHTTP had approximately 5 times as many > tasks and 5 times the task duration for a given 5 minute stats window. And, > the statistics of Bytes Read and Bytes Transferred remain at zero for > InvokeHTTP; this area accurate statistics also needs to be addressed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12648) Refactor components in elasticsearch bundle using current API methods
[
https://issues.apache.org/jira/browse/NIFI-12648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17808940#comment-17808940
]
David Handermann commented on NIFI-12648:
-
You're welcome!
> Refactor components in elasticsearch bundle using current API methods
> -
>
> Key: NIFI-12648
> URL: https://issues.apache.org/jira/browse/NIFI-12648
> Project: Apache NiFi
> Issue Type: Sub-task
>Reporter: endzeit
>Assignee: endzeit
>Priority: Minor
>
> Based on improvements to support for {{DescribedValue}} in NiFi API and other
> improvements in Java 21, the _elasticsearch_ bundle components should be
> updated to use current methods.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12648) Refactor components in elasticsearch bundle using current API methods
[
https://issues.apache.org/jira/browse/NIFI-12648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17808939#comment-17808939
]
endzeit commented on NIFI-12648:
Thank you for removing the version information from the Jira ticket
@[~exceptionfactory]. I haven't noticed that it was copied as well.
> Refactor components in elasticsearch bundle using current API methods
> -
>
> Key: NIFI-12648
> URL: https://issues.apache.org/jira/browse/NIFI-12648
> Project: Apache NiFi
> Issue Type: Sub-task
>Reporter: endzeit
>Assignee: endzeit
>Priority: Minor
>
> Based on improvements to support for {{DescribedValue}} in NiFi API and other
> improvements in Java 21, the _elasticsearch_ bundle components should be
> updated to use current methods.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12648) Refactor components in elasticsearch bundle using current API methods
[
https://issues.apache.org/jira/browse/NIFI-12648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-12648:
Fix Version/s: (was: 2.0.0-M2)
> Refactor components in elasticsearch bundle using current API methods
> -
>
> Key: NIFI-12648
> URL: https://issues.apache.org/jira/browse/NIFI-12648
> Project: Apache NiFi
> Issue Type: Sub-task
>Reporter: endzeit
>Assignee: endzeit
>Priority: Minor
>
> Based on improvements to support for {{DescribedValue}} in NiFi API and other
> improvements in Java 21, the _elasticsearch_ bundle components should be
> updated to use current methods.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12646) Conduct Apache NiFi 2.0.0-M2 Release
[ https://issues.apache.org/jira/browse/NIFI-12646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17808929#comment-17808929 ] ASF subversion and git services commented on NIFI-12646: Commit b462c7051d004be70fba34f2795bd5c682cd1124 in nifi's branch refs/heads/NIFI-12646-RC1 from David Handermann [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=b462c7051d ] NIFI-12646-RC1 prepare release nifi-2.0.0-M2-RC1 > Conduct Apache NiFi 2.0.0-M2 Release > > > Key: NIFI-12646 > URL: https://issues.apache.org/jira/browse/NIFI-12646 > Project: Apache NiFi > Issue Type: Task >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Fix For: 2.0.0-M2 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12646) Conduct Apache NiFi 2.0.0-M2 Release
[ https://issues.apache.org/jira/browse/NIFI-12646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17808930#comment-17808930 ] ASF subversion and git services commented on NIFI-12646: Commit a624db46fdf2b7f26fe7f6d01667a0da465a7b8e in nifi's branch refs/heads/NIFI-12646-RC1 from David Handermann [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=a624db46fd ] NIFI-12646-RC1 prepare for next development iteration > Conduct Apache NiFi 2.0.0-M2 Release > > > Key: NIFI-12646 > URL: https://issues.apache.org/jira/browse/NIFI-12646 > Project: Apache NiFi > Issue Type: Task >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Fix For: 2.0.0-M2 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12646) Conduct Apache NiFi 2.0.0-M2 Release
[ https://issues.apache.org/jira/browse/NIFI-12646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17808928#comment-17808928 ] ASF subversion and git services commented on NIFI-12646: Commit 7c0fae9877d055bcee6e9c4f091becc1a4def2ed in nifi's branch refs/heads/NIFI-12646-RC1 from David Handermann [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=7c0fae9877 ] NIFI-12646 Set Python Processor version to 2.0.0-M2 Signed-off-by: David Handermann > Conduct Apache NiFi 2.0.0-M2 Release > > > Key: NIFI-12646 > URL: https://issues.apache.org/jira/browse/NIFI-12646 > Project: Apache NiFi > Issue Type: Task >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Fix For: 2.0.0-M2 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12632 Extract SFTP components out of the standard bundle [nifi]
EndzeitBegins commented on PR #8277: URL: https://github.com/apache/nifi/pull/8277#issuecomment-1902038731 I see. Feel free to suggest changes or even close / take over the PR if you think that's more resourceful. I guess as we're close to the next milestone release, it makes sense to postpone work or at least the merge on this until after the release anyway. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
