[
https://issues.apache.org/jira/browse/ARTEMIS-3449?focusedWorklogId=643861&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-643861
]
ASF GitHub Bot logged work on ARTEMIS-3449:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 31/Aug/21 04:51
Start Date: 31/Aug/21 04:51
Worklog Time Spent: 10m
Work Description: franz1981 edited a comment on pull request #3711:
URL: https://github.com/apache/activemq-artemis/pull/3711#issuecomment-908895949
I see I didn't state clearly on
https://issues.apache.org/jira/browse/ARTEMIS-3449 that the major benefit of
this change is not just speeding up file reading, but it reducing the chances
of OOM due to lack of native resources too (see
[io_util::readBytes](https://github.com/frohoff/jdk8u-jdk/blob/da0da73ab82ed714dc5be94acd2f0d00fbdfe2e9/src/share/native/java/io/io_util.c#L95-L99)
for reference).
In the original code, byte[] was used to perform read vs `SequentialFile`,
using a `RandomAccessFile` under the hood as explained on
https://github.com/apache/activemq-artemis/pull/2844#issuecomment-533606931.
Sadly, `RandomAccessFile::read` is using platform `malloc/free` (for
writing/reading >
[8192](https://github.com/frohoff/jdk8u-jdk/blob/da0da73ab82ed714dc5be94acd2f0d00fbdfe2e9/src/share/native/java/io/io_util.c#L58),
that's > of the AMQP default max frame size on the broker) under the hood that
can massively degrade over time due to fragmentation (if using the default
system memory allocator) until been unable to allocate anything, eventually
causing broker to hit OOM. With the current approach of this PR we both save
this to happen while benefitting from the speedup mentioned in
https://github.com/apache/activemq-artemis/pull/2844#issuecomment-533906809 and
reported here:
```
Benchmark (chunkSize) (fileName)
(fileSize) (ioSize) Mode Cnt Score Error Units
ChunkedFileBenchmark.readFullyFile 32768 /tmp/raf.tmp
10485760 32768 avgt 4 2153.527 ± 715.641 us/op
ChunkedFileBenchmark.readFullyFileWithNio 32768 /tmp/raf.tmp
10485760 32768 avgt 4 1775.999 ± 831.715 us/op
```
`readFullyFile` is using `byte[]` , while `readFullyFileWithNio` is using
direct `ByteBuffer`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@activemq.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 643861)
Time Spent: 4.5h (was: 4h 20m)
> Speedup AMQP large message streaming
> ------------------------------------
>
> Key: ARTEMIS-3449
> URL: https://issues.apache.org/jira/browse/ARTEMIS-3449
> Project: ActiveMQ Artemis
> Issue Type: Improvement
> Reporter: Francesco Nigro
> Assignee: Francesco Nigro
> Priority: Major
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> AMQP is using unpooled heap ByteBuffer(s) to stream AMQP large messages:
> given that the underline NIO sequential file can both use FileChannel or
> RandomAccessFile (depending if the ByteBuffer used is direct/heap based),
> both approaches would benefit from using Netty pooled direct buffers and save
> additional copies (performed by RandomAccessFile) to happen, reducing GC too.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
