[
https://issues.apache.org/jira/browse/HDDS-9228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tanvi Penumudy resolved HDDS-9228.
----------------------------------
Resolution: Fixed
> Poor S3G read performance
> -------------------------
>
> Key: HDDS-9228
> URL: https://issues.apache.org/jira/browse/HDDS-9228
> Project: Apache Ozone
> Issue Type: Bug
> Components: S3
> Affects Versions: 1.4.0
> Reporter: Kirill Sizov
> Assignee: Tanvi Penumudy
> Priority: Critical
> Labels: pull-request-available
>
> h3. TL;DR:
> *S3G writes all its responses byte-after-byte.*
> h3. Details
> This issue was discovered during a performance test run
> h4. Cluster configuration
> 3 master nodes, 5 datanodes.
> Each machine runs 96core CPU.
> S3G instances are installed on master nodes (3 gateways).
> h4. Test preparation
> Before the test we uploaded 300000 files to Ozone, 20MB each.
> h4. Test configuration
> We ran two tests
> 1. pure writes, no concurrent reads
> 2. pure reads, no concurrent writes
> h4. Load generator
> 3 load generator nodes, each runs 50 threads.
> h4. Ozone configuration
> The buckets were created with Erasure Coding RS-3-2-1024k
> h3. Results
> We found that writes are 3 times faster than reads, moreover reads caused
> ~70% CPU usage.
> Thread dumps and JFR showed the following stacktraces of HTTP threads:
> Stacktrace:
> {noformat}
> "qtp2079179914-1055393" Id=1055393 RUNNABLE
> at
> org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.write(ResponseWriter.java:291)
> at
> org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:215)
> at java.io.FilterOutputStream.write(FilterOutputStream.java:77)
> at java.io.FilterOutputStream.write(FilterOutputStream.java:125)
> at
> org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:276)
> at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1310)
> at org.apache.commons.io.IOUtils.copy(IOUtils.java:978)
> at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1282)
> at
> org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.lambda$get$0(ObjectEndpoint.java:382)
> {noformat}
> JFR:
> {noformat}
> Stack Trace Count Percentage
> void org.eclipse.jetty.server.HttpOutput.write(int) 431146 39 %
> void
> org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.write(int)
> 431145 39 %
> void org.glassfish.jersey.message.internal.CommittingOutputStream.write(int)
> 431145 39 %
> void java.io.FilterOutputStream.write(int) 431145 39 %
> void java.io.FilterOutputStream.write(byte[], int, int) 431145 39 %
> void
> org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(byte[],
> int, int) 431145 39 %
> long org.apache.commons.io.IOUtils.copyLarge(InputStream, OutputStream,
> byte[]) 431145 39 %
> {noformat}
> We can clearly see the transition {{FilterOutputStream.write(byte[], int,
> int) -> FilterOutputStream.write(int)}}, meaning that any incoming array is
> written as single bytes, not as an array as a whole.
> The place in the code that creates {{FilterOutputStream}} is
> {{org.apache.hadoop.ozone.s3.TracingFilter}}:
> {code}
> OutputStream out = responseContext.getEntityStream();
> if (out != null) {
> responseContext.setEntityStream(new FilterOutputStream(out) {
> @Override
> public void close() throws IOException {
> super.close();
> finishAndClose(scope, span);
> }
> });
> }
> {code}
> Removing this filter or fixing {{FilterOutputStream.write(byte[], int, int)}}
> method resolves performance issues and we see a *5x better throughput and CPU
> usage around 12%*.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
