[
https://issues.apache.org/jira/browse/HDDS-16121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18103894#comment-18103894
]
Uma Maheswara Rao G commented on HDDS-16121:
--------------------------------------------
+1 on exploring to upgrade to Jetty 12+.
Jetty 12+ version is coming with Retaining Buffers, which may help to explore
the zero copy options at S3G.
> Use Jetty VirtualThreadPool for S3G
> -----------------------------------
>
> Key: HDDS-16121
> URL: https://issues.apache.org/jira/browse/HDDS-16121
> Project: Apache Ozone
> Issue Type: Improvement
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
>
> Currently, S3G use the blocking thread-per-request threading model. This has
> the advantage of being simple to implement (as opposed to the asynchronous IO
> event loop model).
> However, we encountered performance issues where S3G HTTP threads are blocked
> which causes the threads to hit hadoop.http.max.threads and therefore causes
> the subsequent requests to be waiting in the waitingTaskCount which increases
> the latenncy. This issue can happen even if the downstream services are fine
> (i.e. Ozone cluster is not slow or stuck).
> For example, if an HTTP client stalled while downloading a key from S3G (e.g.
> client machine is slow, etc), the S3G thread will be blocked serving this
> read request until hadoop.http.idle_timeout.ms which triggers the "Idle
> timeout expired". Check the Jetty IdleTimeout.java for the logic for idle,
> the idea is if no data is written in the socket for a while (e.g. notIdle is
> not invoked), a scheduled task will fail this request to prevent request
> hogging the thread for too long.
> {code:java}
> SEVERE: An I/O error has occurred while writing a response message entity to
> the container output stream.
> org.glassfish.jersey.server.internal.process.MappableException:
> java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout
> expired: 60000/60000 ms
> at
> org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:67)
> at
> org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
> at
> org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1116)
> at
> org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:649)
> at
> org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:380)
> at
> org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:370)
> {code}
> Meaning that a single request can block a single thread although there are
> still a lot of CPU capacity left.
> One idea is to migrate the Jetty threading model to Servlet 3.1 Asynchronous
> IO which implements WriteListener and ReadListener, this would prevent a
> single request holding the thread for too long. However, this requires
> changes in the S3G implementation which is tedious and risky.
> A better solution is to use virtual threads since we already support JDK21+
> version. Since virtual threads are not real threads, a single OS thread can
> be mapped to multiple virtual threads and blocked virtual threads should not
> cause OS thread to be blocked since OS thread will pick another virtual
> thread which can do useful work. Additionally, since the threading model of
> virtual thread is similar to OS thread, we don't need to change threading
> model so the risk is lower. Of course, there are some risks in virtual
> threads (i.e. thread pinning, overloading the downstream services, etc), so
> we should still evaluate it or we can make a flag to whether enable or
> disable virtual threads and do canary testing.
> This requires Jetty upgrade, but Jetty upgrade is long overdue (we are still
> using a very old 9.x Jetty version) and we should do it asap. See
> [https://jetty.org/docs/jetty/12.1/programming-guide/arch/threads.html] for
> the Jetty virtual thread support.
> Note: I would like to hear from people who have worked with virtual threads
> before.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
