henrybear327 commented on code in PR #10760:
URL: https://github.com/apache/ozone/pull/10760#discussion_r4262837832
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/local/LocalOzoneCluster.java:
##########
@@ -254,6 +282,11 @@ public void start() throws Exception {
startOm(prepared.getConfiguration());
startDatanodes(prepared.getDatanodeConfigurations());
waitForClusterReadiness(config.getStartupTimeout());
+ if (config.isReconEnabled()) {
+ startRecon(prepared.getConfiguration());
+ waitForHttpEndpointReadiness(getReconEndpoint(), "Recon",
+ config.getStartupTimeout());
+ }
if (config.isS3gEnabled()) {
startS3Gateway(prepared.getConfiguration());
}
Review Comment:
Starting them concurrently throws an exception for me. I changed `start()`
to start Recon and S3 Gateway on two threads and ran the s3g=true, recon=true
case of `TestLocalOzoneRecon#enabledServicesServeAndStopOnClose` five times,
one JVM per run.
Recon failed every time. S3 Gateway finished starting in under a second, and
Recon failed 1.2 to 1.5 s in, when it started its HTTP server:
```
java.io.IOException: Unable to initialize WebAppContext
at
org.apache.hadoop.hdds.server.http.HttpServer2.start(HttpServer2.java:1369)
at
org.apache.hadoop.hdds.server.http.BaseHttpServer.start(BaseHttpServer.java:357)
at org.apache.hadoop.ozone.recon.ReconServer.start(ReconServer.java:325)
...
Caused by: javax.ws.rs.WebApplicationException: Trying to register multiple
service locators into single service locator application.
at
org.glassfish.jersey.ext.cdi1x.internal.SingleInjectionManagerStore.registerInjectionManager(SingleInjectionManagerStore.java:40)
at
org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider.addInjectionManager(CdiComponentProvider.java:569)
at
org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider.initialize(CdiComponentProvider.java:163)
at
org.glassfish.jersey.ext.cdi1x.internal.CdiServerComponentProvider.initialize(CdiServerComponentProvider.java:39)
...
at
org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:154)
```
---
I tired to debug the situation and it seems to be related to Jersey's CDI
integration and the fact that we run everything under one JVM.
The root cause might be at Jersey's CDI integration (jersey-cdi1x), which S3
Gateway uses. When Recon's Jersey app initializes, `CdiServerComponentProvider`
finds a Weld container that S3 Gateway started and tries to register Recon's
app with Jersey's CDI extension in that container. The extension keeps its app
in a `SingleInjectionManagerStore`, which already holds S3 Gateway's app, so it
throws.
The reason for it is that `ozone local` runs every service in one JVM with
one classpath. `ozone-tools` depends on both `ozone-recon` and
`ozone-s3gateway`, so jersey-cdi1x runs for Recon's Jersey app too, and it
looks up the CDI container JVM-wide through `CDI.current()`. Before S3 Gateway
starts there is no Weld container, the lookup returns nothing, and jersey-cdi1x
does nothing for Recon. So Recon has to start before S3 Gateway, which is the
current order.
---
Not entirely sure if the analysis is correct, and if there is any fix we can
go about it in the meantime to make things able to start concurrently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
