Dan Ziemba created HADOOP-20034:
-----------------------------------
Summary: WebHdfsFileSystem fails with the 3.5.0 shaded client:
hadoop-client-runtime has no JAX-RS implementation
Key: HADOOP-20034
URL: https://issues.apache.org/jira/browse/HADOOP-20034
Project: Hadoop Common
Issue Type: Bug
Components: build
Affects Versions: 3.5.0
Reporter: Dan Ziemba
With hadoop-client-api and hadoop-client-runtime 3.5.0, every WebHDFS call that
parses a JSON response (getFileStatus, listStatus, globStatus, delete, ...)
fails with a ClassNotFoundException for the relocated Jersey
RuntimeDelegateImpl. open() still works because it parses no JSON. The same
client code works with 3.4.x.
WebHdfsFileSystem.jsonParse() calls
javax.ws.rs.core.MediaType.valueOf(contentType), which looks up a JAX-RS
RuntimeDelegate. Since HADOOP-19652, hadoop-client-runtime bundles the
relocated JAX-RS API, but it still excludes org.glassfish.jersey.core:*, so the
default delegate
org.apache.hadoop.shaded.org.glassfish.jersey.internal.RuntimeDelegateImpl is
missing. In 3.4.x the runtime jar bundled Jersey 1's RuntimeDelegateImpl, which
the Jersey 2 upgrade (HADOOP-15984) removed. branch-3.5 still has the same
exclusion and call.
*To reproduce* (Java 21; classpath: hadoop-client-api-3.5.0.jar,
hadoop-client-runtime-3.5.0.jar, slf4j-api, commons-logging; any WebHDFS or
HttpFS endpoint, seen against a 3.4.1 HttpFS):
{code:java}
FileSystem fs =
FileSystem.get(URI.create("webhdfs://namenode.example.com:9870"), new
Configuration());
fs.getFileStatus(new Path("/"));
{code}
With the 3.4.3 jars, the same code returns the file status.
{noformat}
Exception in thread "main" java.io.IOException: namenode.example.com:9870:
Response decoding failure: java.lang.RuntimeException:
java.lang.ClassNotFoundException:
org.apache.hadoop.shaded.org.glassfish.jersey.internal.RuntimeDelegateImpl
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$FsPathResponseRunner.getResponse(WebHdfsFileSystem.java:971)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:829)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:689)
at
java.base/javax.security.auth.Subject.lambda$callAs$0(Subject.java:379)
at
java.base/java.security.AccessController.doPrivileged(AccessController.java:714)
at java.base/javax.security.auth.Subject.doAs(Subject.java:525)
at java.base/javax.security.auth.Subject.callAs(Subject.java:381)
at
org.apache.hadoop.security.authentication.util.SubjectUtil.callAs(SubjectUtil.java:242)
at
org.apache.hadoop.security.authentication.util.SubjectUtil.doAs(SubjectUtil.java:314)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1958)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:685)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:1109)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:1120)
at Repro.main(Repro.java:9)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.hadoop.shaded.org.glassfish.jersey.internal.RuntimeDelegateImpl
at
org.apache.hadoop.shaded.javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:129)
at
org.apache.hadoop.shaded.javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:96)
at
org.apache.hadoop.shaded.javax.ws.rs.core.MediaType.valueOf(MediaType.java:172)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.jsonParse(WebHdfsFileSystem.java:496)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$FsPathResponseRunner.getResponse(WebHdfsFileSystem.java:961)
... 13 more
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.shaded.org.glassfish.jersey.internal.RuntimeDelegateImpl
at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
at
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:423)
at java.base/java.lang.Class.forName(Class.java:414)
at
org.apache.hadoop.shaded.javax.ws.rs.ext.FactoryFinder.newInstance(FactoryFinder.java:87)
at
org.apache.hadoop.shaded.javax.ws.rs.ext.FactoryFinder.find(FactoryFinder.java:185)
at
org.apache.hadoop.shaded.javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:111)
... 17 more
{noformat}
*Possible fixes:* bundle and relocate jersey-common (which has
RuntimeDelegateImpl) in hadoop-client-runtime, or have jsonParse() check the
Content-Type without JAX-RS.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
