Surendra Singh Lilhore created HADOOP-19848:
-----------------------------------------------
Summary: AzureNativeFileSystemStore fails with NoSuchMethodError
on Jetty 9.4.57+
Key: HADOOP-19848
URL: https://issues.apache.org/jira/browse/HADOOP-19848
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.4.0
Reporter: Surendra Singh Lilhore
Assignee: Surendra Singh Lilhore
When using hadoop-azure with Apache Spark 4.0, {{AzureNativeFileSystemStore}}
fails to load with:
{noformat}
Caused by: java.lang.ExceptionInInitializerError: Exception
java.lang.NoSuchMethodError: 'java.util.Properties
org.eclipse.jetty.util.log.Log.getProperties()' [in thread "Thread-31"]
at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.createPermissionJsonSerializer(AzureNativeFileSystemStore.java:429)
at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.<clinit>(AzureNativeFileSystemStore.java:331)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.createDefaultStore(NativeAzureFileSystem.java:1485)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.initialize(NativeAzureFileSystem.java:1410)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3615){noformat}
*Root cause:*
{{AzureNativeFileSystemStore.createPermissionJsonSerializer()}} calls
{{org.eclipse.jetty.util.log.Log.getProperties()}} to suppress Jetty's log
announcement. While this method exists in {{jetty-util}} 9.4.57 (as shipped with
Hadoop 3.4.x), Spark 4.0 bundles a different version of Jetty on the runtime
classpath where this method has been removed. This causes a compile-vs-runtime
classpath mismatch — the code compiles fine against Hadoop's Jetty version but
fails at runtime under Spark's classloader.
This is a fragile dependency on a Jetty internal API
({{{}org.eclipse.jetty.util.log.Log{}}}) that is unnecessary. Jetty reads the
{{org.eclipse.jetty.util.log.announce}} property from system properties, so
{{System.setProperty() }}achieves the same effect without coupling to Jetty
internals.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
