JoegenUSTC opened a new issue, #11704:
URL: https://github.com/apache/gravitino/issues/11704
### Version
main branch
### Describe what's wrong
### Bug Description
Since PR #9460 refactored the Hive catalog to use `HiveClient`, several
classes were
moved from `org.apache.gravitino.catalog.hive.*` to a new package
`org.apache.gravitino.hive.*` (e.g., `HiveExceptionConverter`,
`HiveClientPool`).
However, `IsolatedClassLoader.isCatalogClass()` was not updated to recognize
the new
package prefix. As a result, these classes are incorrectly treated as
**shared classes**
and delegated to the server classloader for loading.
### Root Cause
`isCatalogClass()` only checks for `org.apache.gravitino.catalog.*`
sub-packages:
```java
private boolean isCatalogClass(String name) {
return name.startsWith("org.apache.gravitino.catalog")
&& (name.startsWith("org.apache.gravitino.catalog.hive.")
|| ...);
}
```
Classes under `org.apache.gravitino.hive.*` are not matched, so
`isSharedClass()`
returns `true` for them. The server classloader then tries to load these
classes but
cannot find them in the server classpath (they only exist under
`/gravitino/catalogs/hive/libs/`).
In practice, the server classloader falls back to `super.loadClass()`
(URLClassLoader
over execJars) and succeeds for the class itself. But the
**compiler-generated synthetic
class `$1`** (produced by `switch(enum)` statements) is loaded through the
defining
classloader of the enclosing class. If the enclosing class happens to be
loaded by the
server classloader in a specific timing window, `$1` is also requested from
the server
classloader — which cannot find it. **The JVM permanently caches this load
failure**,
causing all subsequent calls to throw `NoClassDefFoundError` until the
process restarts.
### Impact
- `NoClassDefFoundError:
org/apache/gravitino/hive/client/HiveExceptionConverter$1`
- Affects any API that triggers
`HiveExceptionConverter.toNoSuchObjectException()`
(e.g., loading a non-existent table via Hive catalog)
- Failure is **permanent** within the process lifetime — not recoverable
without restart
- Appears intermittent because it depends on classloader initialization
timing
### Fix
Add `org.apache.gravitino.hive.` as a recognized catalog class prefix in
`IsolatedClassLoader.isCatalogClass()`:
```java
private boolean isCatalogClass(String name) {
return name.startsWith("org.apache.gravitino.hive.") // ← add this
|| (name.startsWith("org.apache.gravitino.catalog")
&& (name.startsWith("org.apache.gravitino.catalog.hive.")
|| ...));
}
```
### Steps to Reproduce
1. Deploy Gravitino with Hive catalog
2. Call an API that triggers `HiveExceptionConverter` with a non-existent
resource
(e.g., `loadTable` on a table that does not exist in Hive metastore)
3. Under specific classloader initialization timing, observe
`NoClassDefFoundError:
org/apache/gravitino/hive/client/HiveExceptionConverter$1`
4. All subsequent calls to the same catalog on this process instance fail
with the
same error until restart
### Related
- Introduced by: #9460
### Error message and/or stacktrace
java.lang.NoClassDefFoundError:
org/apache/gravitino/hive/client/HiveExceptionConverter$1
at
org.apache.gravitino.hive.client.HiveExceptionConverter.toNoSuchObjectException(HiveExceptionConverter.java:233)
at
org.apache.gravitino.hive.client.HiveExceptionConverter.toGravitinoException(HiveExceptionConverter.java:77)
at
org.apache.gravitino.catalog.hive.HiveCatalogOperations.loadTable(HiveCatalogOperations.java:...)
at org.apache.gravitino.catalog.SchemaOperationDispatcher.lambda$...
at
org.apache.gravitino.utils.IsolatedClassLoader.withClassLoader(IsolatedClassLoader.java:86)
...
### How to reproduce
+ Which Gravitino version to use
0.9.x / main (after #9460 merged)
+ Steps
1. Deploy Gravitino with a Hive catalog configured
2. Under specific JVM classloader initialization timing (e.g. first
request to a catalog that has never been accessed, or after a
long-running process where classloader init races with a request),
call any API that triggers HiveExceptionConverter internally —
for example, load a table that does not exist in Hive Metastore:
GET
/api/metalakes/{metalake}/catalogs/{hive-catalog}/schemas/{schema}/tables/{non-existent-table}
3. Observe NoClassDefFoundError:
org/apache/gravitino/hive/client/HiveExceptionConverter$1
4. All subsequent requests to the same catalog on this process instance
fail with the same error — the JVM permanently caches the load
failure until the process is restarted.
+ Note
The bug is intermittent because it depends on whether
IsolatedClassLoader is initialized before or during the first request.
Once triggered, it is permanent within the process lifetime.
Introduced by #9460 which moved HiveExceptionConverter to
org.apache.gravitino.hive.client.* without updating isCatalogClass().
### Additional context
+ Which Gravitino version to use
0.9.x / main (after #9460 merged)
+ Steps
1. Deploy Gravitino with a Hive catalog configured
2. Under specific JVM classloader initialization timing (e.g. first
request to a catalog that has never been accessed, or after a
long-running process where classloader init races with a request),
call any API that triggers HiveExceptionConverter internally —
for example, load a table that does not exist in Hive Metastore:
GET
/api/metalakes/{metalake}/catalogs/{hive-catalog}/schemas/{schema}/tables/{non-existent-table}
3. Observe NoClassDefFoundError:
org/apache/gravitino/hive/client/HiveExceptionConverter$1
4. All subsequent requests to the same catalog on this process instance
fail with the same error — the JVM permanently caches the load
failure until the process is restarted.
+ Note
The bug is intermittent because it depends on whether
IsolatedClassLoader is initialized before or during the first request.
Once triggered, it is permanent within the process lifetime.
Introduced by #9460 which moved HiveExceptionConverter to
org.apache.gravitino.hive.client.* without updating isCatalogClass().
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
