[
https://issues.apache.org/jira/browse/IMPALA-11260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17555553#comment-17555553
]
Quanlong Huang commented on IMPALA-11260:
-----------------------------------------
I try replacing Ehcache.sizeof with JOL from openJDK. JOL doesn't have the
underestimating issue. Looking into its code, when reflections fail, it will
fall back to use Unsafe to get the private fields:
{code:java}
/**
* Get the object field value.
* @param o object to get field value from
* @param f field descriptor
* @return value, maybe a boxed primitive
*/
public static Object value(Object o, Field f) {
// Try 1. Get with Reflection:
try {
return f.get(o);
} catch (Exception e) {
// fall-through
}
// Try 2. Get with Reflection and setAccessible:
try {
f.setAccessible(true);
return f.get(o);
} catch (Exception e) {
// fall-through
}
// Try 3. Get with VM hack
VirtualMachine vm = VM.current();
long off = vm.fieldOffset(f);
Class<?> t = f.getType();
if (t.isPrimitive()) {
if (t == boolean.class) {
return vm.getBoolean(o, off);
} else
if (t == byte.class) {
return vm.getByte(o, off);
} else
if (t == char.class) {
return vm.getChar(o, off);
} else
if (t == short.class) {
return vm.getShort(o, off);
} else
if (t == int.class) {
return vm.getInt(o, off);
} else
if (t == float.class) {
return vm.getFloat(o, off);
} else
if (t == long.class) {
return vm.getLong(o, off);
} else
if (t == double.class) {
return vm.getDouble(o, off);
} else {
throw new IllegalStateException("Unhandled primitive: " + t);
}
} else {
return vm.getObject(o, off);
}
}
{code}
https://github.com/openjdk/jol/blob/5ec187a6d38443604221a63abec39c545d4d1674/jol-core/src/main/java/org/openjdk/jol/util/ObjectUtils.java#L134
However, JOL uses GPL-2.0 license. It seems we can't use it directly.
> Catalog cache item sizes of CatalogdMetaProvider are underestimated on Java9+
> -----------------------------------------------------------------------------
>
> Key: IMPALA-11260
> URL: https://issues.apache.org/jira/browse/IMPALA-11260
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 3.1.0, Impala 3.2.0, Impala 4.0.0, Impala 3.3.0,
> Impala 3.4.0, Impala 3.4.1, Impala 4.1.0
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
>
> When running local catalog mode on Java11, the Ehcache sizeof lib complains
> that cache sizes may be underestimated:
> {code:java}
> W0421 20:50:44.238312 9819 ObjectGraphWalker.java:251]
> 744e548159a57cb5:879ee74c00000000] The JVM is preventing Ehcache from
> accessing the subgraph beneath 'final jdk.internal.loader.URLClassPath
> jdk.internal.loader.ClassLoaders$AppClassLoader.ucp' - cache sizes may be
> underestimated as a result
> Java exception follows:
> java.lang.reflect.InaccessibleObjectException: Unable to make field final
> jdk.internal.loader.URLClassPath
> jdk.internal.loader.ClassLoaders$AppClassLoader.ucp accessible: module
> java.base does not "opens jdk.internal.loader" to unnamed module @6ba7383d
> at
> java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:340)
> at
> java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:280)
> at
> java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:176)
> at java.base/java.lang.reflect.Field.setAccessible(Field.java:170)
> at
> org.ehcache.sizeof.ObjectGraphWalker.getAllFields(ObjectGraphWalker.java:245)
> at
> org.ehcache.sizeof.ObjectGraphWalker.getFilteredFields(ObjectGraphWalker.java:204)
> at
> org.ehcache.sizeof.ObjectGraphWalker.walk(ObjectGraphWalker.java:159)
> at org.ehcache.sizeof.SizeOf.deepSizeOf(SizeOf.java:74)
> at
> org.apache.impala.catalog.local.CatalogdMetaProvider$SizeOfWeigher.weigh(CatalogdMetaProvider.java:1999)
> at
> com.google.common.cache.LocalCache$Segment.setValue(LocalCache.java:2010)
> at
> com.google.common.cache.LocalCache$Segment.replace(LocalCache.java:2956)
> at com.google.common.cache.LocalCache.replace(LocalCache.java:4258)
> at
> org.apache.impala.catalog.local.CatalogdMetaProvider.loadWithCaching(CatalogdMetaProvider.java:540)
> at
> org.apache.impala.catalog.local.CatalogdMetaProvider.loadIcebergApiTable(CatalogdMetaProvider.java:1056)
> at
> org.apache.impala.catalog.local.LocalIcebergTable.loadIcebergTableViaMetaProvider(LocalIcebergTable.java:87)
> at
> org.apache.impala.catalog.local.LocalTable.load(LocalTable.java:107)
> at org.apache.impala.catalog.local.LocalDb.getTable(LocalDb.java:127)
> at
> org.apache.impala.analysis.StmtMetadataLoader.getMissingTables(StmtMetadataLoader.java:310)
> at
> org.apache.impala.analysis.StmtMetadataLoader.loadTables(StmtMetadataLoader.java:165)
> at
> org.apache.impala.analysis.StmtMetadataLoader.loadTables(StmtMetadataLoader.java:141)
> at
> org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:2014)
> at
> org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:1926)
> at
> org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1750)
> at
> org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:164){code}
> Similar errors on other classes:
> {code}
> The JVM is preventing Ehcache from accessing the subgraph beneath 'final
> jdk.internal.loader.AbstractClassLoaderValue
> jdk.internal.loader.AbstractClassLoaderValue$Sub.this$0' - cache sizes may be
> underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'final
> jdk.internal.loader.URLClassPath
> jdk.internal.loader.ClassLoaders$AppClassLoader.ucp' - cache sizes may be
> underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.lang.Object jdk.internal.loader.AbstractClassLoaderValue$Sub.key'
> - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.lang.String java.lang.module.Configuration.targetPlatform' - cache
> sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.lang.String java.lang.module.ModuleDescriptor.mainClass' - cache
> sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.lang.String java.lang.module.ModuleDescriptor.name' - cache sizes
> may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.lang.String java.lang.module.ModuleDescriptor.rawVersionString' -
> cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.lang.module.ModuleDescriptor$Version
> java.lang.module.ModuleDescriptor.version' - cache sizes may be
> underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.util.List java.lang.module.Configuration.parents' - cache sizes
> may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.util.Map java.lang.module.Configuration.graph' - cache sizes may
> be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.util.Map java.lang.module.Configuration.nameToModule' - cache
> sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.util.Map jdk.internal.loader.BuiltinClassLoader.moduleToReader' -
> cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.util.Map jdk.internal.loader.BuiltinClassLoader.nameToModule' -
> cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.util.Map jdk.internal.module.ServicesCatalog.map' - cache sizes
> may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.util.Set java.lang.module.Configuration.modules' - cache sizes may
> be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.util.Set java.lang.module.ModuleDescriptor.exports' - cache sizes
> may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.util.Set java.lang.module.ModuleDescriptor.modifiers' - cache
> sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.util.Set java.lang.module.ModuleDescriptor.opens' - cache sizes
> may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.util.Set java.lang.module.ModuleDescriptor.packages' - cache sizes
> may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.util.Set java.lang.module.ModuleDescriptor.provides' - cache sizes
> may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.util.Set java.lang.module.ModuleDescriptor.requires' - cache sizes
> may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.util.Set java.lang.module.ModuleDescriptor.uses' - cache sizes may
> be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final jdk.internal.loader.BuiltinClassLoader
> jdk.internal.loader.BuiltinClassLoader.parent' - cache sizes may be
> underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final jdk.internal.loader.URLClassPath
> jdk.internal.loader.BuiltinClassLoader.ucp' - cache sizes may be
> underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> jdk.internal.reflect.ConstructorAccessorImpl
> jdk.internal.reflect.DelegatingConstructorAccessorImpl.delegate' - cache
> sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> volatile java.lang.ref.SoftReference
> jdk.internal.loader.BuiltinClassLoader.resourceCache' - cache sizes may be
> underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> volatile java.util.List java.lang.module.Configuration.allConfigurations' -
> cache sizes may be underestimated as a result
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
