ndimiduk commented on code in PR #5488:
URL: https://github.com/apache/hbase/pull/5488#discussion_r1384678667
##########
hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java:
##########
@@ -1552,13 +1571,23 @@ public static ComparatorProtos.Comparator
toComparator(ByteArrayComparable compa
public static ByteArrayComparable toComparator(ComparatorProtos.Comparator
proto)
throws IOException {
String type = proto.getName();
- String funcName = "parseFrom";
byte[] value = proto.getSerializedComparator().toByteArray();
+
try {
+ ByteArrayComparable result = COMPARATORS.getAndCallByName(type, value);
+ if (result != null) {
+ return result;
+ }
+
+ if (!ALLOW_FAST_REFLECTION_FALLTHROUGH) {
+ throw new IllegalStateException("Failed to deserialize comparator " +
type
+ + " because fast reflection returned null and fallthrough is
disabled");
+ }
Review Comment:
Should we add an else clause that ticks up a metric? Or maybe an
emitted-once-per-`type` WARN that notifies the operator that we've fallen back
to the slower path? I don't yet understand how often this might occur. Same for
Filters.
##########
hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java:
##########
@@ -304,6 +305,24 @@ public static boolean isClassLoaderLoaded() {
return classLoaderLoaded;
}
+ private static final String PARSE_FROM = "parseFrom";
+
+ // We don't bother using the dynamic CLASS_LOADER above, because currently
we can't support
+ // optimizing dynamically loaded classes. We can do it once we build for
java9+, see the todo
+ // in ReflectedFunctionCache
+ private static final ReflectedFunctionCache<byte[], Filter> FILTERS =
ReflectedFunctionCache
+ .create(ProtobufUtil.class.getClassLoader(), Filter.class, byte[].class,
PARSE_FROM);
+ private static final ReflectedFunctionCache<byte[], ByteArrayComparable>
COMPARATORS =
+ ReflectedFunctionCache.create(ProtobufUtil.class.getClassLoader(),
ByteArrayComparable.class,
+ byte[].class, PARSE_FROM);
+
+ private static volatile boolean ALLOW_FAST_REFLECTION_FALLTHROUGH = true;
Review Comment:
Is there a reason to gate this feature behind a configuration point that's
exposed to the operator?
##########
hbase-common/src/main/java/org/apache/hadoop/hbase/util/ReflectedFunctionCache.java:
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.util;
+
+import java.io.IOException;
+import java.lang.reflect.Modifier;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hbase.thirdparty.com.google.common.reflect.ClassPath;
+
+/**
+ * Cache to hold resolved Functions generated through reflection. These can be
costly to create, but
+ * then are much faster than typical Method.invoke calls when executing. Upon
construction, finds
+ * all subclasses in the same package of the passed baseClass. For each found
class, creates a
+ * lambda using
+ * {@link ReflectionUtils#getOneArgStaticMethodAsFunction(Class, String,
Class, Class)}. These are
+ * added to a hashmap for fast lookup by name later.
+ * @param <I> the input argument type for the resolved functions
+ * @param <R> the return type for the resolved functions
+ */
+@InterfaceAudience.Private
+final public class ReflectedFunctionCache<I, R> {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(ReflectedFunctionCache.class);
+
+ private final Map<String, Function<I, ? extends R>> lambdasByClass;
+
+ private ReflectedFunctionCache(Map<String, Function<I, ? extends R>>
lambdasByClass) {
+ this.lambdasByClass = lambdasByClass;
+ }
+
+ /**
+ * Create a cache of reflected functions using the provided classloader and
baseClass. Will find
+ * all subclasses of the provided baseClass (in the same package), and then
foreach look for a
+ * static one-arg method with the methodName and argClass. The expectation
is that the method
+ * returns a value whose class extends the baseClass. This was primarily
designed for use by our
+ * Filter and Comparator parseFrom methods.
+ */
+ public static <I, R> ReflectedFunctionCache<I, R> create(ClassLoader
classLoader,
+ Class<R> baseClass, Class<I> argClass, String methodName) {
+ Map<String, Function<I, ? extends R>> lambdasByClass = new HashMap<>();
+ Set<? extends Class<? extends R>> classes =
getSubclassesInPackage(classLoader, baseClass);
+ for (Class<? extends R> clazz : classes) {
+ Function<I, ? extends R> func = createFunction(clazz, methodName,
argClass, clazz);
+ if (func != null) {
+ lambdasByClass.put(clazz.getName(), func);
+ }
+ }
+ return new ReflectedFunctionCache<>(lambdasByClass);
+ }
+
+ /**
+ * Get and execute the Function for the given className, passing the
argument to the function and
+ * returning the result.
+ * @param className the full name of the class to lookup
+ * @param argument the argument to pass to the function, if found.
+ * @return null if a function is not found for classname, otherwise the
result of the function.
+ */
+ public R getAndCallByName(String className, I argument) {
+ Function<I, ? extends R> lambda = lambdasByClass.get(className);
+
+ // todo: if we ever make java9+ our lowest supported jdk version, we can
+ // handle generating these for newly loaded classes from our
DynamicClassLoader using
+ // MethodHandles.privateLookupIn(). For now this is not possible, because
we can't easily
+ // create a privileged lookup in a non-default ClassLoader.
+ if (lambda == null) {
+ return null;
+ }
+
+ return lambda.apply(argument);
+ }
+
+ private static <R> Set<Class<? extends R>>
getSubclassesInPackage(ClassLoader classLoader,
+ Class<R> baseClass) {
+ try {
+ return ClassPath.from(classLoader).getAllClasses().stream()
Review Comment:
Reading the code for `ClassPath#getAllClasses`, it seems that it will build
an exhaustive set of all classes on the classpath that are loadable. It doesn't
actually load the classes. My understanding is that `Stream` operations are
lazy, so the only classes loaded should be those that materialize in the final
`collect`.
##########
hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java:
##########
@@ -304,6 +305,24 @@ public static boolean isClassLoaderLoaded() {
return classLoaderLoaded;
}
+ private static final String PARSE_FROM = "parseFrom";
+
+ // We don't bother using the dynamic CLASS_LOADER above, because currently
we can't support
+ // optimizing dynamically loaded classes. We can do it once we build for
java9+, see the todo
+ // in ReflectedFunctionCache
+ private static final ReflectedFunctionCache<byte[], Filter> FILTERS =
ReflectedFunctionCache
+ .create(ProtobufUtil.class.getClassLoader(), Filter.class, byte[].class,
PARSE_FROM);
Review Comment:
Is the content of the directory specified in `hbase.dynamic.jars.dir`
included in the classpath that is under the domain of this classloader? I think
that if there are user-provided Filter classes in the path, we should load
them. I guess that we cannot assume that they will be in the `o.a.h.h.filter`
package, so we'd have to relax our class selection criteria.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
