[GitHub] [spark] LuciferYang commented on pull request #32536: [SPARK-35398][SQL] Simplify the way to get classes from ClassBodyEvaluator in `CodeGenerator.updateAndGetCompilationStats` method
LuciferYang commented on pull request #32536: URL: https://github.com/apache/spark/pull/32536#issuecomment-842148206 New test cases will fail when running with other tests, let me check this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #32536: [SPARK-35398][SQL] Simplify the way to get classes from ClassBodyEvaluator in `CodeGenerator.updateAndGetCompilationStats` method
LuciferYang commented on pull request #32536: URL: https://github.com/apache/spark/pull/32536#issuecomment-842148206 New test cases will fail when running with other tests, let me check this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #32536: [SPARK-35398][SQL] Simplify the way to get classes from ClassBodyEvaluator in `CodeGenerator.updateAndGetCompilationStats` method
LuciferYang commented on pull request #32536: URL: https://github.com/apache/spark/pull/32536#issuecomment-842973792  @maropu `Reservoir` in `com.codahale.metrics.Histogram` is `ExponentiallyDecayingReservoir`, it's values are statistical samples, not complete sets, when the size of values exceeds the threshold(1028), the new values may not be inserted into values. I printed size of `METRIC_GENERATED_CLASS_BYTECODE_SIZE` and `METRIC_GENERATED_METHOD_BYTECODE_SIZE` in `CodeGenerationSuite.beforeAll` method, their results are all 1028. So this case will fail when run ``` mvn test -pl sql/catalyst ``` and this case will success when run ``` mvn test -pl sql/catalyst -DwildcardSuites=org.apache.spark.sql.catalyst.expressions.CodeGenerationSuite ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #32536: [SPARK-35398][SQL] Simplify the way to get classes from ClassBodyEvaluator in `CodeGenerator.updateAndGetCompilationStats` method
LuciferYang commented on pull request #32536:
URL: https://github.com/apache/spark/pull/32536#issuecomment-842979946
@maropu for the above problem, can we add a case similar to the following to
compare the result of `reflection api and using ClassBodyEvaluator.getBytecodes
api` directly ?
```
val codeBody = s"""
public java.lang.Object generate(Object[] references) {
return new TestMetricCode(references);
}
class TestMetricCode {
public TestMetricCode(Object[] references) {
}
public long sumOfSquares(long left, long right) {
return left * left + right * right;
}
}
"""
val evaluator = CodeGenerator.createClassBodyEvaluator()
evaluator.cook("generated.java", codeBody)
import scala.collection.JavaConverters._
val bytecodesWithApi = evaluator.getBytecodes.asScala
val bytecodesWithReflectionApi = {
val scField = classOf[ClassBodyEvaluator].getDeclaredField("sc")
scField.setAccessible(true)
val compiler = scField.get(evaluator).asInstanceOf[SimpleCompiler]
val loader = compiler.getClassLoader.asInstanceOf[ByteArrayClassLoader]
val classesField = loader.getClass.getDeclaredField("classes")
classesField.setAccessible(true)
classesField.get(loader).asInstanceOf[java.util.Map[String,
Array[Byte]]].asScala
}
assert(bytecodesWithApi == bytecodesWithReflectionApi)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #32536: [SPARK-35398][SQL] Simplify the way to get classes from ClassBodyEvaluator in `CodeGenerator.updateAndGetCompilationStats` method
LuciferYang commented on pull request #32536: URL: https://github.com/apache/spark/pull/32536#issuecomment-843759422 thx all ~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #32536: [SPARK-35398][SQL] Simplify the way to get classes from ClassBodyEvaluator in `CodeGenerator.updateAndGetCompilationStats` method
LuciferYang commented on pull request #32536: URL: https://github.com/apache/spark/pull/32536#issuecomment-840441694 Waiting for CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #32536: [SPARK-35398][SQL] Simplify the way to get classes from ClassBodyEvaluator in `CodeGenerator.updateAndGetCompilationStats` method
LuciferYang commented on pull request #32536: URL: https://github.com/apache/spark/pull/32536#issuecomment-840441827 cc @maropu @srowen -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
