[GitHub] spark pull request #19083: [SPARK-21871][SQL] Check actual bytecode size whe...
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/19083#discussion_r142059351 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala --- @@ -432,25 +432,31 @@ class DataFrameAggregateSuite extends QueryTest with SharedSQLContext { } test("zero moments") { -val input = Seq((1, 2)).toDF("a", "b") -checkAnswer( - input.agg(stddev('a), stddev_samp('a), stddev_pop('a), variance('a), -var_samp('a), var_pop('a), skewness('a), kurtosis('a)), - Row(Double.NaN, Double.NaN, 0.0, Double.NaN, Double.NaN, 0.0, -Double.NaN, Double.NaN)) +// In SPARK-21871, we added code to check the actual bytecode size of gen'd methods. If the size +// goes over `hugeMethodLimit`, Spark fails to compile the methods and the execution also fails +// in a test mode. So, we explicitly made this threshold higher here. +// This workaround can be removed if this issue fixed. +withSQLConf(SQLConf.CODEGEN_HUGE_METHOD_LIMIT.key -> "16000") { --- End diff -- Do you think is it ok to silently fall back to non-codegen cases in tests? I though that the too-long function cases also should explicitly throws exceptions in tests by checking `!Utils.isTesting && sqlContext.conf.codegenFallback`. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19083: [SPARK-21871][SQL] Check actual bytecode size whe...
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/19083#discussion_r142059729 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala --- @@ -432,25 +432,31 @@ class DataFrameAggregateSuite extends QueryTest with SharedSQLContext { } test("zero moments") { -val input = Seq((1, 2)).toDF("a", "b") -checkAnswer( - input.agg(stddev('a), stddev_samp('a), stddev_pop('a), variance('a), -var_samp('a), var_pop('a), skewness('a), kurtosis('a)), - Row(Double.NaN, Double.NaN, 0.0, Double.NaN, Double.NaN, 0.0, -Double.NaN, Double.NaN)) +// In SPARK-21871, we added code to check the actual bytecode size of gen'd methods. If the size +// goes over `hugeMethodLimit`, Spark fails to compile the methods and the execution also fails +// in a test mode. So, we explicitly made this threshold higher here. +// This workaround can be removed if this issue fixed. +withSQLConf(SQLConf.CODEGEN_HUGE_METHOD_LIMIT.key -> "16000") { --- End diff -- sorry, my bad. I'll do so. Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org