GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/15480
[SPARK-16845][SQL] `GeneratedClass$SpecificOrdering` grows beyond 64 KB ## What changes were proposed in this pull request? Prior to this patch, we'll generate `compare(...)` for `GeneratedClass$SpecificOrdering` like below, leading to Janino exceptions saying the code grows beyond 64 KB. ```scala /* 005 */ class SpecificOrdering extends org.apache.spark.sql.catalyst.expressions.codegen.BaseOrdering { /* ..... */ ... /* 10969 */ private int compare(InternalRow a, InternalRow b) { /* 10970 */ InternalRow i = null; // Holds current row being evaluated. /* 10971 */ /* 1.... */ code for comparing field0 /* 1.... */ code for comparing field1 /* 1.... */ ... /* 1.... */ code for comparing field449 /* 15012 */ /* 15013 */ return 0; /* 15014 */ } /* 15015 */ } ``` This patch would break `compare(...)` into smaller `compare_xxx(...)` methods when necessary; then we'll get generated `compare(...)` like: ```scala /* 001 */ public SpecificOrdering generate(Object[] references) { /* 002 */ return new SpecificOrdering(references); /* 003 */ } /* 004 */ /* 005 */ class SpecificOrdering extends org.apache.spark.sql.catalyst.expressions.codegen.BaseOrdering { /* 006 */ /* 007 */ ... /* 10968 */ /* 10969 */ private int compare_0(InternalRow a, InternalRow b) { /* 10970 */ InternalRow i = null; // Holds current row being evaluated. /* 10971 */ /* 10972 */ i = a; /* 10973 */ boolean isNullA; /* 10974 */ UTF8String primitiveA; /* 10975 */ { /* 10976 */ /* 10977 */ Object obj = ((Expression) references[0]).eval(null); /* 10978 */ UTF8String value = (UTF8String) obj; /* 10979 */ isNullA = false; /* 10980 */ primitiveA = value; /* 10981 */ } /* 10982 */ i = b; /* 10983 */ boolean isNullB; /* 10984 */ UTF8String primitiveB; /* 10985 */ { /* 10986 */ /* 10987 */ Object obj = ((Expression) references[0]).eval(null); /* 10988 */ UTF8String value = (UTF8String) obj; /* 10989 */ isNullB = false; /* 10990 */ primitiveB = value; /* 10991 */ } /* 10992 */ if (isNullA && isNullB) { /* 10993 */ // Nothing /* 10994 */ } else if (isNullA) { /* 10995 */ return -1; /* 10996 */ } else if (isNullB) { /* 10997 */ return 1; /* 10998 */ } else { /* 10999 */ int comp = primitiveA.compare(primitiveB); /* 11000 */ if (comp != 0) { /* 11001 */ return comp; /* 11002 */ } /* 11003 */ } /* 11004 */ /* 11005 */ ... /* 12258 */ /* 12259 */ i = a; /* 12260 */ boolean isNullA39; /* 12261 */ UTF8String primitiveA39; /* 12262 */ { /* 12263 */ /* 12264 */ Object obj39 = ((Expression) references[39]).eval(null); /* 12265 */ UTF8String value39 = (UTF8String) obj39; /* 12266 */ isNullA39 = false; /* 12267 */ primitiveA39 = value39; /* 12268 */ } /* 12269 */ i = b; /* 12270 */ boolean isNullB39; /* 12271 */ UTF8String primitiveB39; /* 12272 */ { /* 12273 */ /* 12274 */ Object obj39 = ((Expression) references[39]).eval(null); /* 12275 */ UTF8String value39 = (UTF8String) obj39; /* 12276 */ isNullB39 = false; /* 12277 */ primitiveB39 = value39; /* 12278 */ } /* 12279 */ if (isNullA39 && isNullB39) { /* 12280 */ // Nothing /* 12281 */ } else if (isNullA39) { /* 12282 */ return -1; /* 12283 */ } else if (isNullB39) { /* 12284 */ return 1; /* 12285 */ } else { /* 12286 */ int comp = primitiveA39.compare(primitiveB39); /* 12287 */ if (comp != 0) { /* 12288 */ return comp; /* 12289 */ } /* 12290 */ } /* 12291 */ /* 12292 */ return 0; /* 12293 */ } /* 12294 */ /* 12295 */ ... /* 14949 */ /* 14950 */ public int compare(InternalRow a, InternalRow b) { /* 14951 */ int result; /* 14952 */ /* 14953 */ result = compare_0(a, b); /* 14954 */ if (result != 0) { /* 14955 */ return result; /* 14956 */ } /* 14957 */ /* 14958 */ result = compare_1(a, b); /* 14959 */ if (result != 0) { /* 14960 */ return result; /* 14961 */ } /* 14962 */ /* 14963 */ ... /* 15007 */ /* 15008 */ result = compare_11(a, b); /* 15009 */ if (result != 0) { /* 15010 */ return result; /* 15011 */ } /* 15012 */ /* 15013 */ return 0; /* 15014 */ } /* 15015 */ } ``` ## How was this patch tested? - a new added test case which - would fail prior to this patch - would pass with this patch - ordering correctness should already be covered by existing tests like those in `OrderingSuite` You can merge this pull request into a Git repository by running: $ git pull https://github.com/lw-lin/spark spec-ordering-64k- Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/15480.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #15480 ---- commit ecc6720ed96c750333173fc53a81e850abcd9013 Author: Liwei Lin <lwl...@gmail.com> Date: 2016-10-13T08:16:29Z [SPARK-16845][SQL] `GeneratedClass$SpecificOrdering` grows beyond 64 KB ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org