GitHub user lw-lin opened a pull request:
https://github.com/apache/spark/pull/15461
[SPARK-16845][SQL] `GeneratedClass$SpecificOrdering` grows beyond 64 KB
## What changes were proposed in this pull request?
Prior to this patch, we'll generate `compare(...)` for
`GeneratedClass$SpecificOrdering` like below, leading to Janino exceptions
saying the code grows beyond 64 KB.
```scala
/* 005 */ class SpecificOrdering extends
org.apache.spark.sql.catalyst.expressions.codegen.BaseOrdering {
/* ..... */ ...
/* 10969 */ private int compare(InternalRow a, InternalRow b) {
/* 10970 */ InternalRow i = null; // Holds current row being evaluated.
/* 10971 */
/* 1.... */ code for comparing field0
/* 1.... */ code for comparing field1
/* 1.... */ ...
/* 1.... */ code for comparing field449
/* 15012 */
/* 15013 */ return 0;
/* 15014 */ }
/* 15015 */ }
```
This patch would break `compare(...)` into smaller `compare_xxx(...)`
methods when necessary; then we'll get generated `compare(...)` like:
```scala
/* 001 */ public SpecificOrdering generate(Object[] references) {
/* 002 */ return new SpecificOrdering(references);
/* 003 */ }
/* 004 */
/* 005 */ class SpecificOrdering extends
org.apache.spark.sql.catalyst.expressions.codegen.BaseOrdering {
/* 006 */
/* 007 */ ...
/* 10968 */
/* 10969 */ private int compare_0(InternalRow a, InternalRow b) {
/* 10970 */ InternalRow i = null; // Holds current row being evaluated.
/* 10971 */
/* 10972 */ i = a;
/* 10973 */ boolean isNullA;
/* 10974 */ UTF8String primitiveA;
/* 10975 */ {
/* 10976 */
/* 10977 */ Object obj = ((Expression) references[0]).eval(null);
/* 10978 */ UTF8String value = (UTF8String) obj;
/* 10979 */ isNullA = false;
/* 10980 */ primitiveA = value;
/* 10981 */ }
/* 10982 */ i = b;
/* 10983 */ boolean isNullB;
/* 10984 */ UTF8String primitiveB;
/* 10985 */ {
/* 10986 */
/* 10987 */ Object obj = ((Expression) references[0]).eval(null);
/* 10988 */ UTF8String value = (UTF8String) obj;
/* 10989 */ isNullB = false;
/* 10990 */ primitiveB = value;
/* 10991 */ }
/* 10992 */ if (isNullA && isNullB) {
/* 10993 */ // Nothing
/* 10994 */ } else if (isNullA) {
/* 10995 */ return -1;
/* 10996 */ } else if (isNullB) {
/* 10997 */ return 1;
/* 10998 */ } else {
/* 10999 */ int comp = primitiveA.compare(primitiveB);
/* 11000 */ if (comp != 0) {
/* 11001 */ return comp;
/* 11002 */ }
/* 11003 */ }
/* 11004 */
/* 11005 */ ...
/* 12258 */
/* 12259 */ i = a;
/* 12260 */ boolean isNullA39;
/* 12261 */ UTF8String primitiveA39;
/* 12262 */ {
/* 12263 */
/* 12264 */ Object obj39 = ((Expression) references[39]).eval(null);
/* 12265 */ UTF8String value39 = (UTF8String) obj39;
/* 12266 */ isNullA39 = false;
/* 12267 */ primitiveA39 = value39;
/* 12268 */ }
/* 12269 */ i = b;
/* 12270 */ boolean isNullB39;
/* 12271 */ UTF8String primitiveB39;
/* 12272 */ {
/* 12273 */
/* 12274 */ Object obj39 = ((Expression) references[39]).eval(null);
/* 12275 */ UTF8String value39 = (UTF8String) obj39;
/* 12276 */ isNullB39 = false;
/* 12277 */ primitiveB39 = value39;
/* 12278 */ }
/* 12279 */ if (isNullA39 && isNullB39) {
/* 12280 */ // Nothing
/* 12281 */ } else if (isNullA39) {
/* 12282 */ return -1;
/* 12283 */ } else if (isNullB39) {
/* 12284 */ return 1;
/* 12285 */ } else {
/* 12286 */ int comp = primitiveA39.compare(primitiveB39);
/* 12287 */ if (comp != 0) {
/* 12288 */ return comp;
/* 12289 */ }
/* 12290 */ }
/* 12291 */
/* 12292 */ return 0;
/* 12293 */ }
/* 12294 */
/* 12295 */ ...
/* 14949 */
/* 14950 */ public int compare(InternalRow a, InternalRow b) {
/* 14951 */ int result;
/* 14952 */
/* 14953 */ result = compare_0(a, b);
/* 14954 */ if (result != 0) {
/* 14955 */ return result;
/* 14956 */ }
/* 14957 */
/* 14958 */ result = compare_1(a, b);
/* 14959 */ if (result != 0) {
/* 14960 */ return result;
/* 14961 */ }
/* 14962 */
/* 14963 */ ...
/* 15007 */
/* 15008 */ result = compare_11(a, b);
/* 15009 */ if (result != 0) {
/* 15010 */ return result;
/* 15011 */ }
/* 15012 */
/* 15013 */ return 0;
/* 15014 */ }
/* 15015 */ }
```
## How was this patch tested?
- A new added test case which
- would fail prior to this patch
- would pass with this patch
- ordering correctness should already be covered by existing tests like
those in `OrderingSuite`
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/lw-lin/spark spec-ordering-64k
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15461.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15461
----
commit 124172a707b3053cfc407adcf1cb495a5f16f182
Author: Liwei Lin <lwl...@gmail.com>
Date: 2016-10-13T08:16:29Z
[SPARK-16845][SQL] `GeneratedClass$SpecificOrdering` grows beyond 64 KB
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
