[ https://issues.apache.org/jira/browse/SPARK-41535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-41535. ---------------------------------- Fix Version/s: 3.3.2 3.2.3 3.4.0 Resolution: Fixed Issue resolved by pull request 39117 [https://github.com/apache/spark/pull/39117] > InterpretedUnsafeProjection and InterpretedMutableProjection can corrupt > unsafe buffer when used with calendar interval data > ---------------------------------------------------------------------------------------------------------------------------- > > Key: SPARK-41535 > URL: https://issues.apache.org/jira/browse/SPARK-41535 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 3.3.1, 3.2.3, 3.4.0 > Reporter: Bruce Robbins > Assignee: Bruce Robbins > Priority: Major > Fix For: 3.3.2, 3.2.3, 3.4.0 > > > This returns the wrong answer: > {noformat} > set spark.sql.codegen.wholeStage=false; > set spark.sql.codegen.factoryMode=NO_CODEGEN; > select first(col1), last(col2) from values > (make_interval(0, 0, 0, 7, 0, 0, 0), make_interval(17, 0, 0, 2, 0, 0, 0)) > as data(col1, col2); > +---------------+---------------+ > |first(col1) |last(col2) | > +---------------+---------------+ > |16 years 2 days|16 years 2 days| > +---------------+---------------+ > {noformat} > In the above case, {{TungstenAggregationIterator}} uses > {{InterpretedUnsafeProjection}} to create the aggregation buffer and then > initializes all the fields to null. {{InterpretedUnsafeProjection}} > incorrectly calls {{UnsafeRowWriter#setNullAt}}, rather than > {{unsafeRowWriter#write}}, for the two calendar interval fields. As a result, > the writer never allocates memory from the variable length region for the two > decimals, and the pointers in the fixed region get left as zero. Later, when > {{InterpretedMutableProjection}} attempts to update the first field, > {{UnsafeRow#setInterval}} picks up the zero pointer and stores interval data > on top of the null-tracking bit set. The call to UnsafeRow#setInterval for > the second field also stomps the null-tracking bit set. Later updates to the > null-tracking bit set (e.g., calls to setNotNullAt) further corrupt the > interval data, turning {{interval 7 years 2 days}} into {{interval 16 years 2 > days}}. > Even if you fix the above bug to {{InterpretedUnsafeProjection}} so that the > buffer is created correctly, {{InterpretedMutableProjection}} has a similar > bug to SPARK-41395, except this time for calendar interval data: > {noformat} > set spark.sql.codegen.wholeStage=false; > set spark.sql.codegen.factoryMode=NO_CODEGEN; > select first(col1), last(col2), max(col3) from values > (null, null, 1), > (make_interval(0, 0, 0, 7, 0, 0, 0), make_interval(17, 0, 0, 2, 0, 0, 0), 3) > as data(col1, col2, col3); > +---------------+---------------+---------+ > |first(col1) |last(col2) |max(col3)| > +---------------+---------------+---------+ > |16 years 2 days|16 years 2 days|3 | > +---------------+---------------+---------+ > {noformat} > These two bugs could get exercised during codegen fallback. Take for example > this case where I forced codegen to fail for the Greatest expression: > {noformat} > spark-sql> select first(col1), last(col2), max(col3) from values > (null, null, 1), > (make_interval(0, 0, 0, 7, 0, 0, 0), make_interval(17, 0, 0, 2, 0, 0, 0), 3) > as data(col1, col2, col3); > 22/12/15 13:06:23 ERROR CodeGenerator: failed to compile: > org.codehaus.commons.compiler.CompileException: File 'generated.java', Line > 70, Column 1: ';' expected instead of 'if' > ... > 22/12/15 13:06:24 WARN MutableProjection: Expr codegen error and falling back > to interpreter mode > java.util.concurrent.ExecutionException: > org.codehaus.commons.compiler.CompileException: File 'generated.java', Line > 78, Column 1: failed to compile: > org.codehaus.commons.compiler.CompileException: File 'generated.java', Line > 78, Column 1: ';' expected instead of 'boolean' > ... > 16 years 2 days 16 years 2 days 3 > Time taken: 5.852 seconds, Fetched 1 row(s) > spark-sql> > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org