[
https://issues.apache.org/jira/browse/SPARK-39093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruce Robbins updated SPARK-39093:
----------------------------------
Description:
Assume this data:
{noformat}
create or replace temp view v1 as
select * FROM VALUES
(interval '10' months, interval '10' day, 2)
as v1(period, duration, num);
cache table v1;
{noformat}
These two queries work:
{noformat}
spark-sql> select period/num from v1;
0-5
Time taken: 0.143 seconds, Fetched 1 row(s)
{noformat}
{noformat}
spark-sql> select duration/num from v1;
5 00:00:00.000000000
Time taken: 0.094 seconds, Fetched 1 row(s)
{noformat}
However, these two queries get a codegen compilation error:
{noformat}
spark-sql> select period/(num + 3) from v1;
22/05/03 08:56:37 ERROR CodeGenerator: failed to compile:
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 40,
Column 44: Expression "project_value_2" is not an rvalue
...
22/05/03 08:56:37 WARN UnsafeProjection: Expr codegen error and falling back to
interpreter mode
...
0-2
Time taken: 0.149 seconds, Fetched 1 row(s)
{noformat}
{noformat}
spark-sql> select duration/(num + 3) from v1;
22/05/03 08:57:29 ERROR CodeGenerator: failed to compile:
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 40,
Column 54: Expression "project_value_2" is not an rvalue
...
22/05/03 08:57:29 WARN UnsafeProjection: Expr codegen error and falling back to
interpreter mode
...
2 00:00:00.000000000
Time taken: 0.089 seconds, Fetched 1 row(s)
{noformat}
Even the first two queries will get a compilation error if you turn off
whole-stage codegen:
{noformat}
spark-sql> set spark.sql.codegen.wholeStage=false;
spark.sql.codegen.wholeStage false
Time taken: 0.055 seconds, Fetched 1 row(s)
spark-sql> select period/num from v1;
22/05/03 09:16:42 ERROR CodeGenerator: failed to compile:
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 37,
Column 5: Expression "value_1" is not an rvalue
....
0-5
Time taken: 0.175 seconds, Fetched 1 row(s)
spark-sql> select duration/num from v1;
22/05/03 09:17:41 ERROR CodeGenerator: failed to compile:
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 37,
Column 5: Expression "value_1" is not an rvalue
...
5 00:00:00.000000000
Time taken: 0.104 seconds, Fetched 1 row(s)
{noformat}
Note that in the error cases, the queries still return a result because Spark
falls back on interpreting the divide expression (so I marked this as "minor").
was:
Assume this data:
{noformat}
create or replace temp view v1 as
select * FROM VALUES
(interval '10' months, interval '10' day, 2)
as v1(period, duration, num);
cache table v1;
{noformat}
These two queries work:
{noformat}
spark-sql> select period/num from v1;
0-5
Time taken: 0.143 seconds, Fetched 1 row(s)
{noformat}
{noformat}
spark-sql> select duration/num from v1;
5 00:00:00.000000000
Time taken: 0.094 seconds, Fetched 1 row(s)
{noformat}
However, these two queries get a codegen compilation error:
{noformat}
spark-sql> select period/(num + 3) from v1;
22/05/03 08:56:37 ERROR CodeGenerator: failed to compile:
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 40,
Column 44: Expression "project_value_2" is not an rvalue
...
22/05/03 08:56:37 WARN UnsafeProjection: Expr codegen error and falling back to
interpreter mode
...
0-2
Time taken: 0.149 seconds, Fetched 1 row(s)
{noformat}
{noformat}
spark-sql> select duration/(num + 3) from v1;
22/05/03 08:57:29 ERROR CodeGenerator: failed to compile:
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 40,
Column 54: Expression "project_value_2" is not an rvalue
...
22/05/03 08:57:29 WARN UnsafeProjection: Expr codegen error and falling back to
interpreter mode
...
2 00:00:00.000000000
Time taken: 0.089 seconds, Fetched 1 row(s)
{noformat}
Note that both queries still return a result because Spark falls back on
interpreting the divide expression (so I marked this as "minor").
> Dividing interval by integral can result in codegen compilation error
> ---------------------------------------------------------------------
>
> Key: SPARK-39093
> URL: https://issues.apache.org/jira/browse/SPARK-39093
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.4.0
> Reporter: Bruce Robbins
> Priority: Minor
>
> Assume this data:
> {noformat}
> create or replace temp view v1 as
> select * FROM VALUES
> (interval '10' months, interval '10' day, 2)
> as v1(period, duration, num);
> cache table v1;
> {noformat}
> These two queries work:
> {noformat}
> spark-sql> select period/num from v1;
> 0-5
> Time taken: 0.143 seconds, Fetched 1 row(s)
> {noformat}
> {noformat}
> spark-sql> select duration/num from v1;
> 5 00:00:00.000000000
> Time taken: 0.094 seconds, Fetched 1 row(s)
> {noformat}
> However, these two queries get a codegen compilation error:
> {noformat}
> spark-sql> select period/(num + 3) from v1;
> 22/05/03 08:56:37 ERROR CodeGenerator: failed to compile:
> org.codehaus.commons.compiler.CompileException: File 'generated.java', Line
> 40, Column 44: Expression "project_value_2" is not an rvalue
> ...
> 22/05/03 08:56:37 WARN UnsafeProjection: Expr codegen error and falling back
> to interpreter mode
> ...
> 0-2
> Time taken: 0.149 seconds, Fetched 1 row(s)
> {noformat}
> {noformat}
> spark-sql> select duration/(num + 3) from v1;
> 22/05/03 08:57:29 ERROR CodeGenerator: failed to compile:
> org.codehaus.commons.compiler.CompileException: File 'generated.java', Line
> 40, Column 54: Expression "project_value_2" is not an rvalue
> ...
> 22/05/03 08:57:29 WARN UnsafeProjection: Expr codegen error and falling back
> to interpreter mode
> ...
> 2 00:00:00.000000000
> Time taken: 0.089 seconds, Fetched 1 row(s)
> {noformat}
> Even the first two queries will get a compilation error if you turn off
> whole-stage codegen:
> {noformat}
> spark-sql> set spark.sql.codegen.wholeStage=false;
> spark.sql.codegen.wholeStage false
> Time taken: 0.055 seconds, Fetched 1 row(s)
> spark-sql> select period/num from v1;
> 22/05/03 09:16:42 ERROR CodeGenerator: failed to compile:
> org.codehaus.commons.compiler.CompileException: File 'generated.java', Line
> 37, Column 5: Expression "value_1" is not an rvalue
> ....
> 0-5
> Time taken: 0.175 seconds, Fetched 1 row(s)
> spark-sql> select duration/num from v1;
> 22/05/03 09:17:41 ERROR CodeGenerator: failed to compile:
> org.codehaus.commons.compiler.CompileException: File 'generated.java', Line
> 37, Column 5: Expression "value_1" is not an rvalue
> ...
> 5 00:00:00.000000000
> Time taken: 0.104 seconds, Fetched 1 row(s)
> {noformat}
> Note that in the error cases, the queries still return a result because Spark
> falls back on interpreting the divide expression (so I marked this as
> "minor").
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
