fbaranski opened a new issue, #20631:
URL: https://github.com/apache/druid/issues/20631
### Affected Version
38.0.0 (Calcite 1.37.0)
### Description
Planning a SQL query where `JSON_QUERY_ARRAY` is applied to a constant JSON
value fails with a `java.lang.StackOverflowError` (HTTP 500 on
`/druid/v2/sql`). The constant reduction of the resulting
`ARRAY<COMPLEX<json>>` expression recurses forever in Calcite's
`RexBuilder.makeLiteral`.
**Cluster size**
- Reproduced in the Druid SQL test framework (`BaseCalciteQueryTest`,
`SqlTestFramework.StandardComponentSupplier`) with built-in functions only.
- First seen on a single-machine local Docker cluster (coordinator, broker,
historical, middleManager, router), on the broker at planning time.
**Configurations in use**
Default configuration and query context; nothing specific is needed to
reproduce it.
**Steps to reproduce the problem**
```sql
SELECT TO_JSON_STRING(v)
FROM (SELECT PARSE_JSON('{"a":[{"b":1}]}') AS j) t
CROSS JOIN UNNEST(JSON_QUERY_ARRAY(t.j, '$.a')) AS u(v)
```
Expected: one row, `{"b":1}`.
Actual: `StackOverflowError` during planning. The same happens whatever is
read from the unnested value (`JSON_VALUE(v, '$.b' RETURNING BIGINT)`,
`TO_JSON_STRING(JSON_QUERY(v, '$'))`). Reading a scalar from a constant without
`JSON_QUERY_ARRAY` works (`JSON_VALUE(PARSE_JSON('{"a":1}'), '$.a')`).
**Error message / stack trace**
```
java.lang.StackOverflowError
at
org.apache.calcite.rel.type.RelDataTypeFactoryImpl.copySimpleType(RelDataTypeFactoryImpl.java:334)
at
org.apache.calcite.rel.type.RelDataTypeFactoryImpl.createTypeWithNullability(RelDataTypeFactoryImpl.java:410)
at
org.apache.calcite.sql.type.SqlTypeFactoryImpl.createTypeWithNullability(SqlTypeFactoryImpl.java:233)
at org.apache.calcite.rex.RexBuilder.makeLiteral(RexBuilder.java:1695)
at org.apache.calcite.rex.RexBuilder.makeLiteral(RexBuilder.java:1664)
at org.apache.calcite.rex.RexBuilder.makeLiteral(RexBuilder.java:1697)
at org.apache.calcite.rex.RexBuilder.makeLiteral(RexBuilder.java:1664)
... (1664 / 1697 repeated until the stack overflows)
at
org.apache.druid.sql.calcite.planner.DruidRexExecutor.reduce(DruidRexExecutor.java:204)
at
org.apache.calcite.rel.rules.ReduceExpressionsRule.reduceExpressionsInternal(ReduceExpressionsRule.java:774)
at
org.apache.calcite.rel.rules.ReduceExpressionsRule.reduceExpressions(ReduceExpressionsRule.java:714)
at
org.apache.calcite.rel.rules.ReduceExpressionsRule$ProjectReduceExpressionsRule.onMatch(ReduceExpressionsRule.java:313)
...
at
org.apache.druid.sql.calcite.planner.CalcitePlanner.transform(CalcitePlanner.java:476)
at
org.apache.druid.sql.calcite.planner.QueryHandler.planWithDruidConvention(QueryHandler.java:586)
at
org.apache.druid.sql.calcite.planner.DruidPlanner.plan(DruidPlanner.java:248)
```
**Debugging already done**
In `DruidRexExecutor.reduce`, a complex constant (`SqlTypeName.OTHER`) is
left as an expression:
```java
} else if (sqlTypeName == SqlTypeName.OTHER) {
// complex constant is not reducible, so just leave it as an expression
literal = constExp;
}
```
but an `ARRAY` constant whose component type is complex (here
`ARRAY<COMPLEX<json>>`, the type of `JSON_QUERY_ARRAY`) is neither numeric nor
boolean, so it reaches the generic branch:
```java
} else {
literal = rexBuilder.makeLiteral(Arrays.asList(array), constExp.getType(),
true);
}
```
`RexBuilder.makeLiteral(value, type, allowCast=true)` then recurses between
lines 1664 and 1697 (the nullable branch that strips the nullability and calls
itself again) without ever terminating, apparently because the not-null copy of
the complex element type is still seen as nullable.
Leaving arrays of complex elements as expressions, like complex scalars,
fixes it:
```java
} else if (constExp.getType().getComponentType().getSqlTypeName() ==
SqlTypeName.OTHER) {
// array of complex constants is not reducible either, leave it as an
expression
literal = constExp;
} else {
literal = rexBuilder.makeLiteral(Arrays.asList(array), constExp.getType(),
true);
}
```
Tested against 38.0.0 by shadowing `DruidRexExecutor` with this change in a
`BaseCalciteQueryTest`: the query above and its `JSON_VALUE` / `JSON_QUERY`
variants plan and return the expected row (vectorized and not); without the
change they all fail with the `StackOverflowError`. The Druid test suite itself
was not run with the change. A `CalciteNestedDataQueryTest` case with the query
above would cover it; happy to open a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
