dd-annarose opened a new issue, #26665:
URL: https://github.com/apache/datafusion/issues/26665
### Describe the bug
In Substrait plans, function outputs have a declared type.
The **substrait consumer ignores the declared type** which can lead to a
schema that does not match its substrait plan. This concerns numeric types
(Decimals, Integers, Floats), timestamps, and nested types.
For example, with `divide:dec_dec`, the output is declared as `Decimal<38,
6>` in the plan below.
`from_scalar_function` maps add/subtract/multiply/divide straight to a
BinaryExpr through name_to_op. It never reads f.output_type, so the plan's
decimal<38,6> is dropped. The planner instead uses `Decimal<(38,22)>` in this
case.
```
"scalarFunction": {
"functionReference": 13,
"outputType": {
"decimal": {
"scale": 6,
"precision": 38,
"nullability": "NULLABILITY_NULLABLE"
}
},
```
This issue is linked to #25043 but I would like this contract enforced for
all impacted types, not only for decimals.
### To Reproduce
using the plan below, you can see the output type with:
```
let session_state = SessionStateBuilder::new().build();
let substrait = include_str!("../substrait/upstream.json");
let plan = from_substrait_plan(&session_state, &substrait.to_plan()?).await?;
println!("{:?}", plan.schema().field(0).data_type());
```
```
{
"extensionUris": [
{
"extensionUriAnchor": 1,
"uri":
"https://github.com/substrait-io/substrait/blob/main/extensions/functions_arithmetic_decimal.yaml";
}
],
"extensions": [
{
"extensionFunction": {
"extensionUriReference": 1,
"functionAnchor": 1,
"name": "multiply:dec_dec"
}
},
{
"extensionFunction": {
"extensionUriReference": 1,
"functionAnchor": 2,
"name": "divide:dec_dec"
}
}
],
"relations": [
{
"root": {
"names": [
"result"
],
"input": {
"project": {
"common": {
"emit": {
"outputMapping": [
1
]
}
},
"expressions": [
{
"scalarFunction": {
"functionReference": 2,
"outputType": {
"decimal": {
"scale": 6,
"precision": 38,
"nullability": "NULLABILITY_REQUIRED"
}
},
"arguments": [
{
"value": {
"scalarFunction": {
"functionReference": 1,
"outputType": {
"decimal": {
"scale": 6,
"precision": 38,
"nullability": "NULLABILITY_REQUIRED"
}
},
"arguments": [
{
"value": {
"literal": {
"decimal": {
"value": "AOh2SBcAAAAAAAAAAAAAAA==",
"precision": 38,
"scale": 9
}
}
}
},
{
"value": {
"selection": {
"directReference": {
"structField": {
"field": 0
}
},
"rootReference": {}
}
}
}
]
}
}
},
{
"value": {
"selection": {
"directReference": {
"structField": {
"field": 0
}
},
"rootReference": {}
}
}
}
]
}
}
],
"input": {
"read": {
"common": {
"direct": {}
},
"baseSchema": {
"names": [
"x"
],
"struct": {
"types": [
{
"decimal": {
"scale": 9,
"precision": 38,
"nullability": "NULLABILITY_REQUIRED"
}
}
],
"nullability": "NULLABILITY_REQUIRED"
}
},
"virtualTable": {
"expressions": [
{
"fields": [
{
"literal": {
"decimal": {
"value": "AEAZ7Y8oAQAAAAAAAAAAAA==",
"precision": 38,
"scale": 9
}
}
}
]
}
]
}
}
}
}
}
}
}
]
}
```
### Expected behavior
Results should respect the plan's schema — and should fail if that schema is
not valid.
Unexpected behavior regarding return types is a gap in the contract between
planning and execution.
### Additional context
Potential fix:
- `from_scalar_function` checks the planned output type against the
Substrait type
- Decimals: cast only if the declared type has at least as many integer
digits (p − s) as the inferred one. Fewer fractional digits are allowed,
because that rounds instead of erroring.
- Integers and floats: cast only when the target is at least as wide (Int32
→ Int64, UInt32 → Int64, Float32 → Float64). Narrowing casts and int → float
are skipped.
### Willingness to contribute
I plan to contribute a fix for this bug independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
