ashokraminedi opened a new issue, #20143:
URL: https://github.com/apache/hudi/issues/20143
### Bug Description
**What happened:**
While investigating #20108 and the fix in #20130, I found that the same
schema-on-read batch-read issue can be reproduced for additional source types
when evolving a column to a DECIMAL type whose precision is too small for the
existing value.
The issue reproduces for:
- FLOAT -> DECIMAL
- DOUBLE -> DECIMAL
- STRING -> DECIMAL
For example, with an existing value `12345.0` evolved to `DECIMAL(4,2)`, the
vectorized batch read returns `1234.50`, while the non-vectorized read returns
`NULL`.
I reproduced the same behavior for both COW and MOR tables.
Results:
| Source type | Vectorized read | Non-vectorized read |
|---|---|---|
| FLOAT | `1234.50` | `NULL` |
| DOUBLE | `1234.50` | `NULL` |
| STRING | `1234.50` | `NULL` |
**What you expected:**
Vectorized and non-vectorized schema-on-read paths should return consistent
results.
With ANSI mode disabled, when an existing value cannot fit the target
DECIMAL precision, the vectorized read should return `NULL` rather than an
incorrectly rescaled value.
**Steps to reproduce:**
1. Create a Hudi COW or MOR table with a `price` column of type FLOAT,
DOUBLE, or STRING and insert a value that cannot fit into `DECIMAL(4,2)`. For
example, for FLOAT:
```sql
create table <table_name> (
id int,
price float,
ts long
) using hudi
tblproperties (
type = 'cow',
primaryKey = 'id',
orderingFields = 'ts'
);
insert into <table_name> values (1, 12345.0, 1000);
```
2. Enable schema-on-read, disable ANSI mode, and evolve the column to
`DECIMAL(4,2)`:
```
hoodie.schema.on.read.enable=true
spark.sql.ansi.enabled=false
```
```
alter table <table_name>
alter column price type decimal(4, 2);
```
3. Run the same query with the vectorized reader enabled and disabled:
```
select id, cast(price as string) from <table_name>;
```
With the vectorized reader enabled:
```
spark.sql.parquet.enableVectorizedReader=true
```
the result is:
```
[1,1234.50]
```
With the vectorized reader disabled:
```
spark.sql.parquet.enableVectorizedReader=false
```
the result is:
```
[1,null]
```
The same discrepancy reproduces when the original `price` type is FLOAT,
DOUBLE, or STRING, for both COW and MOR tables.
### Environment
**Hudi version:** master (`23a62e4fe6bc`)
**Query engine:** Spark
**Relevant configs:**
```
hoodie.schema.on.read.enable=true
spark.sql.ansi.enabled=false
spark.sql.parquet.enableVectorizedReader=true/false
```
### Logs and Stack Trace
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
