voonhous opened a new issue, #20659:
URL: https://github.com/apache/hudi/issues/20659
### Bug Description
**What happened:**
On a MOR snapshot (`_rt`) read, the Trino connector returns different CHAR
values than the read-optimized table or the base file does for the same row. It
returns no error, only wrong values and missing rows:
- Leading spaces are dropped: `' ab'` reads back as `'ab'`.
- Tabs and other control characters at either end are dropped: `'\tx\t'`
reads back as `'x'`.
- A filter like `WHERE code = CAST(' ab' AS CHAR(5))` silently skips the row
on the snapshot read but matches it on the read-optimized read. Joins and GROUP
BY on the column are affected the same way.
Both updated rows and rows that were never updated are affected, as long as
their file group has a log file, because the merge converts every base row in
the group as well.
Cause: `HudiAvroSerializer.writeSlice` converts CHAR with `String.trim()`,
which strips all leading and trailing characters <= U+0020. Trino's Parquet
reader only removes trailing U+0020 padding
(`Chars.byteCountWithoutTrailingSpace`). The code came in with the connector
migration in #18837 and has not changed since.
**What you expected:**
The snapshot read returns the same CHAR value as the read-optimized read:
truncated to the declared length and with only trailing spaces removed.
**Steps to reproduce:**
1. Create a MOR table where the metastore declares a column `code char(5)`
over a Hudi string field.
2. Insert `k3` with `code = ' ab'` and `k1` with any value, then upsert `k1`
with `code = '\tx\t'` so the file group gets a log file.
3. Compare `SELECT code FROM t WHERE key = 'k3'` with the same query on
`t_rt`: `' ab '` vs `'ab '`.
A functional test that reproduces this is below. It fails on master and
passes with the fix in #20638.
<details><summary>Repro test (TestHudiMorMergeModeSemantics, on
CommitTimeOrderingHudiTablesInitializer)</summary>
Fixture change: add `new Column("code", charHiveType(5), ...)` with Avro
type STRING; base rows `k1 = "k1"`, `k2 = "k2"`, `k3 = " ab"`; the k1 log
update writes `"\tx\t"`.
```java
@Test
public void testCharMatchesBaseReaderOnUntouchedRow()
{
String query = "SELECT code FROM %s WHERE key = 'k3'";
Object readOptimized =
computeScalar(query.formatted(CommitTimeOrderingHudiTablesInitializer.TABLE_NAME));
assertThat(readOptimized).isEqualTo(" ab ");
assertThat(computeScalar(query.formatted(CommitTimeOrderingHudiTablesInitializer.RT_TABLE_NAME))).isEqualTo(readOptimized);
}
@Test
public void testCharFilterMatchesLeadingSpaceOnSnapshotRead()
{
assertQuery(
"SELECT key FROM " +
CommitTimeOrderingHudiTablesInitializer.RT_TABLE_NAME + " WHERE code = CAST('
ab' AS CHAR(5))",
"VALUES 'k3'");
}
@Test
public void testCharKeepsTabsOnUpdatedRow()
{
assertThat(computeScalar("SELECT code FROM " +
CommitTimeOrderingHudiTablesInitializer.RT_TABLE_NAME + " WHERE key = 'k1'"))
.isEqualTo("\tx\t ");
}
```
</details>
### Environment
**Hudi version:** master (1.3.0-SNAPSHOT, at a65a573a)
**Query engine:** Trino, in-repo `hudi-trino` connector
**Relevant configs:** MOR table, snapshot (`_rt`) query, metastore column
type `char(n)`
### Logs and Stack Trace
No error is raised. The test output on master is below.
```
testCharMatchesBaseReaderOnUntouchedRow expected: " ab " but was:
"ab "
testCharKeepsTabsOnUpdatedRow expected: "\tx\t " but was: "x
"
testCharFilterMatchesLeadingSpaceOnSnapshotRead expected 1 row (k3), got 0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]