JingsongLi commented on PR #10394: URL: https://github.com/apache/paimon/pull/10394#issuecomment-6081695274
**[P1] Fix the physical row-sidecar schema before statistics ranges can select it** I reproduced incorrect query results on `859e1e6392096607f1c295c14e5e51b403cf20cd`. The new `groupRowRanges = candidates` path in `DataEvolutionSplitRead` makes an ordinary filtered merged read eligible for `readTarget`'s row sidecar. Previously, the same query had no explicit row ranges and used Parquet. This HEAD's `formatBuilder` still adds the virtual row-ID/sequence fields to the physical data schema (`rowTypeWithRowTracking(schema.logicalRowType(), true, true)`). A `.row` file with seven physical INT fields has a one-byte null header; its reader is built as nine fields and consumes a two-byte header, shifting the decoded values. Actual table reproduction on JDK 8: 1. Create seven nullable INT columns `c0` … `c6`, with row tracking, data evolution and row sidecars enabled, Parquet block size 40000 and page size 1024. Keep the default sidecar selection thresholds. 2. Write 20,000 base rows (`c0=i`, `c1=i`, remaining columns `i+columnIndex`). Commit a projected update of only `c1=19999-i`, covering the original row-ID range, as a partial-column MERGE does. 3. Read the projection `c0,c1` with predicate `c1 < 100`, without supplying any row ranges. With statistics pushdown disabled, all 100 rows match the oracle: `19900:99`, `19901:98`, `19902:97`, etc. With the new default enabled, 100 rows are returned but `c0` is corrupted (for example `-100663219:37`, `-1006632883:91`). Setting the sidecar selection ratio to `0.000001`, so the same statistics ranges are read from Parquet, restores the exact expected results with pushdown still enabled. The existing sidecar test uses four physical columns, so adding two virtual fields does not cross the null-header size boundary. Please build row-sidecar readers from the actual physical schema, or keep this new path from selecting them until that fix is present, and add this wider-table regression. The optimization has clear value; this wrong-result path needs to be resolved before enabling it by default. Validation: the seven targeted existing/new core and Parquet test classes passed all 131 tests with standard Maven checks. The additional table reproduction failed only with the new path enabled; its Parquet control passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
