raghavyadav01 opened a new pull request, #19999: URL: https://github.com/apache/pinot/pull/19999
## Problem Parquet stores an unsigned 32-bit integer (`INT32` annotated `INTEGER(32, false)`, or legacy `UINT_32`) as the signed int bit pattern. Both Parquet record readers returned that int as is, so any value above `Integer.MAX_VALUE` was ingested as a negative number: `3540738458` came back as `-754228838`. ## Change - `ParquetNativeRecordExtractor` (the default reader for files without an embedded Avro schema) widens unsigned 32-bit values to `Long` with `Integer.toUnsignedLong`. This applies at every level, including list elements, struct fields and map values. - `ParquetAvroRecordReader` reads the file schema once to find top-level unsigned 32-bit fields and `ParquetAvroRecordExtractor` widens them the same way. parquet-avro maps these columns to a signed Avro `int` and has no signedness information, so nested values on this reader are still read as signed; the class doc says so, and the native reader handles them. - Unsigned 8- and 16-bit values always fit in an `int` and are unchanged. Unsigned 64-bit is out of scope (values at or above 2^63 still read negative and would need `BigDecimal`). ## Compatibility Only rows with values at or above 2^31 change, and only when the Pinot column is not `INT`: - `INT` columns store exactly what they stored before (`Long` → `int` narrowing keeps the bit pattern). - `LONG`, `STRING`, `DOUBLE`, `BIG_DECIMAL`, `JSON` and `MAP` columns now store the correct value instead of the negative one. Segments built before this change keep the old value until they are re-ingested, so filters, joins and upsert/dedup keys on such a column can disagree between old and new segments. Ingestion transforms and filter functions that run before type conversion now see a `Long`. Hence the `backward-incompat` label. ## Tests `ParquetReaderLogicalTypesTest`: - `testUnsignedInt32Type` writes `INTEGER(32, false)`, `UINT_32` and `INTEGER(16, false)` columns and reads them through `ParquetNativeRecordReader`, `ParquetAvroRecordReader` and `ParquetRecordReader`. It fails on master (`ParquetNativeRecordReader expected [3540738458] but found [-754228838]`). - `testNestedUnsignedInt32TypeWithNativeReader` covers a list and a struct of unsigned 32-bit values. The `pinot-parquet` module passes (61 tests). 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
