[GitHub] [spark] yaooqinn commented on a change in pull request #31921: [SPARK-34817][SQL] Read parquet unsigned types that stored as int32 physical type in parquet

[GitBox]

yaooqinn commented on a change in pull request #31921:
URL: https://github.com/apache/spark/pull/31921#discussion_r600360823



##########
File path: 
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -565,6 +565,10 @@ private void readIntBatch(int rowId, int num, 
WritableColumnVector column) throw
         canReadAsIntDecimal(column.dataType())) {
       defColumn.readIntegers(
           num, column, rowId, maxDefLevel, (VectorizedValuesReader) 
dataColumn);
+    } else if (column.dataType() == DataTypes.LongType) {
+      //  We use LongType to handle UINT32
+      defColumn.readIntegersAsUnsigned(

Review comment:
       I have added the dictionary decoding code path, change the parquet data 
generator a bit to produce right encoded/plain data




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org