kevinjqliu commented on issue #3633:
URL: https://github.com/apache/parquet-java/issues/3633#issuecomment-4996459310
Backported #3669 to the `parquet-1.15.x` branch and verified that it fixed
opening parquet files with variant type and unknown type. Variant is not known
to `1.15.x` branch, it was introduced in `1.16.x`
Setup:
```
sdk use java 8.0.492-zulu
java -version
# My laptop has Thrift 0.23; this should not affect the result.
./mvnw -Plocal -pl parquet-cli -am -DskipTests \
-Dthrift.version=0.23.0 \
-Dformat.thrift.version=0.23.0 \
package
```
Run:
```
CLI="$PWD/parquet-cli/target/parquet-cli-1.15.3-SNAPSHOT-runtime.jar"
for file in ~/Downloads/unknown-logical-type.parquet
~/Downloads/case-001.parquet; do
echo "Testing: $file"
java -jar "$CLI" schema --parquet "$file"
java -jar "$CLI" meta "$file"
done
```
```
Testing: /Users/kevinliu/Downloads/unknown-logical-type.parquet
message schema {
optional binary column with known type (STRING);
optional binary column with unknown type;
}
File path: /Users/kevinliu/Downloads/unknown-logical-type.parquet
Created by: parquet-cpp-arrow version 20.0.0-SNAPSHOT
Properties:
ARROW:schema:
/////2ABAAAQAAAAAAAKAAwABgAFAAgACgAAAAABBAAMAAAACAAIAAAABAAIAAAABAAAAAIAAADwAAAAGAAAAAAAEgAYAAgABgAHAAwAAAAQABQAEgAAAAAAAQQUAAAAtAAAAAgAAAAoAAAAAAAAABgAAABjb2x1bW4gd2l0aCB1bmtub3duIHR5cGUAAAAAAgAAAEwAAAAEAAAAxP///xwAAAAEAAAADAAAAGdlb2Fycm93LndrYgAAAAAUAAAAQVJST1c6ZXh0ZW5zaW9uOm5hbWUAAAAACAAMAAQACAAIAAAAEAAAAAQAAAACAAAAe30AABgAAABBUlJPVzpleHRlbnNpb246bWV0YWRhdGEAAAAAuP///xAAFAAIAAYABwAMAAAAEAAQAAAAAAABBRAAAAAsAAAABAAAAAAAAAAWAAAAY29sdW1uIHdpdGgga25vd24gdHlwZQAABAAEAAQAAAAAAAAA
Schema:
message schema {
optional binary column with known type (STRING);
optional binary column with unknown type;
}
Row group 0: count: 3 62.33 B records start: 4 total(compressed): 187 B
total(uncompressed):234 B
--------------------------------------------------------------------------------
type encodings count avg size nulls
min / max
column with known type BINARY S _ R 3 36.33 B 0
"known string 1" / "known string 3"
column with unknown type BINARY S _ R 3 26.00 B
Testing: /Users/kevinliu/Downloads/case-001.parquet
message table {
required int32 id = 1;
optional group var = 2 {
required binary metadata;
optional binary value;
optional group typed_value (LIST) {
repeated group list {
required group element {
optional binary value;
optional binary typed_value (STRING);
}
}
}
}
}
File path: /Users/kevinliu/Downloads/case-001.parquet
Created by: parquet-mr version 1.16.0-SNAPSHOT (build
ee34713e4d906d61f95d2b09145945638b2e2296)
Properties:
parquet.avro.schema:
{"type":"record","name":"table","fields":[{"name":"id","type":"int"},{"name":"var","type":["null",{"type":"record","name":"var","fields":[{"name":"metadata","type":"bytes"},{"name":"value","type":["null","bytes"],"default":null},{"name":"typed_value","type":["null",{"type":"array","items":{"type":"record","name":"element","fields":[{"name":"value","type":["null","bytes"],"default":null},{"name":"typed_value","type":["null","string"],"default":null}]}}],"default":null}]}],"default":null}]}
writer.model.name: avro
Schema:
message table {
required int32 id = 1;
optional group var = 2 {
required binary metadata;
optional binary value;
optional group typed_value (LIST) {
repeated group list {
required group element {
optional binary value;
optional binary typed_value (STRING);
}
}
}
}
}
Row group 0: count: 1 186.00 B records start: 4 total(compressed): 186 B
total(uncompressed):186 B
--------------------------------------------------------------------------------
type encodings count avg
size nulls min / max
id INT32 _ _ 1
27.00 B 0 "1" / "1"
var.metadata BINARY _ _ 1
36.00 B 0 "0x010000" / "0x010000"
var.value BINARY _ _ 1
30.00 B 1
var.typed_value.list.element.value BINARY _ _ 2
18.50 B 2
var.typed_value.list.element.typed_value BINARY _ _ 2
28.00 B 0 "comedy" / "drama"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
