manuzhang commented on code in PR #7106:
URL: https://github.com/apache/datafusion-comet/pull/7106#discussion_r4390233573
##########
native/core/src/parquet/parquet_support.rs:
##########
@@ -330,6 +330,33 @@ fn parquet_convert_array_impl(
Ok(Arc::new(string_array))
}
+ // Unannotated BINARY read as a string. Spark's cast replaces
malformed UTF-8 with
+ // U+FFFD, where Arrow's safe cast below would return NULL. Top-level
columns don't
+ // reach this arm (#7032). The reader keeps a file's ARROW:schema
types, so an Arrow
+ // writer's `LargeBinary` or `BinaryView` arrives as is; Spark's cast
takes only
+ // `Binary`, and converting `LargeBinary` reuses its value buffer.
+ (Binary | LargeBinary | BinaryView, Utf8) => {
+ let array = match from_type {
+ Binary => array,
+ _ => cast_with_options(&array, &Binary, &PARQUET_OPTIONS)?,
+ };
+ // Binary to Utf8 reads only the binary output style, which a scan
leaves unset.
+ let cast_options =
SparkCastOptions::new_without_timezone(EvalMode::Legacy);
+ let len = array.len();
+ spark_cast(ColumnarValue::Array(array), to_type,
&cast_options)?.into_array(len)
+ }
+ // The same schema can dictionary-encode such a value. Decode each
dictionary value once
+ // through the arm above, then expand the keys, as Arrow's cast would.
+ (Dictionary(_, value_type), Utf8) if matches!(value_type.as_ref(),
Binary | LargeBinary) => {
+ let dictionary = array.as_any_dictionary();
+ let values = parquet_convert_array_impl(
+ Arc::clone(dictionary.values()),
+ to_type,
+ parquet_options,
+ None,
+ )?;
+ Ok(take(values.as_ref(), dictionary.keys(), None)?)
Review Comment:
I don't think it's feasible in this PR so a clearer error message has been
added with suggestion of a smaller `spark.comet.batchSize`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
