unikdahal commented on code in PR #5361:
URL: https://github.com/apache/datafusion-comet/pull/5361#discussion_r3915590152
##########
spark/src/main/scala/org/apache/comet/serde/operator/CometIcebergNativeWrite.scala:
##########
@@ -172,21 +169,23 @@ object CometIcebergNativeWrite extends
CometOperatorSerde[IcebergWriteExec] with
PropertyKeys.WriteLocationProviderImpl,
"custom location provider unsupported"),
requireFormatVersionAtMostTwo,
+ requireNoUuidColumns,
requireNoEncryptionPrefix,
- requireSupportedMetricsModes,
requireNoBloomFilterColumnsEnabled,
requireRowGroupCheckMinRecordCountAtDefault,
requireRowGroupCheckMaxRecordCountAtDefault,
requireParquetPageVersionDefault,
requireShredVariantsDisabled,
+ requireParseableCompressionLevel,
requireOnlyVettedParquetWriteProperties,
requirePropertyAbsent(
PropertyKeys.ParquetEnableDictionary,
"dictionary override unsupported"),
requireNoUnvettedParquetMrProperties,
requirePropertyAbsent(PropertyKeys.FileIOImpl, "custom FileIO
unsupported"),
Review Comment:
Thanks, validating the instantiated table.io() fixes the custom-FileIO case
I raised, but I think there is still a configuration-equivalence gap for a
recognized HadoopFileIO.
The write proto currently builds catalogProperties only from
getFileIOProperties(table). properties() is not part of the FileIO interface
and HadoopFileIO normally carries its S3A configuration through Hadoop
Configuration instead.
The existing native Iceberg scan path already handles this explicitly: it
calls NativeConfig.extractObjectStoreOptions(...) and translates fs.s3a.*
settings into the s3.* properties iceberg-rust/OpenDAL consumes. The write path
doesn't do that.
So a HadoopFileIO table at s3a://... can pass this new compatibility gate
while the JVM writer sees credentials/custom endpoint/path-style settings from
spark.hadoop.fs.s3a.* that the native writer never receives.
Could we reuse the scan-side Hadoop-property extraction/translation here and
merge it with the FileIO properties, with table/vended FileIO properties taking
precedence?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
