andygrove opened a new issue, #7637:
URL: https://github.com/apache/datafusion-comet/issues/7637

   ### What is the problem the feature request solves?
   
   Native Iceberg writes fall back to iceberg-java for an `hdfs` data location. 
The gate in `CometIcebergNativeWrite` admits no `hdfs` scheme, so every write 
to an Iceberg table on HDFS runs on the JVM writer.
   
   This no longer waits on iceberg-rust. apache/iceberg-rust#3111 added the 
`hdfs-native` storage backend, and #5898 already reads Iceberg tables on HDFS 
natively through it (#5894). The remaining work is on Comet's side.
   
   ### Describe the potential solution
   
   Admit `hdfs` data locations in the native write gate, and route writes 
through the same `hdfs-native` backend and configuration the native scan uses. 
Keep the cases the native scan declines falling back, in particular a cluster 
with Kerberos and a location whose nameservice cannot be resolved from the 
Hadoop configuration.
   
   ### Additional context
   
   Part of #5649. Recorded under "Plans for each fallback" in the Iceberg user 
guide (#6961).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to