alexanderbianchi opened a new issue, #84: URL: https://github.com/apache/datafusion-iceberg/issues/84
Lazy Iceberg scans currently expose one DataFusion output partition. Eager planning in #67 can distribute retained file tasks across local partitions, but partitioned execution should also be possible without collecting all tasks first. Support distributing one lazy file-discovery stream across multiple local output partitions. The design should avoid independently listing the same manifests for every partition, preserve exactly-once task assignment within each execution, and define cancellation/backpressure behavior when partitions stop early or are never opened. Re-executing a physical plan must get independent execution state. Reader clones should continue sharing delete-loading state within an execution. Validate complete and filtered results, position deletes, LIMIT/cancellation, and repeated executions. This is a follow-up to #67, not a prerequisite for opt-in eager planning. Requested in review: https://github.com/apache/datafusion-iceberg/pull/67#discussion_r4348219778 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
