malinjawi commented on PR #12836: URL: https://github.com/apache/gluten/pull/12836#issuecomment-5396913399
@marin-ma I checked the remaining cases and do not see a normal workload that benefits from loading on-disk DVs on the driver. The only theoretical differences are earlier failure for a missing/corrupt sidecar and avoiding a repeated DV request on a retried or speculative task; neither justifies preserving the serial planning path. The standard Delta fallback already reads DVs on executors. I will remove the rollback config and the legacy on-disk driver branch, keep inline DVs eager because their bytes are already in Delta metadata, and retain `spark.gluten.sql.columnar.filescan=false` as the broader fallback. Please let me know if you see a compatibility case that requires a different direction. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
