LuciferYang commented on PR #12981: URL: https://github.com/apache/gluten/pull/12981#issuecomment-5598190420
Pushed 9b4261e, which removes the Spark 3.3 native write post-rule pipeline. I had left this out of the first pass and it belongs here. `NativeWritePostRule` intercepted `DataWritingCommandExec` and injected a fake row adaptor so the vanilla writer could consume columnar data. It reached the plan through `GlutenFormatFactory.getExtendedColumnarPostRule`, and the only shim that ever returned a non-empty `getExtendedColumnarPostRules()` was `Spark33Shims`, so the rule stopped being reachable the moment 3.3 was removed in #12902. Since Spark 3.4 the write is a real `WriteFilesExec` node that `OffloadOthers` turns into `ColumnarWriteFilesExec`, which is why nothing replaced it. Gone with it: `NativeWritePostRule`, `getNativeFormat` and its `formatMapping`, `GlutenFormatFactory.postRuleFactory` and `injectPostRuleFactory`, the `CHListenerApi` registration, and `BackendSettingsApi.skipNativeCtas` / `skipNativeInsertInto` with their `VeloxBackend` overrides, whose only reader was `getNativeFormat`. Kept, because each still has another caller: `enableNativeWriteFiles()` (`Validators`, `DeltaV1Writes`), `injectFakeRowAdaptor` (`GlutenNoopWriterRule`), and `injectSparkLocalProperty` (`ClickhouseOptimisticTransaction`). Compile matrix re-run after this, all Scala 2.13: velox and clickhouse on 3.5 including test sources, and `gluten-core` plus `gluten-substrait` on 3.4, 4.0 and 4.1, with velox on the two 4.x. `spotless:check` clean. Still no suite run locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
