tkaymak commented on PR #38212: URL: https://github.com/apache/beam/pull/38212#issuecomment-4269714692
@Abacn — taking a swing at the source-override restructuring you suggested. Split into two stacked branches on my fork: **PR A (refactor only, no Spark 4):** #38233 - Hoists `runners/spark/3/src/.../structuredstreaming/` to shared `runners/spark/src/` (Spark 3's only sources, so `runners/spark/3/src/` is removed entirely). - Replaces the existing `copySourceBase` block in `spark_runner.gradle` with the same `Copy` + `DuplicatesStrategy.INCLUDE` layering used by `runners/flink/flink_runner.gradle`. - Diff: 76 files, +96/-25. Spark 3 build behavior unchanged. **PR B (Spark 4, on top of PR A):** branch [`spark4-runner-slim`](https://github.com/tkaymak/beam/tree/spark4-runner-slim) on my fork. - Same 14 commits as this PR, plus a final cleanup that drops 62 byte-identical structured-streaming files from `runners/spark/4/src/`. Only the **11** files that genuinely differ for Spark 4 / Scala 2.13 remain as overrides. - Diff vs PR A: ~31 added files / ~3.3K lines (down from this PR's ~94 / ~10.5K). How would you like to proceed? Options I see: 1. **Land PR A first**, then I close this PR and open a fresh PR B against master once A is merged. 2. **Keep this PR open** and rebase it onto PR A's branch once A merges (turns this PR into the slim version automatically). 3. Something else. Happy to switch the head branch of this PR to `spark4-runner-slim` immediately if that's easier — but didn't want to force-push without your sign-off given existing review threads here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
