Etienne Chauchot created FLINK-23327: ----------------------------------------
Summary: Spark Structured Streaming runner sources should be put back to main src folder Key: FLINK-23327 URL: https://issues.apache.org/jira/browse/FLINK-23327 Project: Flink Issue Type: Bug Components: Build System Reporter: Etienne Chauchot When spark 3 was introduced, *the spark structured streaming runner sources were moved to spark/2 module* (https://github.com/apache/beam/pull/14216/commits/9c0f6088078563ece6dc60877b0dcb320f1c9bfe) and the build system was changed. Now spark runners are built per spark version module (2 or 3) and the version specific sources are copied from spark-_version/src_ to spark-_version/build/source-overrides_ and gradle source sets are set to _main spark src_ + spark-_version/build/source-overrides._ The problem is that spark-_version/src_ is outside of gradle source sets and thus *the IDE does not compile this source*, that is not an issue when there is very few classes but in the case of the spark structured streaming runner, it is the whole runner code that is rendered invisible to the IDE which is impracticable for runner development We should just put back spark structured streaming sources to main spark sources. This will not change the build as it is still done per spark-version so deps will be ok. -- This message was sent by Atlassian Jira (v8.3.4#803005)