jiayuasu opened a new pull request #494: URL: https://github.com/apache/incubator-sedona/pull/494
## Is this PR related to a proposed Issue? https://issues.apache.org/jira/browse/SEDONA-7 ## What changes were proposed in this PR? 1. Add a source code converter which can switch the source code between Spark 2.4 and 3.0. Scala does not allow conditional compilation so I have to write a silly preprocessor to convert the source code between 2.4 and 3.0. There are a few function calls incompatible across versions. Run `python3 spark-version-converter.py spark3` or `python3 spark-version-converter.py spark2` to convert the code to the target spark version 2. Configure the pom.xml to yield the following artifacts. Run the following commands: ``` python3 spark-version-converter.py spark2 mvn clean install -DskipTests -Dscala.compat.version="2.11" -Dscala.version="2.11.8" -Dspark.version="2.4.7" -Dspark.compat.version="2.4" ``` Generate * sedona-core_2.11-1.0.0-incubator-SNAPSHOT.jar * sedona-sql-2.4_2.11-1.0.0-incubator-SNAPSHOT.jar * sedona-viz-2.4_2.11-1.0.0-incubator-SNAPSHOT.jar * sedona-python-adapter_2.11-1.0.0-incubator-SNAPSHOT.jar ``` python3 spark-version-converter.py spark3 mvn clean install -DskipTests -Dscala.compat.version="2.12" -Dscala.version="2.12.8" -Dspark.version="3.0.1" -Dspark.compat.version="3.0" ``` Generate * sedona-core_2.12-1.0.0-incubator-SNAPSHOT.jar * sedona-sql-3.0_2.12-1.0.0-incubator-SNAPSHOT.jar * sedona-viz-3.0_2.12-1.0.0-incubator-SNAPSHOT.jar * sedona-python-adapter_2.12-1.0.0-incubator-SNAPSHOT.jar 3. GitHub action will be tested on * Scala and Java: Spark 2.4.7 (Scala 2.11, 2.12) and Spark 3.0 (Scala 2.12) * Python build: Spark 2.4.7 (Scala 2.11) and Spark 3.0 (Scala 2.12), Python 3.7 ## How was this patch tested? ## Did this PR include necessary documentation updates? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
