HyukjinKwon commented on a change in pull request #29312: URL: https://github.com/apache/spark/pull/29312#discussion_r463402736
########## File path: .github/workflows/master.yml ########## @@ -271,3 +271,29 @@ jobs: mkdir -p ~/.m2 ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=11 install rm -rf ~/.m2/repository/org/apache/spark + + scala213: + name: Scala 2.13 + runs-on: ubuntu-latest + steps: + - name: Checkout Spark repository + uses: actions/checkout@v2 + - name: Cache Maven local repository + uses: actions/cache@v2 + with: + path: ~/.m2/repository + key: scala213-maven-${{ hashFiles('**/pom.xml') }} + restore-keys: | + scala213-maven- + - name: Install Java 11 + uses: actions/setup-java@v1 + with: + java-version: 11 + - name: Test with Maven + run: | + export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=1g -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN" + export MAVEN_CLI_OPTS="--no-transfer-progress" + mkdir -p ~/.m2 + dev/change-scala-version.sh 2.13 + build/mvn test -pl core --am -Pscala-2.13 Review comment: My only concern is that it becomes difficult to port these to periodical GitHub Actions jobs or merge it to the main GitHub Actions job (`Build modules: ...` above) later. Eventually I would like to: - put those different profiles build into the matrix in the main job `Build modules: ...` above so we can leverage the optimization in `run-tests.py` (running only relevant jobs, etc.). We could decide which profiles to run by default or optionally. - provide a way to set other optional profiles, for example, via GitHub PR title or tags, which we currently does via Jenkins. - set periodical jobs for other profiles for Maven, Scala, JDK, Hive, Hadoop, etc. that we run in Jenkins, or possibly more. I think we need to resolve SPARK-32264 first to do this. Currently `run-tests.py` runs the build for all components always if Maven is used. So, I was also thinking about fixing `run-tests.py` script to support for Maven to run tests for only relevant components just like we do for SBT, and then we only have one main job (`Build modules: ...`) that adds tests by matrix or environment control in this file rather than adding separate jobs for different profiles. I guess I am okay with this as a temporary status for the time being if we all agree with this plan. I would like to encourage us to stick to this plan in long run. Maybe we could alternatively just think about setting a Jenkins that that runs all tests too, which will ideally be converted into periodic Jenkins jobs. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org