niketanpansare commented on issue #857: [SYSTEMML-2523] Update SystemML to Support Spark 2.3.0 URL: https://github.com/apache/systemml/pull/857#issuecomment-475413637 Interestingly, running a similar code with `1.2.0` jars in `spark-2.3.0../spark-shell` succeeds, i.e. behaves similar to setup 5 rather than setup 6. Here is the Scala code used for testing: ``` val ml = new org.apache.sysml.api.mlcontext.MLContext(spark) System.out.println(ml.version()) val df = spark.read.parquet("shake.parquet") df.show() df.createOrReplaceTempView("df") ``` Based on the above experiments, here are my thoughts: 1. We can continue to support older Spark 2.1 version and can get away with warning on Spark 2.3 in the following setups: - Invoked without any Spark SQL code - Part of Scala/Java pipeline (for example: if invoked via spark-shell) - With PySpark if and only if we recommend our users to not provide any jars in the `driver-class-path or jars` (see setup 5 and 6) 2. If we are uncomfortable with the above restriction, we should consider merging this PR. - Though I have validated that above Python code works with Spark 2.2.3 with a warning, I did not run exhaustive testing to guarantee backward compatibility support for older Spark 2.1 and 2.2 (with the exception of warning).
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
