GitHub user Leemoonsoo opened a pull request: https://github.com/apache/zeppelin/pull/1169
[WIP] [ZEPPELIN-759] Spark 2.0 support ### What is this PR for? This PR implement spark 2.0 support based on #747. This PR has approach from #980 which is reimplementing code in scala. You can try build this branch ``` mvn clean package -Dscala-2.11 -Pspark-2.0 -Dspark.version=2.0.0-preview -Ppyspark -Psparkr -Pyarn -Phadoop-2.6 -DskipTests ``` ### What type of PR is it? Improvements ### Todos * [x] - Spark 2.0 support * [ ] - Rebase after #747 merge * [ ] - .travis.yml to use spark version 2.0.0 instead of 2.0.0-preview after spark 2.0.0 release * [ ] - Update LICENSE file * [ ] - Update related document (build) ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-759 ### How should this be tested? Build and try ``` mvn clean package -Dscala-2.11 -Pspark-2.0 -Dspark.version=2.0.0-preview -Ppyspark -Psparkr -Pyarn -Phadoop-2.6 -DskipTests ``` ### Screenshots (if appropriate) ![spark2](https://cloud.githubusercontent.com/assets/1540981/16771611/fe804038-4805-11e6-8447-3fa4258bb51d.gif) ### Questions: * Does the licenses files need update? yes * Is there breaking changes for older versions? no * Does this needs documentation? yes You can merge this pull request into a Git repository by running: $ git pull https://github.com/Leemoonsoo/zeppelin spark2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/1169.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1169 ---- commit 44622072f633f1aa2329df107f691f4a293bfc31 Author: Luciano Resende <lrese...@apache.org> Date: 2016-02-25T04:46:46Z [ZEPPELIN-605] Add support for Scala 2.11 Enable Zeppelin to be built with both Scala 2.10 and Scala 2.11, mostly to start supporting interpreters that are moving to Scala 2.11 only such as Spark. commit 864e52fc94997e16c2cb0922dd3af1423a4934a9 Author: Luciano Resende <lrese...@apache.org> Date: 2016-04-02T17:14:11Z [ZEPPELIN-605] Add Scala 2.11 build profile commit d8628c7158e29b076b661585ff5359f0e21a8438 Author: Luciano Resende <lrese...@apache.org> Date: 2016-04-07T15:16:10Z [ZEPPELIN-605] Enable Scala 2.11 REPL support for Spark Interpreter Scala 2.11 has changed a lot of it's REPL support and in order to enable building with Scala 2.10 and 2.11 we need to enable the new Scala 2.11 REPL and also make the build to compile based on the scala version. commit ac0088bc86fc25eecec7e737f3acbeb93c181a20 Author: Luciano Resende <lrese...@apache.org> Date: 2016-04-13T05:15:55Z [ZEPPELIN-605] Rewrite Spark interpreter based on Scala 2.11 support commit ceb1480be3d45f0c07f8dd01e42c0496863f2898 Author: Lee moon soo <m...@apache.org> Date: 2016-06-09T16:45:03Z Initial scala-210, 211 support in the single binary commit fefcb2b8ad8589c40cd3849f2b02a50486d89a3d Author: Lee moon soo <m...@apache.org> Date: 2016-06-10T00:40:39Z scala 2.11 support for spark interpreter commit 32650c7b58257af5cf723c4ca55e736e02e6344d Author: Lee moon soo <m...@apache.org> Date: 2016-06-10T02:29:36Z Fix some reflections commit ce4a2d828486b1200394ffe89192ece03c6ca750 Author: Lee moon soo <m...@apache.org> Date: 2016-06-10T02:57:57Z Fix reflection on creating SparkCommandLine commit fe3f8ea1fef6a4b5540cf3c70085156734659c8d Author: Lee moon soo <m...@apache.org> Date: 2016-06-10T19:40:07Z Fix reflection around HttpServer and createTempDir commit 35fc8a08403d24713e3c70c0c0e981fc1c005808 Author: Lee moon soo <m...@apache.org> Date: 2016-06-11T05:55:49Z fix style commit 9542299a56a76a86403119b39b9e2a126b3312de Author: Lee moon soo <m...@apache.org> Date: 2016-06-13T17:24:52Z Fix reflection commit 43d4a700c1d78b40009103b03cb5b6c0ac1d4ea3 Author: Lee moon soo <m...@apache.org> Date: 2016-06-13T17:31:39Z style commit 6da1364fe58df3188c1facdde2eae44f3dd77dfd Author: Lee moon soo <m...@apache.org> Date: 2016-06-13T18:10:19Z SparkContext sharing seems not working in scala 2.11, disable the test commit d7bd3ee10596008e41b6dfe6021449d161d50454 Author: Lee moon soo <m...@apache.org> Date: 2016-06-13T19:10:20Z Update FlinkInterpreter commit c7df53d9007c8c3e78308c881928cacf97cfb468 Author: Lee moon soo <m...@apache.org> Date: 2016-06-13T19:34:19Z Update ignite interpreter commit 1102d00fea1da86fab9c73f8055db37dc9959e87 Author: Lee moon soo <m...@apache.org> Date: 2016-06-13T19:51:27Z Remove unused methods commit 9b2adf844f7dfe0986b975e2e9adda2efad30e30 Author: Luciano Resende <lrese...@apache.org> Date: 2016-06-16T06:47:09Z Force close Closes lresende/incubator-zeppelin/#1 commit 36e4eedaebcc8e9fac2b9cd0992f3c2b042e48b0 Author: Lee moon soo <m...@apache.org> Date: 2016-06-29T23:09:15Z make binary built with scala 2.11 work with spark_2.10 binary Closes lresende/incubator-zeppelin/#2 commit f62dc46bad559d1a8f6b29b0cf8bbb35c60cdcbe Author: Luciano Resende <lrese...@apache.org> Date: 2016-07-07T16:55:14Z Fix pom.xml merge conflict commit 0d3bd464ae362e97aeb8d1fb5e48beac06dbb0d1 Author: Luciano Resende <lrese...@apache.org> Date: 2016-07-07T18:03:23Z Fix new code to support both scala versions commit 49cc8d6ea631c0b3bc122989307c4868eaf6f162 Author: Lee moon soo <m...@apache.org> Date: 2016-07-09T18:07:28Z initialize imain commit a7442049a78090b960cdd796673e924c07d6b72e Author: Lee moon soo <m...@apache.org> Date: 2016-07-09T18:50:21Z -Dscala-2.11 on mvn verify commit bfd7340fe378c37894de0b1e957231c9f9bde1ce Author: Lee moon soo <m...@apache.org> Date: 2016-07-10T02:36:06Z Initial spark 2.0 support commit 482864c61ea8ffd5ae6dedc2aeee9cd16fa250b1 Author: Lee moon soo <m...@apache.org> Date: 2016-07-10T15:04:07Z comment out spark-streaming-twitter commit b3d067087a3032b9669afda98475ddfce729cbce Author: Lee moon soo <m...@apache.org> Date: 2016-07-10T15:04:58Z generate spark2.0 compatible classname commit 5243feaef814140a92560d88ebadce82b63f0a89 Author: Lee moon soo <m...@apache.org> Date: 2016-07-10T15:05:33Z Remove unnecessary refernece to HiveContext commit 1bd101afc1f46f5ee908dec4bd5f2a07ab063958 Author: Lee moon soo <m...@apache.org> Date: 2016-07-10T15:49:36Z spark2 in PySparkInterpreter commit bd4a5fe25eec2875dc483da88276986bb6031aec Author: Lee moon soo <m...@apache.org> Date: 2016-07-10T16:45:43Z get SQLContext correctly commit 5bd564005e84ef76491d84dbbc007dc0193fe2c6 Author: Lee moon soo <m...@apache.org> Date: 2016-07-10T17:02:28Z dd spark 2 test profile in travis commit 55e9b7fbc5e3adde0d36927e7a8898833575532b Author: Lee moon soo <m...@apache.org> Date: 2016-07-10T21:11:43Z SparkSession.wrapped() fallbacks to SparkSession.sqlContext(). Use createDataFrame instead of applySchema ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---