[GitHub] zeppelin issue #980: [ZEPPELIN-871] [WIP] spark 2.0 interpreter on scala 2.1...
Github user echarles commented on the issue: https://github.com/apache/zeppelin/pull/980 Closing this. Spark 2.0 is implemented with #1195 Optionally, a new PR can be opened to port the Spark interpreter from Java to Scala if interest. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #980: [ZEPPELIN-871] [WIP] spark 2.0 interpreter on scala 2.1...
Github user rnirmal commented on the issue: https://github.com/apache/zeppelin/pull/980 Sounds good, I'll keep a lookout for it to land. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #980: [ZEPPELIN-871] [WIP] spark 2.0 interpreter on scala 2.1...
Github user rnirmal commented on the issue: https://github.com/apache/zeppelin/pull/980 @echarles @Leemoonsoo do you know where the direction lies for Spark 2.0 with scala 2.11 support.. i.e as separate items or continued effort on this PR? Since Spark 2.0 is going to be release in the next week or two, would be great to see this made available on Zeppelin as a quick follow on. I can help with any testing if needed, not as familiar with Zeppelin code yet to help with reviews. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #980: [ZEPPELIN-871] [WIP] spark 2.0 interpreter on scala 2.1...
Github user echarles commented on the issue: https://github.com/apache/zeppelin/pull/980 - If you ship in the same folder scala libs from different versions, it will at runtime - You don't have to rebuilt, but you have to arrange classpath is built with scala libs of the same version - Gui, cli... Makes sense. Is t is what @felixcheung was refereing to? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #980: [ZEPPELIN-871] [WIP] spark 2.0 interpreter on scala 2.1...
Github user Leemoonsoo commented on the issue: https://github.com/apache/zeppelin/pull/980 @echarles > I was just saying that a single binary will not fit all various users need (multiple combination of interpeters and versions). If single SparkInterpreter binary is compatible with various spark versions and scala versions, and single FlinkInterpreter binary is compatible with various flink versions and scala versions, user can choose any combination without rebuild. isn't it? > #629 is merged. What remains to get ZEPPELIN-598 operational? After #908 is merged, we'll need proper commandline or GUI to access this feature. And publish all interpreters to maven repository and remove interpreter dependencies from binary package. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #980: [ZEPPELIN-871] [WIP] spark 2.0 interpreter on scala 2.1...
Github user echarles commented on the issue: https://github.com/apache/zeppelin/pull/980 @Leemoonsoo - I was just saying that a single binary will not fit all various users need (multiple combination of interpeters and versions). - #629 is merged. What remains to get ZEPPELIN-598 operational? @felixcheung Is there already a jira for the proposal you are thinkg to? I feel we have nearly everything open, but maybe we see an umbrella? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #980: [ZEPPELIN-871] [WIP] spark 2.0 interpreter on scala 2.1...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/980 +1 on all of that. I think we could work together on a proposal to see what is the best way to go forward. I think we are way overdue on separating "programming languages" (and versions) and "frameworks" (and versions). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #980: [ZEPPELIN-871] [WIP] spark 2.0 interpreter on scala 2.1...
Github user Leemoonsoo commented on the issue: https://github.com/apache/zeppelin/pull/980 Why not single binary works for spark-2-scala-2.11 and flink-0.9-scala-2.12? I'm also not a fan of reflection. But if i have to choose either complexity of code or complexity usage, then i would like choose keeping usage simple, even though code becomes more complex. I believe this is whole point of making a software. And i 100% agree. Once we have https://issues.apache.org/jira/browse/ZEPPELIN-598 and proper UI / command line for list/download interpreter from maven repository, then i think things will become much more easier and flexible. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #980: [ZEPPELIN-871] [WIP] spark 2.0 interpreter on scala 2.1...
Github user Leemoonsoo commented on the issue: https://github.com/apache/zeppelin/pull/980 Thanks for elaborate on idea about 2 lines approach. I think there're always pros and cons. If we go to 2 lines approach, code will be cleaner, simpler and more easier to read. But this means Zeppelin generally enables create binary per interpreter dependency because we'll have the same policy for all interpreters. In the end, i'm afraid we end up with making bunch of binary packages like - zeppelin with spark 1.x on scala 2.10 - zeppelin with spark 1.x on scala 2.11 - zeppelin with spark 2.0 on scala 2.10 - zeppelin with spark 2.0 on scala 2.11 - zeppelin with spark 2.1 on scala 2.10 - zeppelin with spark 2.1 on scala 2.10 - zeppelin with python3.x - zeppelin with python2.x - zeppelin with hive1 - zeppelin with hive2 - . In the perspective of user, it can not be an improvement. A single Zeppelin binary used to work with everything, but now user have to understand difference between of packages and able to select them before uses. I think code complexity will not endlessly increase if we cut the spark support for last x releases. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #980: [ZEPPELIN-871] [WIP] spark 2.0 interpreter on scala 2.1...
Github user echarles commented on the issue: https://github.com/apache/zeppelin/pull/980 Dealing with mixed scala 2.10/2.11 and spark 1.x/2.x with a same implementation is always possible but drives to code plenty of method invocation (see e.g. https://github.com/lresende/incubator-zeppelin/pull/1/files#diff-dbda0c4083ad9c59ff05f0273b5e760fR216). At the end, you have an implementation which a succession of if (...) then invokeMethod... On the other hand, having multiple implementations drives to maintenance overhead and a risk of divergence. In this particular case, I was thinking that having 2 lines was something worth to discuss: 1.- Spark 1.x on scala 2.10 2.- Spark 2.x on scala 2.11 I also don't see why we should still support the DepInterpreter in future developments (normally, deps should be configured via the interperter settings). Also, the 2 lines approach would make easier other evolutions such as for example having a ZeppelinContext available for all interpreters. Also, we have kind-off already more than one spark impementation: The Livy one has its own implementation and features. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #980: [ZEPPELIN-871] [WIP] spark 2.0 interpreter on scala 2.1...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/980 From what I see, we should still be able to use most of the existing code. Could you elaborate more on how bad it would be to support Spark 1.x and 2.x in the same interpreter code? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #980: [ZEPPELIN-871] [WIP] spark 2.0 interpreter on scala 2.1...
Github user Leemoonsoo commented on the issue: https://github.com/apache/zeppelin/pull/980 I'm trying to combine 2.10 and 2.11 into one implementation in lresende#1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
