[GitHub] [incubator-livy] coheigea commented on pull request #299: LIVY-776 - Update Maven Shade Plugin to 3.2.1
coheigea commented on pull request #299: URL: https://github.com/apache/incubator-livy/pull/299#issuecomment-648887533 @tprelle It's up to the maintainers of the project, either it could all be done in one go (as in your PR), or maybe they might prefer to just update Jackson separately. If your PR is merged first, then I'll just close this PR. Either way it would be good from my POV to get us updated to Jackson 2.10.x ;-) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] tprelle commented on pull request #299: LIVY-776 - Update Maven Shade Plugin to 3.2.1
tprelle commented on pull request #299: URL: https://github.com/apache/incubator-livy/pull/299#issuecomment-648874686 Hi @coheigea I faced the same issue (shaded plugin and jackson version) on https://github.com/apache/incubator-livy/pull/289 for scala 2.12 and spark 3 support. I need also to change this. I do not know how deal with multiple PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] tprelle commented on a change in pull request #289: [LIVY-756] Add Support Spark 3
tprelle commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444891736
##
File path: pom.xml
##
@@ -86,24 +87,25 @@
1.9
4.5.3
4.4.4
-2.9.9
+2.10.1
3.1.0
9.3.24.v20180605
3.2.11
4.11
0.9.3
-2.22
+4.0.2
3.1.0
-1.9.5
+1.10.19
4.0.37.Final
${netty.spark-2.11.version}
UTF-8
0.10.7
2.11.12
+2.12.10
2.11
${scala-2.11.version}
-2.2.4
-2.3.0
+3.0.8
+2.6.5
Review comment:
I try to limit the number of version change, but because of scala 2.12
like @andrasbeni says I need to use scalatest 3.0.8 who are some breaking
change like org.scalatest.Suite =>org.scalatest.TestSuite, so to do not copy
the class I need to change also the global version of scalatest. So I need to
change the version of scalatra because of the incompatibility between scalatest
3.0.8 and scalatra-scalatest 2.3.0.
As I was working and easier to move forward with these upgrade of version I
choose to submit like this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-livy] tprelle commented on pull request #289: [LIVY-756] Add Support Spark 3
tprelle commented on pull request #289: URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-648813898 > I just did a quick review about the overall implementation. Seems like we can only support one Scala version for one build here in this PR. But Livy could actually support different versions of Scala in one build (we used to support Scala 2.10 and 2.11 for one build). So I think we should have both Scala 2.12 and 2.11 support, and let Livy to choose jars automatically based on the Spark. I like the idea, I try it, but I was not able to make it work. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] tprelle commented on a change in pull request #289: [LIVY-756] Add Support Spark 3
tprelle commented on a change in pull request #289: URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444880828 ## File path: client-http/pom.xml ## @@ -109,58 +109,6 @@ - Review comment: It was a mistake, i will put it back This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] tprelle commented on a change in pull request #289: [LIVY-756] Add Support Spark 3
tprelle commented on a change in pull request #289: URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444876495 ## File path: api/src/main/java/org/apache/livy/JobContext.java ## @@ -41,12 +41,12 @@ /** * @return The shared SQLContext instance. */ - SQLContext sqlctx(); + SQLContext sqlctx() throws Exception ; Review comment: It was before https://issues.apache.org/jira/browse/SPARK-31088, when they put back HiveContext class back to spark. When I revert the change I forgot to revert also the method signature. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on pull request #289: [WIP] [LIVY-756] Add Support Spark 3
jerryshao commented on pull request #289: URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-648718274 I'm doing some local improvements based on @tprelle 's current PR, I can submit a PR when everything is ready. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3
jerryshao commented on a change in pull request #289: URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444774680 ## File path: api/src/main/java/org/apache/livy/JobContext.java ## @@ -41,12 +41,12 @@ /** * @return The shared SQLContext instance. */ - SQLContext sqlctx(); + SQLContext sqlctx() throws Exception ; Review comment: Yes, I have the same concern about changing the signature of this interface. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3
jerryshao commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444770427
##
File path: pom.xml
##
@@ -86,24 +87,25 @@
1.9
4.5.3
4.4.4
-2.9.9
+2.10.1
3.1.0
9.3.24.v20180605
3.2.11
4.11
0.9.3
-2.22
+4.0.2
3.1.0
-1.9.5
+1.10.19
4.0.37.Final
${netty.spark-2.11.version}
UTF-8
0.10.7
2.11.12
+2.12.10
2.11
${scala-2.11.version}
-2.2.4
-2.3.0
+3.0.8
+2.6.5
Review comment:
Scalatra is only used by Livy Server, we don't have to upgrade Livy
Server's Scala version to 2.12, we could leave the Livy Server Scala upgrade to
another JIRA.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3
jerryshao commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444770427
##
File path: pom.xml
##
@@ -86,24 +87,25 @@
1.9
4.5.3
4.4.4
-2.9.9
+2.10.1
3.1.0
9.3.24.v20180605
3.2.11
4.11
0.9.3
-2.22
+4.0.2
3.1.0
-1.9.5
+1.10.19
4.0.37.Final
${netty.spark-2.11.version}
UTF-8
0.10.7
2.11.12
+2.12.10
2.11
${scala-2.11.version}
-2.2.4
-2.3.0
+3.0.8
+2.6.5
Review comment:
Scalatra is only used by Livy Server, we don't have to upgrade Livy
Server's Scala version to 2.12, we could leave the Livy Server Scala support to
another JIRA.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-livy] andrasbeni commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3
andrasbeni commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444717000
##
File path: pom.xml
##
@@ -86,24 +87,25 @@
1.9
4.5.3
4.4.4
-2.9.9
+2.10.1
3.1.0
9.3.24.v20180605
3.2.11
4.11
0.9.3
-2.22
+4.0.2
3.1.0
-1.9.5
+1.10.19
4.0.37.Final
${netty.spark-2.11.version}
UTF-8
0.10.7
2.11.12
+2.12.10
2.11
${scala-2.11.version}
-2.2.4
-2.3.0
+3.0.8
+2.6.5
Review comment:
Scalatra added scala 2.12 support in version 2.5.0 and scalatest in
3.0.0. So the version currently in use are incompatible with Spark 3.0's Scala
2.12.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-livy] andrasbeni commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3
andrasbeni commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444719602
##
File path: api/src/main/java/org/apache/livy/JobContext.java
##
@@ -41,12 +41,12 @@
/**
* @return The shared SQLContext instance.
*/
- SQLContext sqlctx();
+ SQLContext sqlctx() throws Exception ;
Review comment:
I believe adding throws clauses to these methods could break Java client
applications. I suggest to wrap exceptions that are not RuntimeException in a
RuntimeException instead.
##
File path: pom.xml
##
@@ -86,24 +87,25 @@
1.9
4.5.3
4.4.4
-2.9.9
+2.10.1
3.1.0
9.3.24.v20180605
3.2.11
4.11
0.9.3
-2.22
+4.0.2
3.1.0
-1.9.5
+1.10.19
4.0.37.Final
${netty.spark-2.11.version}
UTF-8
0.10.7
2.11.12
+2.12.10
2.11
${scala-2.11.version}
-2.2.4
-2.3.0
+3.0.8
+2.6.5
Review comment:
Scalatra added scala 2.12 support in version 2.5.0 and scalatest in
3.0.0. So the version currently in use are incompatibel with Spark 3.0's Scala
2.12.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
