[GitHub] [incubator-livy] jerryshao commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3

2020-06-24 Thread GitBox 


jerryshao commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444774680



##
File path: api/src/main/java/org/apache/livy/JobContext.java
##
@@ -41,12 +41,12 @@
   /**
* @return The shared SQLContext instance.
*/
-  SQLContext sqlctx();
+  SQLContext sqlctx() throws Exception ;

Review comment:
   Yes, I have the same concern about changing the signature of this 
interface.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-livy] jerryshao commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3

2020-06-24 Thread GitBox 


jerryshao commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444770427



##
File path: pom.xml
##
@@ -86,24 +87,25 @@
 1.9
 4.5.3
 4.4.4
-2.9.9
+2.10.1
 3.1.0
 9.3.24.v20180605
 3.2.11
 4.11
 0.9.3
-2.22
+4.0.2
 3.1.0
-1.9.5
+1.10.19
 4.0.37.Final
 ${netty.spark-2.11.version}
 UTF-8
 0.10.7
 2.11.12
+2.12.10
 2.11
 ${scala-2.11.version}
-2.2.4
-2.3.0
+3.0.8
+2.6.5

Review comment:
   Scalatra is only used by Livy Server, we don't have to upgrade Livy 
Server's Scala version to 2.12, we could leave the Livy Server Scala upgrade to 
another JIRA.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-livy] jerryshao commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3

2020-06-24 Thread GitBox 


jerryshao commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444770427



##
File path: pom.xml
##
@@ -86,24 +87,25 @@
 1.9
 4.5.3
 4.4.4
-2.9.9
+2.10.1
 3.1.0
 9.3.24.v20180605
 3.2.11
 4.11
 0.9.3
-2.22
+4.0.2
 3.1.0
-1.9.5
+1.10.19
 4.0.37.Final
 ${netty.spark-2.11.version}
 UTF-8
 0.10.7
 2.11.12
+2.12.10
 2.11
 ${scala-2.11.version}
-2.2.4
-2.3.0
+3.0.8
+2.6.5

Review comment:
   Scalatra is only used by Livy Server, we don't have to upgrade Livy 
Server's Scala version to 2.12, we could leave the Livy Server Scala support to 
another JIRA.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-livy] jerryshao commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3

2020-06-23 Thread GitBox 


jerryshao commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444628517



##
File path: pom.xml
##
@@ -86,24 +87,25 @@
 1.9
 4.5.3
 4.4.4
-2.9.9
+2.10.1
 3.1.0
 9.3.24.v20180605
 3.2.11
 4.11
 0.9.3
-2.22
+4.0.2
 3.1.0
-1.9.5
+1.10.19
 4.0.37.Final
 ${netty.spark-2.11.version}
 UTF-8
 0.10.7
 2.11.12
+2.12.10
 2.11
 ${scala-2.11.version}
-2.2.4
-2.3.0
+3.0.8
+2.6.5

Review comment:
   Are this artifacts upgrading necessary for Spark 3.0 support?

##
File path: pom.xml
##
@@ -86,24 +87,25 @@
 1.9
 4.5.3
 4.4.4
-2.9.9
+2.10.1
 3.1.0
 9.3.24.v20180605
 3.2.11
 4.11
 0.9.3
-2.22
+4.0.2
 3.1.0
-1.9.5
+1.10.19
 4.0.37.Final
 ${netty.spark-2.11.version}
 UTF-8
 0.10.7
 2.11.12
+2.12.10
 2.11
 ${scala-2.11.version}
-2.2.4
-2.3.0
+3.0.8
+2.6.5

Review comment:
   Are these artifacts upgrading necessary for Spark 3.0 support?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-livy] jerryshao commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3

2020-06-23 Thread GitBox 


jerryshao commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444618359



##
File path: .travis.yml
##
@@ -34,6 +34,12 @@ matrix:
 env: MVN_FLAG='-Pspark-2.4 -Pthriftserver -DskipITs'
   - name: "Spark 2.4 ITs"
 env: MVN_FLAG='-Pspark-2.4 -Pthriftserver -DskipTests'
+  - name: "Spark 3.0 Unit Tests"
+env: MVN_FLAG='-Pthriftserver -Pspark-3.0 -DskipITs'
+  - name: "Spark 3.0 ITs"
+env: 
+  - MVN_FLAG='-Pthriftserver -Pspark-3.0 -DskipTests'
+  - PYSPARK_ROW_FIELD_SORTING_ENABLED=true

Review comment:
   Got it, seems an incompatible change in Spark 3.0





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-livy] jerryshao commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3

2020-06-23 Thread GitBox 


jerryshao commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444614548



##
File path: .travis.yml
##
@@ -34,6 +34,12 @@ matrix:
 env: MVN_FLAG='-Pspark-2.4 -Pthriftserver -DskipITs'
   - name: "Spark 2.4 ITs"
 env: MVN_FLAG='-Pspark-2.4 -Pthriftserver -DskipTests'
+  - name: "Spark 3.0 Unit Tests"
+env: MVN_FLAG='-Pthriftserver -Pspark-3.0 -DskipITs'
+  - name: "Spark 3.0 ITs"
+env: 
+  - MVN_FLAG='-Pthriftserver -Pspark-3.0 -DskipTests'
+  - PYSPARK_ROW_FIELD_SORTING_ENABLED=true

Review comment:
   What is the purpose of setting this?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-livy] jerryshao commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3

2020-06-23 Thread GitBox 


jerryshao commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444613917



##
File path: client-http/pom.xml
##
@@ -109,58 +109,6 @@
 
   
 
-  

Review comment:
   What's the reason to remove this?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org