[GitHub] incubator-livy pull request #121: [WIP][LIVY-518][BUILD] Support Spark 2.4

2018-10-15 Thread HyukjinKwon 
Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/incubator-livy/pull/121#discussion_r225194172
  
--- Diff: rsc/src/test/java/org/apache/livy/rsc/TestSparkClient.java ---
@@ -271,7 +275,7 @@ public void call(LivyClient client) throws Exception {
 
   @Test
   public void testSparkSQLJob() throws Exception {
-runTest(true, new TestFunction() {
+runTest(true, false, new TestFunction() {
--- End diff --

This also particularly happens when the session that created Hive client is 
restarted with Hive support enabled multiple times(?). This was able to be 
reproduced in the local as well. I am still digging the root cause but so far 
the scope is specified to this test only.


---

[GitHub] incubator-livy pull request #121: [WIP][LIVY-518][BUILD] Support Spark 2.4

2018-10-15 Thread mgaido91 
Github user mgaido91 commented on a diff in the pull request:

https://github.com/apache/incubator-livy/pull/121#discussion_r225189164
  
--- Diff: rsc/src/test/java/org/apache/livy/rsc/TestSparkClient.java ---
@@ -271,7 +275,7 @@ public void call(LivyClient client) throws Exception {
 
   @Test
   public void testSparkSQLJob() throws Exception {
-runTest(true, new TestFunction() {
+runTest(true, false, new TestFunction() {
--- End diff --

> I am not 100% sure why the test becomes failed. The root cause seems that 
multiple Hive clients(?) access to derby and metastore as it creates Hive 
client multiple times(?).

Wouldn't it fail with 2.3 too then? I don't expect any difference among the 
2 versions on that. Am I missing something?


---

[GitHub] incubator-livy pull request #121: [WIP][LIVY-518][BUILD] Support Spark 2.4

2018-10-15 Thread HyukjinKwon 
Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/incubator-livy/pull/121#discussion_r225185417
  
--- Diff: rsc/src/test/java/org/apache/livy/rsc/TestSparkClient.java ---
@@ -271,7 +275,7 @@ public void call(LivyClient client) throws Exception {
 
   @Test
   public void testSparkSQLJob() throws Exception {
-runTest(true, new TestFunction() {
+runTest(true, false, new TestFunction() {
--- End diff --

I am running Travis CI against RC3 
(https://travis-ci.org/HyukjinKwon/incubator-livy/builds/441687251)


---

[GitHub] incubator-livy pull request #121: [WIP][LIVY-518][BUILD] Support Spark 2.4

2018-10-15 Thread mgaido91 
Github user mgaido91 commented on a diff in the pull request:

https://github.com/apache/incubator-livy/pull/121#discussion_r225081934
  
--- Diff: README.md ---
@@ -57,7 +57,7 @@ Required python packages for building Livy:
 To run Livy, you will also need a Spark installation. You can get Spark 
releases at
 https://spark.apache.org/downloads.html.
 
-Livy requires Spark 2.2 or 2.3. You can switch to a different version of 
Spark by setting the
+Livy requires Spark 2.2, 2.3 or 2.4. You can switch to a different version 
of Spark by setting the
--- End diff --

nit: 2.2+?


---