[GitHub] incubator-livy pull request #121: [WIP][LIVY-518][BUILD] Support Spark 2.4

2018-10-15 Thread HyukjinKwon 
Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/incubator-livy/pull/121#discussion_r225194172
  
--- Diff: rsc/src/test/java/org/apache/livy/rsc/TestSparkClient.java ---
@@ -271,7 +275,7 @@ public void call(LivyClient client) throws Exception {
 
   @Test
   public void testSparkSQLJob() throws Exception {
-runTest(true, new TestFunction() {
+runTest(true, false, new TestFunction() {
--- End diff --

This also particularly happens when the session that created Hive client is 
restarted with Hive support enabled multiple times(?). This was able to be 
reproduced in the local as well. I am still digging the root cause but so far 
the scope is specified to this test only.


---

[GitHub] incubator-livy pull request #121: [WIP][LIVY-518][BUILD] Support Spark 2.4

2018-10-15 Thread HyukjinKwon 
Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/incubator-livy/pull/121#discussion_r225191745
  
--- Diff: rsc/src/test/java/org/apache/livy/rsc/TestSparkClient.java ---
@@ -271,7 +275,7 @@ public void call(LivyClient client) throws Exception {
 
   @Test
   public void testSparkSQLJob() throws Exception {
-runTest(true, new TestFunction() {
+runTest(true, false, new TestFunction() {
--- End diff --

I am not very sure on that but that's what's going on here in 2.4.0 - I 
doubly checked in my local and Travis CI.


---

[GitHub] incubator-livy pull request #121: [WIP][LIVY-518][BUILD] Support Spark 2.4

2018-10-15 Thread mgaido91 
Github user mgaido91 commented on a diff in the pull request:

https://github.com/apache/incubator-livy/pull/121#discussion_r225189164
  
--- Diff: rsc/src/test/java/org/apache/livy/rsc/TestSparkClient.java ---
@@ -271,7 +275,7 @@ public void call(LivyClient client) throws Exception {
 
   @Test
   public void testSparkSQLJob() throws Exception {
-runTest(true, new TestFunction() {
+runTest(true, false, new TestFunction() {
--- End diff --

> I am not 100% sure why the test becomes failed. The root cause seems that 
multiple Hive clients(?) access to derby and metastore as it creates Hive 
client multiple times(?).

Wouldn't it fail with 2.3 too then? I don't expect any difference among the 
2 versions on that. Am I missing something?


---

[GitHub] incubator-livy pull request #121: [WIP][LIVY-518][BUILD] Support Spark 2.4

2018-10-15 Thread HyukjinKwon 
Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/incubator-livy/pull/121#discussion_r225185417
  
--- Diff: rsc/src/test/java/org/apache/livy/rsc/TestSparkClient.java ---
@@ -271,7 +275,7 @@ public void call(LivyClient client) throws Exception {
 
   @Test
   public void testSparkSQLJob() throws Exception {
-runTest(true, new TestFunction() {
+runTest(true, false, new TestFunction() {
--- End diff --

I am running Travis CI against RC3 
(https://travis-ci.org/HyukjinKwon/incubator-livy/builds/441687251)


---

[GitHub] incubator-livy pull request #121: [WIP][LIVY-518][BUILD] Support Spark 2.4

2018-10-15 Thread HyukjinKwon 
Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/incubator-livy/pull/121#discussion_r225185258
  
--- Diff: rsc/src/test/java/org/apache/livy/rsc/TestSparkClient.java ---
@@ -271,7 +275,7 @@ public void call(LivyClient client) throws Exception {
 
   @Test
   public void testSparkSQLJob() throws Exception {
-runTest(true, new TestFunction() {
+runTest(true, false, new TestFunction() {
--- End diff --

I am not 100% sure why the test becomes failed. The root cause seems that 
multiple Hive clients(?) access to derby and metastore as it creates Hive 
client multiple times(?).

I spent a whole day and failed to find why and just decided to fix the test 
since it looks the problem is specific to the test scope so far.

So, I fixed the test `testSparkSQLJob` to use Spark only and `testHiveJob` 
to use Hive support. This reminds of me that the current issue at SparkR where 
the tests can't start and stop multiple Hive support sessions (it ended up with 
one hive support session),




---

[GitHub] incubator-livy pull request #121: [WIP][LIVY-518][BUILD] Support Spark 2.4

2018-10-15 Thread mgaido91 
Github user mgaido91 commented on a diff in the pull request:

https://github.com/apache/incubator-livy/pull/121#discussion_r225081934
  
--- Diff: README.md ---
@@ -57,7 +57,7 @@ Required python packages for building Livy:
 To run Livy, you will also need a Spark installation. You can get Spark 
releases at
 https://spark.apache.org/downloads.html.
 
-Livy requires Spark 2.2 or 2.3. You can switch to a different version of 
Spark by setting the
+Livy requires Spark 2.2, 2.3 or 2.4. You can switch to a different version 
of Spark by setting the
--- End diff --

nit: 2.2+?


---

[GitHub] incubator-livy pull request #121: [WIP][LIVY-518][BUILD] Support Spark 2.4

2018-10-14 Thread HyukjinKwon 
GitHub user HyukjinKwon opened a pull request:

https://github.com/apache/incubator-livy/pull/121

[WIP][LIVY-518][BUILD] Support Spark 2.4

## What changes were proposed in this pull request?

This PR proposes to support Spark 2.4 in Livy.

## How was this patch tested?

Unit tests and manual tests.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HyukjinKwon/incubator-livy spark24-support

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-livy/pull/121.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #121


commit 7393630fed6b5f95ef053a1f31bf7d233dd6c6fe
Author: hyukjinkwon 
Date:   2018-10-15T03:08:18Z

[LIVY-518][BUILD] Support Spark 2.4




---