[incubator-livy] branch master updated: [LIVY-622][LIVY-623][LIVY-624][LIVY-625][THRIFT][FOLLOWUP] Use ResultSet in catalog operations

2019-08-30 Thread mgaido 
This is an automated email from the ASF dual-hosted git repository.

mgaido pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git


The following commit(s) were added to refs/heads/master by this push:
 new d0d8028  [LIVY-622][LIVY-623][LIVY-624][LIVY-625][THRIFT][FOLLOWUP] 
Use ResultSet in catalog operations
d0d8028 is described below

commit d0d8028657c7314ad031f51c0a564a8786213187
Author: Marco Gaido 
AuthorDate: Fri Aug 30 14:06:08 2019 +0200

[LIVY-622][LIVY-623][LIVY-624][LIVY-625][THRIFT][FOLLOWUP] Use ResultSet in 
catalog operations

## What changes were proposed in this pull request?

This is a followup of #194 which addresses all the remaining concerns. The 
main changes are:

 - reverting the introduction of a state specific for catalog operations;
 - usage of `ResultSet` to send over the wire the data for catalog 
operations too.

## How was this patch tested?

existing modified UTs

Author: Marco Gaido 

Closes #217 from mgaido91/LIVY-622_followup.
---
 .../livy/thriftserver/LivyOperationManager.scala   | 18 ++---
 .../operation/GetColumnsOperation.scala| 20 +++---
 .../operation/GetFunctionsOperation.scala  | 35 ++---
 .../operation/GetSchemasOperation.scala| 12 ++--
 .../operation/GetTablesOperation.scala | 16 ++---
 .../thriftserver/operation/MetadataOperation.scala |  3 -
 .../operation/SparkCatalogOperation.scala  | 51 ++
 .../session/CleanupCatalogResultJob.java   | 37 --
 .../session/FetchCatalogResultJob.java | 51 --
 .../livy/thriftserver/session/GetColumnsJob.java   | 31 
 .../livy/thriftserver/session/GetFunctionsJob.java | 66 ++---
 .../livy/thriftserver/session/GetSchemasJob.java   | 24 ---
 .../livy/thriftserver/session/GetTablesJob.java| 19 ++---
 .../livy/thriftserver/session/SparkCatalogJob.java | 64 +++--
 .../livy/thriftserver/session/SparkUtils.java  | 46 
 .../thriftserver/session/ThriftSessionState.java   | 32 -
 .../thriftserver/session/ThriftSessionTest.java| 82 +-
 17 files changed, 290 insertions(+), 317 deletions(-)

diff --git 
a/thriftserver/server/src/main/scala/org/apache/livy/thriftserver/LivyOperationManager.scala
 
b/thriftserver/server/src/main/scala/org/apache/livy/thriftserver/LivyOperationManager.scala
index 2454185..eb1dd21 100644
--- 
a/thriftserver/server/src/main/scala/org/apache/livy/thriftserver/LivyOperationManager.scala
+++ 
b/thriftserver/server/src/main/scala/org/apache/livy/thriftserver/LivyOperationManager.scala
@@ -196,12 +196,7 @@ class LivyOperationManager(val livyThriftSessionManager: 
LivyThriftSessionManage
   tableTypes: util.List[String]): OperationHandle = {
 executeOperation(sessionHandle, {
   val op = new GetTablesOperation(
-sessionHandle,
-catalogName,
-schemaName,
-tableName,
-tableTypes,
-livyThriftSessionManager)
+sessionHandle, schemaName, tableName, tableTypes, 
livyThriftSessionManager)
   addOperation(op, sessionHandle)
   op
 })
@@ -214,8 +209,8 @@ class LivyOperationManager(val livyThriftSessionManager: 
LivyThriftSessionManage
   schemaName: String,
   functionName: String): OperationHandle = {
 executeOperation(sessionHandle, {
-  val op = new GetFunctionsOperation(sessionHandle, catalogName, 
schemaName, functionName,
-livyThriftSessionManager)
+  val op = new GetFunctionsOperation(
+sessionHandle, schemaName, functionName, livyThriftSessionManager)
   addOperation(op, sessionHandle)
   op
 })
@@ -227,8 +222,7 @@ class LivyOperationManager(val livyThriftSessionManager: 
LivyThriftSessionManage
   catalogName: String,
   schemaName: String): OperationHandle = {
 executeOperation(sessionHandle, {
-  val op = new GetSchemasOperation(sessionHandle, catalogName, schemaName,
-livyThriftSessionManager)
+  val op = new GetSchemasOperation(sessionHandle, schemaName, 
livyThriftSessionManager)
   addOperation(op, sessionHandle)
   op
 })
@@ -242,8 +236,8 @@ class LivyOperationManager(val livyThriftSessionManager: 
LivyThriftSessionManage
   tableName: String,
   columnName: String): OperationHandle = {
 executeOperation(sessionHandle, {
-  val op = new GetColumnsOperation(sessionHandle, catalogName, schemaName, 
tableName,
-columnName, livyThriftSessionManager)
+  val op = new GetColumnsOperation(
+sessionHandle, schemaName, tableName, columnName, 
livyThriftSessionManager)
   addOperation(op, sessionHandle)
   op
 })
diff --git 
a/thriftserver/server/src/main/scala/org/apache/livy/thriftserver/operation/GetColumnsOperation.scala
 
b/thriftserver/server/src/main/scala/org/apache/livy/thriftserver/operation/Ge

[incubator-livy] branch master updated (a90f4fa -> 4521ef9)

2019-08-30 Thread jshao 
This is an automated email from the ASF dual-hosted git repository.

jshao pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-livy.git.


from a90f4fa  [LIVY-642] Fix a rare status happened in yarn cause SparkApp 
change into error state
 add 4521ef9  [LIVY-586] Fix batch state from starting to dead when startup 
fail

No new revisions were added by this update.

Summary of changes:
 .../scala/org/apache/livy/utils/SparkYarnApp.scala | 10 -
 .../org/apache/livy/utils/SparkYarnAppSpec.scala   | 24 ++
 2 files changed, 33 insertions(+), 1 deletion(-)