[spark] branch branch-2.4 updated: [SPARK-26751][SQL] Fix memory leak when statement run in background and throw exception which is not HiveSQLException

[srowen]

This is an automated email from the ASF dual-hosted git repository.

srowen pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-2.4 by this push:
     new 3d4aa5b  [SPARK-26751][SQL] Fix memory leak when statement run in 
background and throw exception which is not HiveSQLException
3d4aa5b is described below

commit 3d4aa5bd798a793a10c5d769a31f97242e78ce15
Author: zhoukang <zhoukang199...@gmail.com>
AuthorDate: Sun Feb 3 08:45:57 2019 -0600

    [SPARK-26751][SQL] Fix memory leak when statement run in background and 
throw exception which is not HiveSQLException
    
    ## What changes were proposed in this pull request?
    When we run in background and we get exception which is not 
HiveSQLException,
    we may encounter memory leak since handleToOperation will not removed 
correctly.
    The reason is below:
    1. When calling operation.run() in HiveSessionImpl#executeStatementInternal 
we throw an exception which is not HiveSQLException
    2. Then the opHandle generated by SparkSQLOperationManager will not be 
added into opHandleSet of HiveSessionImpl , and 
operationManager.closeOperation(opHandle) will not be called
    3. When we close the session we will also call 
operationManager.closeOperation(opHandle),since we did not add this opHandle 
into the opHandleSet.
    
    For the reasons above,the opHandled will always in 
SparkSQLOperationManager#handleToOperation,which will cause memory leak.
    More details and a case has attached on 
https://issues.apache.org/jira/browse/SPARK-26751
    This patch will always throw HiveSQLException when running in background
    
    ## How was this patch tested?
    Exist UT
    
    Closes #23673 from caneGuy/zhoukang/fix-hivesessionimpl-leak.
    
    Authored-by: zhoukang <zhoukang199...@gmail.com>
    Signed-off-by: Sean Owen <sean.o...@databricks.com>
    (cherry picked from commit 255faaf3436e1f41838062ed460f801bb0be40ec)
    Signed-off-by: Sean Owen <sean.o...@databricks.com>
---
 .../spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala    | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala
 
b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala
index 3cfc81b..fd17f50 100644
--- 
a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala
+++ 
b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala
@@ -204,7 +204,7 @@ private[hive] class SparkExecuteStatementOperation(
         case NonFatal(e) =>
           logError(s"Error executing query in background", e)
           setState(OperationState.ERROR)
-          throw e
+          throw new HiveSQLException(e)
       }
     }
   }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org