[GitHub] spark pull request #19593: [WIP][SPARK-22374][SQL][2.2] closeAllForUGI is re...

[dongjoon-hyun]

GitHub user dongjoon-hyun opened a pull request:

    https://github.com/apache/spark/pull/19593

    [WIP][SPARK-22374][SQL][2.2] closeAllForUGI is required after using 
loginUserFromKeytab

    ## What changes were proposed in this pull request?
    
    In a secure cluster, FileSystem.Cache grows indefinitely when we use
    1. `spark.yarn.principal` and `spark.yarn.keytab` is used.
    2. Spark Thrift Server run with `hive.server2.enable.doAs=false`.
    
    For example, with 6GB (-Xmx6144m) options, `HiveConf` consumes 4GB inside 
FileSystem.CACHE and OOM occurs. This PR aims to clear up `FileSystem.Cache` by 
using `closeAllForUGI`.
    
    
![2](https://user-images.githubusercontent.com/9700541/32129492-b09f6cbc-bb3c-11e7-8f75-fe027d626816.png)
    
    
![3](https://user-images.githubusercontent.com/9700541/32129494-bafa2cce-bb3c-11e7-80f8-5b03e3471109.png)
    
    ## How was this patch tested?
    
    N/A

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/dongjoon-hyun/spark SPARK-22374

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/19593.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #19593
    
----
commit ea3943e977d378609c9440c049bdb86a308ba428
Author: Dongjoon Hyun <dongj...@apache.org>
Date:   2017-10-27T03:23:13Z

    [SPARK-22374][SQL][2.2] closeAllForUGI is required after using 
loginUserFromKeytab

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org