[GitHub] spark pull request #19593: [WIP][SPARK-22374][SQL][2.2] closeAllForUGI is re...

[dongjoon-hyun]

Github user dongjoon-hyun closed the pull request at:

https://github.com/apache/spark/pull/19593


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19593: [WIP][SPARK-22374][SQL][2.2] closeAllForUGI is re...

[dongjoon-hyun]

GitHub user dongjoon-hyun opened a pull request:

https://github.com/apache/spark/pull/19593

[WIP][SPARK-22374][SQL][2.2] closeAllForUGI is required after using 
loginUserFromKeytab

## What changes were proposed in this pull request?

In a secure cluster, FileSystem.Cache grows indefinitely when we use
1. `spark.yarn.principal` and `spark.yarn.keytab` is used.
2. Spark Thrift Server run with `hive.server2.enable.doAs=false`.

For example, with 6GB (-Xmx6144m) options, `HiveConf` consumes 4GB inside 
FileSystem.CACHE and OOM occurs. This PR aims to clear up `FileSystem.Cache` by 
using `closeAllForUGI`.


![2](https://user-images.githubusercontent.com/9700541/32129492-b09f6cbc-bb3c-11e7-8f75-fe027d626816.png)


![3](https://user-images.githubusercontent.com/9700541/32129494-bafa2cce-bb3c-11e7-80f8-5b03e3471109.png)

## How was this patch tested?

N/A

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dongjoon-hyun/spark SPARK-22374

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/19593.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #19593


commit ea3943e977d378609c9440c049bdb86a308ba428
Author: Dongjoon Hyun 
Date:   2017-10-27T03:23:13Z

[SPARK-22374][SQL][2.2] closeAllForUGI is required after using 
loginUserFromKeytab




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org