[
https://issues.apache.org/jira/browse/SPARK-44639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-44639:
----------------------------------
Summary: Fix to use Java tmp dir for local RocksDB state storage on Yarn
(was: Add option to use Java tmp dir for RocksDB state store)
> Fix to use Java tmp dir for local RocksDB state storage on Yarn
> ---------------------------------------------------------------
>
> Key: SPARK-44639
> URL: https://issues.apache.org/jira/browse/SPARK-44639
> Project: Spark
> Issue Type: Improvement
> Components: SQL, Structured Streaming
> Affects Versions: 3.4.1
> Reporter: Adam Binford
> Assignee: Adam Binford
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.1.0
>
>
> Currently local RocksDB state is stored in a local directory given by
> Utils.getLocalDir. On yarn this is a directory created inside the root
> application folder such as
> {{/tmp/nm-local-dir/usercache/<user>/appcache/<app_id>/}}
> The problem with this is that if an executor crashes for some reason (like
> OOM) and the shutdown hooks don't get run, this directory will stay around
> forever until the application finishes, which can cause jobs to slowly
> accumulate more and more temporary space until finally the node manager goes
> unhealthy.
> Because this data will only ever be accessed by the executor that created
> this directory, it would make sense to store the data inside the container
> folder, which will always get cleaned up by the node manager when that yarn
> container gets cleaned up. Yarn sets the `java.io.tmpdir` to a path inside
> this directory, such as
> {{/tmp/nm-local-dir/usercache/<user>/appcache/<app_id>/<container_id>/tmp/}}
> I'm not sure the behavior for other resource managers, so this could be an
> opt-in config that can be specified.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
