Hi,
My jobs create tmp files like so:
java.nio.file.Path tmpFilePath =
java.nio.file.Files.createTempFile("tmpFile", "txt");
They currently appear in /tmp/, but I want them somewhere else, say /my/tmp/
.
The Flink on YARN docs say:
Flink on YARN will overwrite the following configuration parameters
jobmanager.rpc.address (because the JobManager is always allocated at
different machines), taskmanager.tmp.dirs (we are using the tmp directories
given by YARN) and parallelism.default if the number of slots has been
specified.
How would I specify a different tmp directory for a job without modifying
my YARN tmp directories?
I tried the taskmanager.tmp.dirs property in conf/flink-conf.yaml anyway,
that failed.
I appended -Djava.io.tmpdir=/my/tmp/ to JVM_ARGS and all three variations
of DEFAULT_ENV_JAVA_OPTS in bin/config.sh, that failed.
I passed -Djava.io.tmpdir=/my/tmp/ and variations as arguments to .
/bin/yarn-session.sh and ./bin/flink run et cetera, that failed.
Odd observation:
The hadoop.tmp.dir property is set in my core-site.xml to /some/other/tmp/,
yet Flink writes to /tmp/. My yarn-site.xml specifies no tmp.
Side note:
My Flink job is a Beam pipeline. I doubt that's relevant, but let me know
if it is.
Thanks,
Chris
