[ https://issues.apache.org/jira/browse/FLINK-8909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16405980#comment-16405980 ]
Chesnay Schepler commented on FLINK-8909: ----------------------------------------- Have you configured `python.dc.tmp.dir` ? This must be a directory in a distributed filesystem like HDFS. > pyflink.sh not working with yarn > -------------------------------- > > Key: FLINK-8909 > URL: https://issues.apache.org/jira/browse/FLINK-8909 > Project: Flink > Issue Type: Bug > Affects Versions: 1.4.2 > Reporter: Hitesh Tiwari > Priority: Blocker > > Hi, > i want to run the python application from pyflink.sh with yarn-cluster mode. > Added "-m yarn-cluster -yn 1 " in pyflink.sh. so my updated pyflink.sh is > executing below coomand: > "$FLINK_BIN_DIR"/flink run -m yarn-cluster -yn 1 -v > "$FLINK_ROOT_DIR"/lib/flink-python*.jar "$@" > Running pyflink.sh: > ./bin/pyflink.sh /opt/pnda/hitesh/flink-1.4.2/examples/python/WordCount.py > While running getting below Error: > java.lang.Exception: The user defined 'open()' method caused an exception: An > error occurred while copying the file. > at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:485) > at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:355) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.RuntimeException: An error occurred while copying the > file. > at > org.apache.flink.api.common.cache.DistributedCache.getFile(DistributedCache.java:78) > at > org.apache.flink.python.api.streaming.data.PythonStreamer.startPython(PythonStreamer.java:114) > at > org.apache.flink.python.api.streaming.data.PythonStreamer.open(PythonStreamer.java:100) > at > org.apache.flink.python.api.functions.PythonMapPartition.open(PythonMapPartition.java:53) > at > org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36) > at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:481) > ... 3 more > Caused by: java.io.FileNotFoundException: File /tmp/flink_dc does not exist > or the user running Flink ('yarn') has insufficient permissions to access it. > at > org.apache.flink.core.fs.local.LocalFileSystem.getFileStatus(LocalFileSystem.java:115) > at org.apache.flink.runtime.filecache.FileCache.copy(FileCache.java:241) > at > org.apache.flink.runtime.filecache.FileCache$CopyProcess.call(FileCache.java:318) > at > org.apache.flink.runtime.filecache.FileCache$CopyProcess.call(FileCache.java:302) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > ... 1 more > 03/09/2018 11:20:23 Job execution switched to status FAILING. > java.lang.Exception: The user defined 'open()' method caused an exception: > An error occurred while copying the file. > at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:485) > at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:355) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.RuntimeException: An error occurred while copying the > file. > at > org.apache.flink.api.common.cache.DistributedCache.getFile(DistributedCache.java:78) > at > org.apache.flink.python.api.streaming.data.PythonStreamer.startPython(PythonStreamer.java:114) > at > org.apache.flink.python.api.streaming.data.PythonStreamer.open(PythonStreamer.java:100) > at > org.apache.flink.python.api.functions.PythonMapPartition.open(PythonMapPartition.java:53) > at > org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36) > at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:481) > ... 3 more > Caused by: java.io.FileNotFoundException: File /tmp/flink_dc does not exist > or the user running Flink ('yarn') has insufficient permissions to access it. > at > org.apache.flink.core.fs.local.LocalFileSystem.getFileStatus(LocalFileSystem.java:115) > at org.apache.flink.runtime.filecache.FileCache.copy(FileCache.java:241) > at > org.apache.flink.runtime.filecache.FileCache$CopyProcess.call(FileCache.java:318) > at > org.apache.flink.runtime.filecache.FileCache$CopyProcess.call(FileCache.java:302) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > ... 1 more > 03/09/2018 11:20:23 GroupReduce (PythonGroupReducePreStep)(1/1) switched to > CANCELED > 03/09/2018 11:20:23 CHAIN MapPartition (PythonGroupReduce -> PythonMap) -> > Map (PrintSinkPreStep)(1/1) switched to CANCELED > 03/09/2018 11:20:23 DataSink (Print to System.out)(1/1) switched to CANCELED > 03/09/2018 11:20:23 Job execution switched to status FAILED. -- This message was sent by Atlassian JIRA (v7.6.3#76005)