[
https://issues.apache.org/jira/browse/AIRFLOW-6124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sajid Sajid reassigned AIRFLOW-6124:
------------------------------------
Assignee: Sajid Sajid
> tmp folder gets deleted, shell-init: error retrieving current directory
> -----------------------------------------------------------------------
>
> Key: AIRFLOW-6124
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6124
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG
> Affects Versions: 1.10.6
> Reporter: Ravi Kaushik
> Assignee: Sajid Sajid
> Priority: Critical
> Attachments: Error.png
>
>
> My dag has three SparkSubmitOperators and throws this error occasionally
> after executing one task. If I trigger the job manually by ignoring all
> dependencies, it works well. Most likely /tmp folder is getting deleted and
> it cannot find it again
>
> {{```[2019-11-29 12:58:54,548] \{logging_mixin.py:112} INFO - [2019-11-29
> 12:58:54,547] \{base_hook.py:84} INFO - Using connection to: id:
> spark_default. Host: yarn, Port: None, Schema: None, Login: None, Password:
> None, extra: \{'deploy-mode': 'cluster', 'spark-binary':
> '/usr/bin/spark2-submit/'}
> [2019-11-29 12:58:54,548] \{logging_mixin.py:112} INFO - [2019-11-29
> 12:58:54,548] \{spark_submit_hook.py:302} INFO - Spark-Submit cmd:
> ['spark-submit', '--master', 'yarn', '--conf', 'spark.yarn.maxAppAttempts=2',
> '--files', '/home/centos/airflow/dags/conf/aws_keys.pickle',
> '--num-executors', '4', '--executor-cores', '4', '--executor-memory', '20G',
> '--name', 'universal_cross_matrix_joins_AUS_27-11-2019', '--deploy-mode',
> 'cluster',
> '/home/centos/airflow/scripts/near-airflow-repo/scripts/universal_cross_matrix/join_ifa_ncid.py',
> '--input_date', '27-11-2019', '--country', 'AUS', '--tenant_id', '63ce7457']
> [2019-11-29 12:58:54,553] \{logging_mixin.py:112} INFO - [2019-11-29
> 12:58:54,553] \{spark_submit_hook.py:433} INFO - shell-init: error retrieving
> current directory: getcwd: cannot access parent directories: No such file or
> directory
> [2019-11-29 12:58:54,555] \{logging_mixin.py:112} INFO - [2019-11-29
> 12:58:54,555] \{spark_submit_hook.py:433} INFO - shell-init: error retrieving
> current directory: getcwd: cannot access parent directories: No such file or
> directory
> [2019-11-29 12:58:54,557] \{logging_mixin.py:112} INFO - [2019-11-29
> 12:58:54,557] \{spark_submit_hook.py:433} INFO - shell-init: error retrieving
> current directory: getcwd: cannot access parent directories: No such file or
> directory
> [2019-11-29 12:58:54,560] \{logging_mixin.py:112} INFO - [2019-11-29
> 12:58:54,560] \{spark_submit_hook.py:433} INFO - chdir: error retrieving
> current directory: getcwd: cannot access parent directories: No such file or
> directory
> [2019-11-29 12:58:54,563] \{logging_mixin.py:112} INFO - [2019-11-29
> 12:58:54,563] \{spark_submit_hook.py:433} INFO - chdir: error retrieving
> current directory: getcwd: cannot access parent directories: No such file or
> directory
> [2019-11-29 12:58:54,565] \{logging_mixin.py:112} INFO - [2019-11-29
> 12:58:54,564] \{spark_submit_hook.py:433} INFO - chdir: error retrieving
> current directory: getcwd: cannot access parent directories: No such file or
> directory
> [2019-11-29 12:58:54,567] \{logging_mixin.py:112} INFO - [2019-11-29
> 12:58:54,567] \{spark_submit_hook.py:433} INFO - shell-init: error retrieving
> current directory: getcwd: cannot access parent directories: No such file or
> directory
> [2019-11-29 12:58:54,569] \{logging_mixin.py:112} INFO - [2019-11-29
> 12:58:54,569] \{spark_submit_hook.py:433} INFO - chdir: error retrieving
> current directory: getcwd: cannot access parent directories: No such file or
> directory
> [2019-11-29 12:58:54,571] \{logging_mixin.py:112} INFO - [2019-11-29
> 12:58:54,571] \{spark_submit_hook.py:433} INFO - chdir: error retrieving
> current directory: getcwd: cannot access parent directories: No such file or
> directory
> [2019-11-29 12:58:54,574] \{logging_mixin.py:112} INFO - [2019-11-29
> 12:58:54,574] \{spark_submit_hook.py:433} INFO - chdir: error retrieving
> current directory: getcwd: cannot access parent directories: No such file or
> directory
> [2019-11-29 12:58:54,576] \{logging_mixin.py:112} INFO - [2019-11-29
> 12:58:54,576] \{spark_submit_hook.py:433} INFO - chdir: error retrieving
> current directory: getcwd: cannot access parent directories: No such file or
> directory```}}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
