[jira] [Updated] (AIRFLOW-2516) Deadlock found when trying to update task_instance table
[ https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-2516: Fix Version/s: (was: 1.10.10) 1.10.11 > Deadlock found when trying to update task_instance table > > > Key: AIRFLOW-2516 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2516 > Project: Apache Airflow > Issue Type: Bug > Components: DagRun >Affects Versions: 1.8.0, 1.9.0, 1.10.0, 1.10.1, 1.10.2, 1.10.3, 1.10.4, > 1.10.5, 1.10.6, 1.10.7 >Reporter: Jeff Liu >Assignee: Jarek Potiuk >Priority: Major > Fix For: 1.10.11 > > Attachments: Screenshot 2019-12-30 at 10.42.52.png, > image-2019-12-30-10-48-41-313.png, image-2019-12-30-10-58-02-610.png, > jobs.py, jobs_fixed_deadlock_possibly_1.9.py, > scheduler_job_fixed_deadlock_possibly_1.10.6.py > > > > > {code:java} > [2018-05-23 17:59:57,218] {base_task_runner.py:98} INFO - Subtask: > [2018-05-23 17:59:57,217] {base_executor.py:49} INFO - Adding to queue: > airflow run production_wipeout_wipe_manager.Carat Carat_20180227 > 2018-05-23T17:41:18.815809 --local -sd DAGS_FOLDER/wipeout/wipeout.py > [2018-05-23 17:59:57,231] {base_task_runner.py:98} INFO - Subtask: Traceback > (most recent call last): > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/bin/airflow", line 27, in > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: > args.func(args) > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: > pool=args.pool, > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in > wrapper > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: result = > func(*args, **kwargs) > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1532, in > _run_raw_task > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: > self.handle_failure(e, test_mode, context) > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1641, in > handle_failure > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: > session.merge(self) > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line > 1920, in merge > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: > _resolve_conflict_map=_resolve_conflict_map) > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line > 1974, in _merge > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: merged = > self.query(mapper.class_).get(key[1]) > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 882, > in get > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: ident, > loading.load_on_pk_identity) > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 952, > in _get_impl > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return > db_load_fn(self, primary_key_identity) > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 247, > in load_on_pk_i > dentity > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return > q.one() > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2884, > in one > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret = > self.one_or_none() > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2854, > in one_or_none > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret = > list(self) > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2925, > in __iter__ > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: return > self._execute_and_instances(context) > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO -
[jira] [Updated] (AIRFLOW-2516) Deadlock found when trying to update task_instance table
[ https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-2516: Fix Version/s: (was: 1.10.8) 1.10.10 > Deadlock found when trying to update task_instance table > > > Key: AIRFLOW-2516 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2516 > Project: Apache Airflow > Issue Type: Bug > Components: DagRun >Affects Versions: 1.8.0, 1.9.0, 1.10.0, 1.10.1, 1.10.2, 1.10.3, 1.10.4, > 1.10.5, 1.10.6, 1.10.7 >Reporter: Jeff Liu >Assignee: Jarek Potiuk >Priority: Major > Fix For: 1.10.10 > > Attachments: Screenshot 2019-12-30 at 10.42.52.png, > image-2019-12-30-10-48-41-313.png, image-2019-12-30-10-58-02-610.png, > jobs.py, jobs_fixed_deadlock_possibly_1.9.py, > scheduler_job_fixed_deadlock_possibly_1.10.6.py > > > > > {code:java} > [2018-05-23 17:59:57,218] {base_task_runner.py:98} INFO - Subtask: > [2018-05-23 17:59:57,217] {base_executor.py:49} INFO - Adding to queue: > airflow run production_wipeout_wipe_manager.Carat Carat_20180227 > 2018-05-23T17:41:18.815809 --local -sd DAGS_FOLDER/wipeout/wipeout.py > [2018-05-23 17:59:57,231] {base_task_runner.py:98} INFO - Subtask: Traceback > (most recent call last): > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/bin/airflow", line 27, in > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: > args.func(args) > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: > pool=args.pool, > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in > wrapper > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: result = > func(*args, **kwargs) > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1532, in > _run_raw_task > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: > self.handle_failure(e, test_mode, context) > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1641, in > handle_failure > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: > session.merge(self) > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line > 1920, in merge > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: > _resolve_conflict_map=_resolve_conflict_map) > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line > 1974, in _merge > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: merged = > self.query(mapper.class_).get(key[1]) > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 882, > in get > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: ident, > loading.load_on_pk_identity) > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 952, > in _get_impl > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return > db_load_fn(self, primary_key_identity) > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 247, > in load_on_pk_i > dentity > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return > q.one() > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2884, > in one > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret = > self.one_or_none() > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2854, > in one_or_none > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret = > list(self) > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2925, > in __iter__ > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: return > self._execute_and_instances(context) > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - S
[jira] [Updated] (AIRFLOW-2516) Deadlock found when trying to update task_instance table
[ https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-2516: -- Attachment: (was: jobs_fixed_deadlock_possibly_1.9.py) > Deadlock found when trying to update task_instance table > > > Key: AIRFLOW-2516 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2516 > Project: Apache Airflow > Issue Type: Bug > Components: DagRun >Affects Versions: 1.8.0, 1.9.0, 1.10.0, 1.10.1, 1.10.2, 1.10.3, 1.10.4, > 1.10.5, 1.10.6, 1.10.7 >Reporter: Jeff Liu >Assignee: Jarek Potiuk >Priority: Major > Fix For: 1.10.8 > > Attachments: Screenshot 2019-12-30 at 10.42.52.png, > image-2019-12-30-10-48-41-313.png, image-2019-12-30-10-58-02-610.png, > jobs.py, jobs_fixed_deadlock_possibly_1.9.py, > scheduler_job_fixed_deadlock_possibly_1.10.6.py > > > > > {code:java} > [2018-05-23 17:59:57,218] {base_task_runner.py:98} INFO - Subtask: > [2018-05-23 17:59:57,217] {base_executor.py:49} INFO - Adding to queue: > airflow run production_wipeout_wipe_manager.Carat Carat_20180227 > 2018-05-23T17:41:18.815809 --local -sd DAGS_FOLDER/wipeout/wipeout.py > [2018-05-23 17:59:57,231] {base_task_runner.py:98} INFO - Subtask: Traceback > (most recent call last): > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/bin/airflow", line 27, in > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: > args.func(args) > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: > pool=args.pool, > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in > wrapper > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: result = > func(*args, **kwargs) > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1532, in > _run_raw_task > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: > self.handle_failure(e, test_mode, context) > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1641, in > handle_failure > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: > session.merge(self) > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line > 1920, in merge > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: > _resolve_conflict_map=_resolve_conflict_map) > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line > 1974, in _merge > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: merged = > self.query(mapper.class_).get(key[1]) > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 882, > in get > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: ident, > loading.load_on_pk_identity) > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 952, > in _get_impl > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return > db_load_fn(self, primary_key_identity) > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 247, > in load_on_pk_i > dentity > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return > q.one() > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2884, > in one > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret = > self.one_or_none() > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2854, > in one_or_none > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret = > list(self) > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2925, > in __iter__ > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: return > self._execute_and_instances(context) > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO -
[jira] [Updated] (AIRFLOW-2516) Deadlock found when trying to update task_instance table
[ https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-2516: -- Affects Version/s: 1.9.0 1.10.0 1.10.1 1.10.2 1.10.3 1.10.4 1.10.5 1.10.6 1.10.7 > Deadlock found when trying to update task_instance table > > > Key: AIRFLOW-2516 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2516 > Project: Apache Airflow > Issue Type: Bug > Components: DagRun >Affects Versions: 1.8.0, 1.9.0, 1.10.0, 1.10.1, 1.10.2, 1.10.3, 1.10.4, > 1.10.5, 1.10.6, 1.10.7 >Reporter: Jeff Liu >Assignee: Jarek Potiuk >Priority: Major > Attachments: Screenshot 2019-12-30 at 10.42.52.png, > image-2019-12-30-10-48-41-313.png, image-2019-12-30-10-58-02-610.png, > jobs.py, jobs_fixed_deadlock_possibly_1.9.py, > jobs_fixed_deadlock_possibly_1.9.py, > scheduler_job_fixed_deadlock_possibly_1.10.6.py > > > > > {code:java} > [2018-05-23 17:59:57,218] {base_task_runner.py:98} INFO - Subtask: > [2018-05-23 17:59:57,217] {base_executor.py:49} INFO - Adding to queue: > airflow run production_wipeout_wipe_manager.Carat Carat_20180227 > 2018-05-23T17:41:18.815809 --local -sd DAGS_FOLDER/wipeout/wipeout.py > [2018-05-23 17:59:57,231] {base_task_runner.py:98} INFO - Subtask: Traceback > (most recent call last): > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/bin/airflow", line 27, in > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: > args.func(args) > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: > pool=args.pool, > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in > wrapper > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: result = > func(*args, **kwargs) > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1532, in > _run_raw_task > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: > self.handle_failure(e, test_mode, context) > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1641, in > handle_failure > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: > session.merge(self) > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line > 1920, in merge > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: > _resolve_conflict_map=_resolve_conflict_map) > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line > 1974, in _merge > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: merged = > self.query(mapper.class_).get(key[1]) > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 882, > in get > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: ident, > loading.load_on_pk_identity) > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 952, > in _get_impl > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return > db_load_fn(self, primary_key_identity) > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 247, > in load_on_pk_i > dentity > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return > q.one() > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2884, > in one > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret = > self.one_or_none() > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2854, > in one_or_none > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret = > list(self) > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/or
[jira] [Updated] (AIRFLOW-2516) Deadlock found when trying to update task_instance table
[ https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-2516: -- Attachment: (was: jobs_fixed_deadlock_possibly.py) > Deadlock found when trying to update task_instance table > > > Key: AIRFLOW-2516 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2516 > Project: Apache Airflow > Issue Type: Bug > Components: DagRun >Affects Versions: 1.8.0 >Reporter: Jeff Liu >Priority: Major > Attachments: Screenshot 2019-12-30 at 10.42.52.png, > image-2019-12-30-10-48-41-313.png, image-2019-12-30-10-58-02-610.png, > jobs.py, jobs_fixed_deadlock_possibly_1.9.py, > jobs_fixed_deadlock_possibly_1.9.py, > scheduler_job_fixed_deadlock_possibly_1.10.6.py > > > > > {code:java} > [2018-05-23 17:59:57,218] {base_task_runner.py:98} INFO - Subtask: > [2018-05-23 17:59:57,217] {base_executor.py:49} INFO - Adding to queue: > airflow run production_wipeout_wipe_manager.Carat Carat_20180227 > 2018-05-23T17:41:18.815809 --local -sd DAGS_FOLDER/wipeout/wipeout.py > [2018-05-23 17:59:57,231] {base_task_runner.py:98} INFO - Subtask: Traceback > (most recent call last): > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/bin/airflow", line 27, in > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: > args.func(args) > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: > pool=args.pool, > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in > wrapper > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: result = > func(*args, **kwargs) > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1532, in > _run_raw_task > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: > self.handle_failure(e, test_mode, context) > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1641, in > handle_failure > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: > session.merge(self) > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line > 1920, in merge > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: > _resolve_conflict_map=_resolve_conflict_map) > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line > 1974, in _merge > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: merged = > self.query(mapper.class_).get(key[1]) > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 882, > in get > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: ident, > loading.load_on_pk_identity) > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 952, > in _get_impl > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return > db_load_fn(self, primary_key_identity) > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 247, > in load_on_pk_i > dentity > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return > q.one() > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2884, > in one > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret = > self.one_or_none() > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2854, > in one_or_none > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret = > list(self) > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2925, > in __iter__ > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: return > self._execute_and_instances(context) > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2946, > in _exec
[jira] [Updated] (AIRFLOW-2516) Deadlock found when trying to update task_instance table
[ https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-2516: -- Attachment: jobs_fixed_deadlock_possibly_1.9.py > Deadlock found when trying to update task_instance table > > > Key: AIRFLOW-2516 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2516 > Project: Apache Airflow > Issue Type: Bug > Components: DagRun >Affects Versions: 1.8.0 >Reporter: Jeff Liu >Priority: Major > Attachments: Screenshot 2019-12-30 at 10.42.52.png, > image-2019-12-30-10-48-41-313.png, image-2019-12-30-10-58-02-610.png, > jobs.py, jobs_fixed_deadlock_possibly.py, > jobs_fixed_deadlock_possibly_1.9.py, jobs_fixed_deadlock_possibly_1.9.py, > scheduler_job_fixed_deadlock_possibly_1.10.6.py > > > > > {code:java} > [2018-05-23 17:59:57,218] {base_task_runner.py:98} INFO - Subtask: > [2018-05-23 17:59:57,217] {base_executor.py:49} INFO - Adding to queue: > airflow run production_wipeout_wipe_manager.Carat Carat_20180227 > 2018-05-23T17:41:18.815809 --local -sd DAGS_FOLDER/wipeout/wipeout.py > [2018-05-23 17:59:57,231] {base_task_runner.py:98} INFO - Subtask: Traceback > (most recent call last): > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/bin/airflow", line 27, in > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: > args.func(args) > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: > pool=args.pool, > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in > wrapper > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: result = > func(*args, **kwargs) > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1532, in > _run_raw_task > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: > self.handle_failure(e, test_mode, context) > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1641, in > handle_failure > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: > session.merge(self) > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line > 1920, in merge > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: > _resolve_conflict_map=_resolve_conflict_map) > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line > 1974, in _merge > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: merged = > self.query(mapper.class_).get(key[1]) > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 882, > in get > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: ident, > loading.load_on_pk_identity) > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 952, > in _get_impl > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return > db_load_fn(self, primary_key_identity) > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 247, > in load_on_pk_i > dentity > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return > q.one() > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2884, > in one > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret = > self.one_or_none() > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2854, > in one_or_none > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret = > list(self) > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2925, > in __iter__ > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: return > self._execute_and_instances(context) > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.p
[jira] [Updated] (AIRFLOW-2516) Deadlock found when trying to update task_instance table
[ https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-2516: -- Attachment: jobs_fixed_deadlock_possibly_1.9.py scheduler_job_fixed_deadlock_possibly_1.10.6.py > Deadlock found when trying to update task_instance table > > > Key: AIRFLOW-2516 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2516 > Project: Apache Airflow > Issue Type: Bug > Components: DagRun >Affects Versions: 1.8.0 >Reporter: Jeff Liu >Priority: Major > Attachments: Screenshot 2019-12-30 at 10.42.52.png, > image-2019-12-30-10-48-41-313.png, image-2019-12-30-10-58-02-610.png, > jobs.py, jobs_fixed_deadlock_possibly.py, > jobs_fixed_deadlock_possibly_1.9.py, > scheduler_job_fixed_deadlock_possibly_1.10.6.py > > > > > {code:java} > [2018-05-23 17:59:57,218] {base_task_runner.py:98} INFO - Subtask: > [2018-05-23 17:59:57,217] {base_executor.py:49} INFO - Adding to queue: > airflow run production_wipeout_wipe_manager.Carat Carat_20180227 > 2018-05-23T17:41:18.815809 --local -sd DAGS_FOLDER/wipeout/wipeout.py > [2018-05-23 17:59:57,231] {base_task_runner.py:98} INFO - Subtask: Traceback > (most recent call last): > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/bin/airflow", line 27, in > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: > args.func(args) > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: > pool=args.pool, > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in > wrapper > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: result = > func(*args, **kwargs) > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1532, in > _run_raw_task > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: > self.handle_failure(e, test_mode, context) > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1641, in > handle_failure > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: > session.merge(self) > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line > 1920, in merge > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: > _resolve_conflict_map=_resolve_conflict_map) > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line > 1974, in _merge > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: merged = > self.query(mapper.class_).get(key[1]) > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 882, > in get > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: ident, > loading.load_on_pk_identity) > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 952, > in _get_impl > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return > db_load_fn(self, primary_key_identity) > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 247, > in load_on_pk_i > dentity > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return > q.one() > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2884, > in one > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret = > self.one_or_none() > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2854, > in one_or_none > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret = > list(self) > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2925, > in __iter__ > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: return > self._execute_and_instances(context) > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-pack
[jira] [Updated] (AIRFLOW-2516) Deadlock found when trying to update task_instance table
[ https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-2516: -- Attachment: jobs_fixed_deadlock_possibly.py > Deadlock found when trying to update task_instance table > > > Key: AIRFLOW-2516 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2516 > Project: Apache Airflow > Issue Type: Bug > Components: DagRun >Affects Versions: 1.8.0 >Reporter: Jeff Liu >Priority: Major > Attachments: Screenshot 2019-12-30 at 10.42.52.png, > image-2019-12-30-10-48-41-313.png, image-2019-12-30-10-58-02-610.png, > jobs.py, jobs_fixed_deadlock_possibly.py > > > > > {code:java} > [2018-05-23 17:59:57,218] {base_task_runner.py:98} INFO - Subtask: > [2018-05-23 17:59:57,217] {base_executor.py:49} INFO - Adding to queue: > airflow run production_wipeout_wipe_manager.Carat Carat_20180227 > 2018-05-23T17:41:18.815809 --local -sd DAGS_FOLDER/wipeout/wipeout.py > [2018-05-23 17:59:57,231] {base_task_runner.py:98} INFO - Subtask: Traceback > (most recent call last): > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/bin/airflow", line 27, in > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: > args.func(args) > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: > pool=args.pool, > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in > wrapper > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: result = > func(*args, **kwargs) > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1532, in > _run_raw_task > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: > self.handle_failure(e, test_mode, context) > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1641, in > handle_failure > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: > session.merge(self) > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line > 1920, in merge > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: > _resolve_conflict_map=_resolve_conflict_map) > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line > 1974, in _merge > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: merged = > self.query(mapper.class_).get(key[1]) > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 882, > in get > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: ident, > loading.load_on_pk_identity) > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 952, > in _get_impl > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return > db_load_fn(self, primary_key_identity) > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 247, > in load_on_pk_i > dentity > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return > q.one() > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2884, > in one > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret = > self.one_or_none() > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2854, > in one_or_none > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret = > list(self) > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2925, > in __iter__ > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: return > self._execute_and_instances(context) > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2946, > in _execute_and_instances > [2018-05-23 17:59:57,240] {base_task_runner.py:98} INFO - Subtask: > close_with_result
[jira] [Updated] (AIRFLOW-2516) Deadlock found when trying to update task_instance table
[ https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-2516: -- Attachment: jobs.py > Deadlock found when trying to update task_instance table > > > Key: AIRFLOW-2516 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2516 > Project: Apache Airflow > Issue Type: Bug > Components: DagRun >Affects Versions: 1.8.0 >Reporter: Jeff Liu >Priority: Major > Attachments: Screenshot 2019-12-30 at 10.42.52.png, > image-2019-12-30-10-48-41-313.png, image-2019-12-30-10-58-02-610.png, jobs.py > > > > > {code:java} > [2018-05-23 17:59:57,218] {base_task_runner.py:98} INFO - Subtask: > [2018-05-23 17:59:57,217] {base_executor.py:49} INFO - Adding to queue: > airflow run production_wipeout_wipe_manager.Carat Carat_20180227 > 2018-05-23T17:41:18.815809 --local -sd DAGS_FOLDER/wipeout/wipeout.py > [2018-05-23 17:59:57,231] {base_task_runner.py:98} INFO - Subtask: Traceback > (most recent call last): > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/bin/airflow", line 27, in > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: > args.func(args) > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: > pool=args.pool, > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in > wrapper > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: result = > func(*args, **kwargs) > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1532, in > _run_raw_task > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: > self.handle_failure(e, test_mode, context) > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1641, in > handle_failure > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: > session.merge(self) > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line > 1920, in merge > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: > _resolve_conflict_map=_resolve_conflict_map) > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line > 1974, in _merge > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: merged = > self.query(mapper.class_).get(key[1]) > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 882, > in get > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: ident, > loading.load_on_pk_identity) > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 952, > in _get_impl > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return > db_load_fn(self, primary_key_identity) > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 247, > in load_on_pk_i > dentity > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return > q.one() > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2884, > in one > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret = > self.one_or_none() > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2854, > in one_or_none > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret = > list(self) > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2925, > in __iter__ > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: return > self._execute_and_instances(context) > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2946, > in _execute_and_instances > [2018-05-23 17:59:57,240] {base_task_runner.py:98} INFO - Subtask: > close_with_result=True) > [2018-05-23 17:59:57,240] {base_task_runner.py:98}
[jira] [Updated] (AIRFLOW-2516) Deadlock found when trying to update task_instance table
[ https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-2516: -- Attachment: image-2019-12-30-10-58-02-610.png > Deadlock found when trying to update task_instance table > > > Key: AIRFLOW-2516 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2516 > Project: Apache Airflow > Issue Type: Bug > Components: DagRun >Affects Versions: 1.8.0 >Reporter: Jeff Liu >Priority: Major > Attachments: Screenshot 2019-12-30 at 10.42.52.png, > image-2019-12-30-10-48-41-313.png, image-2019-12-30-10-58-02-610.png > > > > > {code:java} > [2018-05-23 17:59:57,218] {base_task_runner.py:98} INFO - Subtask: > [2018-05-23 17:59:57,217] {base_executor.py:49} INFO - Adding to queue: > airflow run production_wipeout_wipe_manager.Carat Carat_20180227 > 2018-05-23T17:41:18.815809 --local -sd DAGS_FOLDER/wipeout/wipeout.py > [2018-05-23 17:59:57,231] {base_task_runner.py:98} INFO - Subtask: Traceback > (most recent call last): > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/bin/airflow", line 27, in > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: > args.func(args) > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: > pool=args.pool, > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in > wrapper > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: result = > func(*args, **kwargs) > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1532, in > _run_raw_task > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: > self.handle_failure(e, test_mode, context) > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1641, in > handle_failure > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: > session.merge(self) > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line > 1920, in merge > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: > _resolve_conflict_map=_resolve_conflict_map) > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line > 1974, in _merge > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: merged = > self.query(mapper.class_).get(key[1]) > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 882, > in get > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: ident, > loading.load_on_pk_identity) > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 952, > in _get_impl > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return > db_load_fn(self, primary_key_identity) > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 247, > in load_on_pk_i > dentity > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return > q.one() > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2884, > in one > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret = > self.one_or_none() > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2854, > in one_or_none > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret = > list(self) > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2925, > in __iter__ > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: return > self._execute_and_instances(context) > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2946, > in _execute_and_instances > [2018-05-23 17:59:57,240] {base_task_runner.py:98} INFO - Subtask: > close_with_result=True) > [2018-05-23 17:59:57,240] {base_ta
[jira] [Updated] (AIRFLOW-2516) Deadlock found when trying to update task_instance table
[ https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-2516: -- Attachment: image-2019-12-30-10-48-41-313.png > Deadlock found when trying to update task_instance table > > > Key: AIRFLOW-2516 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2516 > Project: Apache Airflow > Issue Type: Bug > Components: DagRun >Affects Versions: 1.8.0 >Reporter: Jeff Liu >Priority: Major > Attachments: Screenshot 2019-12-30 at 10.42.52.png, > image-2019-12-30-10-48-41-313.png > > > > > {code:java} > [2018-05-23 17:59:57,218] {base_task_runner.py:98} INFO - Subtask: > [2018-05-23 17:59:57,217] {base_executor.py:49} INFO - Adding to queue: > airflow run production_wipeout_wipe_manager.Carat Carat_20180227 > 2018-05-23T17:41:18.815809 --local -sd DAGS_FOLDER/wipeout/wipeout.py > [2018-05-23 17:59:57,231] {base_task_runner.py:98} INFO - Subtask: Traceback > (most recent call last): > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/bin/airflow", line 27, in > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: > args.func(args) > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: > pool=args.pool, > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in > wrapper > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: result = > func(*args, **kwargs) > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1532, in > _run_raw_task > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: > self.handle_failure(e, test_mode, context) > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1641, in > handle_failure > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: > session.merge(self) > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line > 1920, in merge > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: > _resolve_conflict_map=_resolve_conflict_map) > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line > 1974, in _merge > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: merged = > self.query(mapper.class_).get(key[1]) > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 882, > in get > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: ident, > loading.load_on_pk_identity) > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 952, > in _get_impl > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return > db_load_fn(self, primary_key_identity) > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 247, > in load_on_pk_i > dentity > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return > q.one() > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2884, > in one > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret = > self.one_or_none() > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2854, > in one_or_none > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret = > list(self) > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2925, > in __iter__ > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: return > self._execute_and_instances(context) > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2946, > in _execute_and_instances > [2018-05-23 17:59:57,240] {base_task_runner.py:98} INFO - Subtask: > close_with_result=True) > [2018-05-23 17:59:57,240] {base_task_runner.py:98} INFO - Subtask: Fi
[jira] [Updated] (AIRFLOW-2516) Deadlock found when trying to update task_instance table
[ https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-2516: -- Attachment: Screenshot 2019-12-30 at 10.42.52.png > Deadlock found when trying to update task_instance table > > > Key: AIRFLOW-2516 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2516 > Project: Apache Airflow > Issue Type: Bug > Components: DagRun >Affects Versions: 1.8.0 >Reporter: Jeff Liu >Priority: Major > Attachments: Screenshot 2019-12-30 at 10.42.52.png > > > > > {code:java} > [2018-05-23 17:59:57,218] {base_task_runner.py:98} INFO - Subtask: > [2018-05-23 17:59:57,217] {base_executor.py:49} INFO - Adding to queue: > airflow run production_wipeout_wipe_manager.Carat Carat_20180227 > 2018-05-23T17:41:18.815809 --local -sd DAGS_FOLDER/wipeout/wipeout.py > [2018-05-23 17:59:57,231] {base_task_runner.py:98} INFO - Subtask: Traceback > (most recent call last): > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/bin/airflow", line 27, in > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: > args.func(args) > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run > [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: > pool=args.pool, > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in > wrapper > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: result = > func(*args, **kwargs) > [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1532, in > _run_raw_task > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: > self.handle_failure(e, test_mode, context) > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1641, in > handle_failure > [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: > session.merge(self) > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line > 1920, in merge > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: > _resolve_conflict_map=_resolve_conflict_map) > [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line > 1974, in _merge > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: merged = > self.query(mapper.class_).get(key[1]) > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 882, > in get > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: ident, > loading.load_on_pk_identity) > [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 952, > in _get_impl > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return > db_load_fn(self, primary_key_identity) > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 247, > in load_on_pk_i > dentity > [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return > q.one() > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2884, > in one > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret = > self.one_or_none() > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2854, > in one_or_none > [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret = > list(self) > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2925, > in __iter__ > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: return > self._execute_and_instances(context) > [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2946, > in _execute_and_instances > [2018-05-23 17:59:57,240] {base_task_runner.py:98} INFO - Subtask: > close_with_result=True) > [2018-05-23 17:59:57,240] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/lib/python2.7/si