[GitHub] [airflow] Taragolis commented on pull request #29616: Refactor docker-compose quick start test
Taragolis commented on PR #29616: URL: https://github.com/apache/airflow/pull/29616#issuecomment-1444510331 Well, I guess we if we could add only this changes which might help to get a bit more information rather than previously in case if the error become "popular" again -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] Taragolis commented on pull request #29616: Refactor docker-compose quick start test
Taragolis commented on PR #29616: URL: https://github.com/apache/airflow/pull/29616#issuecomment-1441985039 `Tests / Test docker-compose quick start (pull_request)` just stop failing and potentially we might fix it by another PR or it became rare in public runners :octocat: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] Taragolis commented on pull request #29616: Refactor docker-compose quick start test
Taragolis commented on PR #29616:
URL: https://github.com/apache/airflow/pull/29616#issuecomment-1437988917
Something new
```console
airflow-scheduler_1 |
airflow-scheduler_1 | BACKEND=redis
airflow-scheduler_1 | DB_HOST=redis
airflow-scheduler_1 | DB_PORT=6379
airflow-scheduler_1 |
airflow-scheduler_1 |
/home/airflow/.local/lib/python3.7/site-packages/airflow/models/base.py:49
MovedIn20Warning: Deprecated API features detected! These feature(s) are not
compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to
updating applications, ensure requirements files are pinned to
"sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all
deprecation warnings. Set environment variable
SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on
SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
airflow-scheduler_1 | _
airflow-scheduler_1 | |__( )_ __/__ / __
airflow-scheduler_1 | /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
airflow-scheduler_1 | ___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
airflow-scheduler_1 | _/_/ |_/_/ /_//_//_/ \//|__/
airflow-scheduler_1 | [2023-02-21T07:28:42.954+]
{executor_loader.py:114} INFO - Loaded executor: CeleryExecutor
airflow-scheduler_1 | [2023-02-21T07:28:43.045+]
{scheduler_job.py:724} INFO - Starting the scheduler
airflow-scheduler_1 | [2023-02-21T07:28:43.054+]
{scheduler_job.py:731} INFO - Processing each file at most -1 times
airflow-scheduler_1 | [2023-02-21T07:28:43.061+] {manager.py:164}
INFO - Launched DagFileProcessorManager with pid: 32
airflow-scheduler_1 | [2023-02-21T07:28:43.071+]
{scheduler_job.py:1437} INFO - Resetting orphaned tasks for active dag runs
airflow-scheduler_1 | [2023-02-21T07:28:43.080+] {settings.py:61}
INFO - Configured default timezone Timezone('UTC')
airflow-scheduler_1 | [2023-02-21T07:28:46.099+]
{scheduler_job.py:788} ERROR - Exception when executing
SchedulerJob._run_scheduler_loop
airflow-scheduler_1 | Traceback (most recent call last):
airflow-scheduler_1 | File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py",
line 771, in _execute
airflow-scheduler_1 | self._run_scheduler_loop()
airflow-scheduler_1 | File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py",
line 899, in _run_scheduler_loop
airflow-scheduler_1 | num_queued_tis = self._do_scheduling(session)
airflow-scheduler_1 | File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py",
line 1006, in _do_scheduling
airflow-scheduler_1 | num_queued_tis =
self._critical_section_enqueue_task_instances(session=session)
airflow-scheduler_1 | File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py",
line 589, in _critical_section_enqueue_task_instances
airflow-scheduler_1 | queued_tis =
self._executable_task_instances_to_queued(max_tis, session=session)
airflow-scheduler_1 | File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py",
line 280, in _executable_task_instances_to_queued
airflow-scheduler_1 | pools = Pool.slots_stats(lock_rows=True,
session=session)
airflow-scheduler_1 | File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py",
line 73, in wrapper
airflow-scheduler_1 | return func(*args, **kwargs)
airflow-scheduler_1 | File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/models/pool.py", line
174, in slots_stats
airflow-scheduler_1 | .group_by(TaskInstance.pool, TaskInstance.state)
airflow-scheduler_1 | File
"/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/query.py",
line 2773, in all
airflow-scheduler_1 | return self._iter().all()
airflow-scheduler_1 | File
"/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/result.py",
line 1129, in all
airflow-scheduler_1 | return self._allrows()
airflow-scheduler_1 | File
"/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/result.py",
line 401, in _allrows
airflow-scheduler_1 | rows = self._fetchall_impl()
airflow-scheduler_1 | File
"/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/result.py",
line 1813, in _fetchall_impl
airflow-scheduler_1 | return list(self.iterator)
airflow-scheduler_1 | File
"/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/loading.py",
line 147, in chunks
airflow-scheduler_1 | fetch = cursor._raw_all_rows()
airflow-scheduler_1 | File
"/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/result.py",
[GitHub] [airflow] Taragolis commented on pull request #29616: Refactor docker-compose quick start test
Taragolis commented on PR #29616:
URL: https://github.com/apache/airflow/pull/29616#issuecomment-1437330747
Yeah as soon as just run a go to do my daily routine it finally failed 磊
That is quite a bit interesting thing, some of them mostly could be a "wrong
assumptions"
Dag Runs
---
```console
HTTP: GET dags/example_bash_operator/dagRuns
{'dag_runs': [{'conf': {},
'dag_id': 'example_bash_operator',
'dag_run_id': 'test_dag_run_id',
'data_interval_end': '2023-02-20T00:00:00+00:00',
'data_interval_start': '2023-02-19T00:00:00+00:00',
'end_date': None,
'execution_date': '2023-02-20T10:30:00.702880+00:00',
'external_trigger': True,
'last_scheduling_decision': None,
'logical_date': '2023-02-20T10:30:00.702880+00:00',
'note': None,
'run_type': 'manual',
'start_date': None,
'state': 'queued'}],
'total_entries': 1}
```
`example_bash_operator` DAG has scheduling interval, as result we should see
here 2 DAG Runs, first for scheduled and second manual, in this case we could
see only one - manual which created during the test.
Scheduler Logs
---
```console
airflow-scheduler_1 |
airflow-scheduler_1 | BACKEND=redis
airflow-scheduler_1 | DB_HOST=redis
airflow-scheduler_1 | DB_PORT=6379
airflow-scheduler_1 |
airflow-scheduler_1 |
/home/airflow/.local/lib/python3.7/site-packages/airflow/models/base.py:49
MovedIn20Warning: Deprecated API features detected! These feature(s) are not
compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to
updating applications, ensure requirements files are pinned to
"sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all
deprecation warnings. Set environment variable
SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on
SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
airflow-scheduler_1 | _
airflow-scheduler_1 | |__( )_ __/__ / __
airflow-scheduler_1 | /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
airflow-scheduler_1 | ___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
airflow-scheduler_1 | _/_/ |_/_/ /_//_//_/ \//|__/
airflow-scheduler_1 | [2023-02-20T10:29:14.618+]
{executor_loader.py:114} INFO - Loaded executor: CeleryExecutor
airflow-scheduler_1 | [2023-02-20T10:29:14.664+]
{scheduler_job.py:724} INFO - Starting the scheduler
airflow-scheduler_1 | [2023-02-20T10:29:14.665+]
{scheduler_job.py:731} INFO - Processing each file at most -1 times
airflow-scheduler_1 | [2023-02-20T10:29:14.669+] {manager.py:164}
INFO - Launched DagFileProcessorManager with pid: 33
airflow-scheduler_1 | [2023-02-20T10:29:14.671+]
{scheduler_job.py:1437} INFO - Resetting orphaned tasks for active dag runs
airflow-scheduler_1 | [2023-02-20T10:29:14.685+] {settings.py:61}
INFO - Configured default timezone Timezone('UTC')
```
Thats all, seems like it scheduler is just hang but service reported that it
healthy. Is it problem with recent changes in health check
https://github.com/apache/airflow/pull/29408 and maybe problem with simple http
server in scheduler.
I would add output from `/health` endpoint in case of failure
Docker services after test failure
---
```console
$ docker ps
CONTAINER ID IMAGE
COMMAND CREATED STATUS
PORTS NAMES
8da8ebd97f17
ghcr.io/apache/airflow/main/prod/python3.7:a8723aa63be724652809c141714af95493aea68c
"/usr/bin/dumb-init …" 2 minutes ago Up 2 minutes (healthy) 8080/tcp
quick-start_airflow-triggerer_1
88a829428ce8
ghcr.io/apache/airflow/main/prod/python3.7:a8723aa63be724652809c141714af95493aea68c
"/usr/bin/dumb-init …" 2 minutes ago Up 2 minutes (healthy)
0.0.0.0:8080->8080/tcp, :::8080->8080/tcp quick-start_airflow-webserver_1
f3baa9496225
ghcr.io/apache/airflow/main/prod/python3.7:a8723aa63be724652809c141714af95493aea68c
"/usr/bin/dumb-init …" 2 minutes ago Up 2 minutes (healthy) 8080/tcp
quick-start_airflow-scheduler_1
134b3356ed96
ghcr.io/apache/airflow/main/prod/python3.7:a8723aa63be724652809c141714af95493aea68c
"/usr/bin/dumb-init …" 2 minutes ago Up 2 minutes (healthy) 8080/tcp
quick-start_airflow-worker_1
a5f5e8250820 redis:latest
[GitHub] [airflow] Taragolis commented on pull request #29616: Refactor docker-compose quick start test
Taragolis commented on PR #29616: URL: https://github.com/apache/airflow/pull/29616#issuecomment-1436664937 The more you look on error the less chance that this error will reproduce 藍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
