devender-yadav opened a new issue #22059:
URL: https://github.com/apache/airflow/issues/22059
### Apache Airflow version
2.2.3
### What happened
Metastore = Postgres
concurrency=8
max_active_runs=1
DagBag import timeout is happening while retrieving Variable
```
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/__main__.py", line
48, in main
args.func(args)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py",
line 48, in command
return func(*args, **kwargs)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py", line
92, in wrapper
return f(*args, **kwargs)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py",
line 282, in task_run
dag = get_dag(args.subdir, args.dag_id)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py", line
193, in get_dag
f"Dag {dag_id!r} could not be found; either it does not exist or it
failed to parse."
airflow.exceptions.AirflowException: Dag 'Test1' could not be found; either
it does not exist or it failed to parse.
[2022-03-01 19:04:30,452] {variable.py:274} ERROR - Unable to retrieve
variable from secrets backend (MetastoreBackend). Checking subsequent secrets
backend.
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/models/variable.py",
line 267, in get_variable_from_secrets
var_val = secrets_backend.get_variable(key=key)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py",
line 70, in wrapper
return func(*args, session=session, **kwargs)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/secrets/metastore.py",
line 64, in get_variable
var_value = session.query(Variable).filter(Variable.key == key).first()
File
"/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/query.py",
line 3429, in first
ret = list(self[0:1])
File
"/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/query.py",
line 3203, in __getitem__
return list(res)
File
"/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/query.py",
line 3535, in __iter__
return self._execute_and_instances(context)
File
"/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/query.py",
line 3557, in _execute_and_instances
querycontext, self._connection_from_session, close_with_result=True
File
"/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/query.py",
line 3572, in _get_bind_args
mapper=self._bind_mapper(), clause=querycontext.statement, **kw
File
"/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/query.py",
line 3550, in _connection_from_session
conn = self.session.connection(**kw)
File
"/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/session.py",
line 1145, in connection
execution_options=execution_options,
File
"/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/session.py",
line 1151, in _connection_for_bind
engine, execution_options
File
"/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/session.py",
line 433, in _connection_for_bind
conn = bind._contextual_connect()
File
"/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py",
line 2302, in _contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File
"/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py",
line 2336, in _wrap_pool_connect
return fn()
File
"/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/base.py",
line 364, in connect
return _ConnectionFairy._checkout(self)
File
"/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/base.py",
line 778, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File
"/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/base.py",
line 495, in checkout
rec = pool._do_get()
File
"/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/impl.py",
line 241, in _do_get
return self._create_connection()
File
"/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/base.py",
line 309, in _create_connection
return _ConnectionRecord(self)
File
"/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/base.py",
line 440, in __init__
self.__connect(first_connect_check=True)
File
"/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/base.py",
line 661, in __connect
pool.logger.debug("Error on connect(): %s", e)
File
"/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py",
line 70, in __exit__
with_traceback=exc_tb,
File
"/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/compat.py",
line 182, in raise_
raise exception
File
"/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/base.py",
line 656, in __connect
connection = pool._invoke_creator(self)
File
"/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/strategies.py",
line 114, in connect
return dialect.connect(*cargs, **cparams)
File
"/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/default.py",
line 508, in connect
return self.dbapi.connect(*cargs, **cparams)
File
"/home/airflow/.local/lib/python3.7/site-packages/psycopg2/__init__.py", line
122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/timeout.py",
line 37, in handle_timeout
raise AirflowTaskTimeout(self.error_message)
airflow.exceptions.AirflowTaskTimeout: DagBag import timeout for
/opt/airflow/dags/repo/dags/test1.py after 30.0s.
Please take a look at these docs to improve your DAG import time:
*
https://airflow.apache.org/docs/apache-airflow/2.2.3/best-practices.html#top-level-python-code
*
https://airflow.apache.org/docs/apache-airflow/2.2.3/best-practices.html#reducing-dag-complexity,
PID: 7
[2022-03-01 19:04:30,460] {dagbag.py:334} ERROR - Failed to import:
/opt/airflow/dags/repo/dags/test1.py
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/models/dagbag.py",
line 331, in _load_modules_from_file
loader.exec_module(new_module)
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in
_call_with_frames_removed
File "/opt/airflow/dags/repo/dags/test1.py", line 276, in <module>
```
### What you expected to happen
_No response_
### How to reproduce
_No response_
### Operating System
Debian GNU/Linux 10 (buster)
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
