potiuk opened a new issue, #52020:
URL: https://github.com/apache/airflow/issues/52020
### Body
After #51930 where @amoghrajesh introduces a way how tests can define
connections without DB and follow up in #52017 where I made an attempt to
remove some of the `pytest.mark.db_test` we should now remove those marks from
providers, where it is easy. Some providers still use DB for other things, but
likely there are many providers that only used db to create a Connection and we
can turn those tests into non-db tests.
- [x] airbyte - #52017 - clean
- [ ] alibaba
- [x] amazon - #52017 - still db tests left
- [ ] apache
- [ ] apprise
- [ ] arangodb
- [ ] asana
- [ ] atlassian
- [ ] celery
- [ ] cloudant
- [ ] cncf
- [ ] common
- [ ] databricks
- [ ] datadog
- [ ] dbt
- [ ] dingding
- [ ] discord
- [ ] docker
- [ ] edge3
- [ ] elasticsearch
- [ ] exasol
- [ ] fab
- [ ] ftp
- [ ] git
- [ ] github
- [ ] google
- [ ] http
- [ ] imap
- [ ] jdbc
- [ ] keycloak
- [ ] microsoft
- [ ] mongo
- [ ] mysql
- [ ] openlineage
- [ ] opensearch
- [ ] opsgenie
- [ ] oracle
- [ ] pagerduty
- [ ] papermill
- [ ] qdrant
- [ ] redis
- [ ] salesforce
- [ ] samba
- [ ] sftp
- [ ] slack
- [ ] smtp
- [ ] snowflake
- [ ] sqlite
- [ ] ssh
- [ ] standard
- [ ] tableau
- [ ] telegram
- [ ] weaviate
- [ ] ydb
- [ ] zendesk
We shouild join the efforts and review the providers and attempt to remove
db_tests in as many places as possible.
The process of doing it is rather easy:
1) Run `breeze shell --backend none --skip-db-tests`
2) Remove all `pytest.mark.db_test` by find replace in the provider:
`@pytest.mark.db_test` -> ''
`pytestmark = pytest.mark.db_test` -> ''
There are likely also providers with `pytestmark =
[pytest.mark.db_test,other_marks]` or similar
3) In breeze go to the provider `cd providers/PROVIDER`
4) run `pytest test` -> when some tests fail because they "need dn", restore
the marks for those tests
5) After all tests succed, run `pytest test -n auto` -> that should run all
the non-db tests with xdist - in parallel, running them with multiple processes
- as many processors you have. There are a few tests that are marked as
`db_tests` to be run sequentially for other reasons than DB - running them with
xdist should help to see if there are some of them that still need it
6) Add PR with the changes. Our aim is not yet to remove those db calls
that hapen currently (yet) - but to see which of the `pytest.mark.db_test`
marks can be removed easily because they were only used by Connections. But it
might turn out that some providers will not have any more `pytest.mark.db_test`
- such as airbyte. So if everything is green and provider does not have any
more `pytest.mark.db_test` - please add it to .pre-commit-config.yml (in the
same PR).
Add it here adding `|` at the end followed by `^providers/PROVIDER/.*\.py$`
```
- id: check-pytest-mark-db-test-in-providers
language: pygrep
name: Check pytest.mark.db_test use in providers
entry: pytest\.mark\.db_test
pass_filenames: true
# Here we should add providers that are already free from the
pytest.mark.db_test
# and we want to keep them clean and only use non-db-tests
files: >
(?x)
^providers/airbyte/.*\.py$
```
### Committer
- [x] I acknowledge that I am a maintainer/committer of the Apache Airflow
project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
