Re: [I] Trigger die with DB deadlock between scheduler [airflow]

2023-10-26 Thread via GitHub
github-actions[bot] commented on issue #27000: URL: https://github.com/apache/airflow/issues/27000#issuecomment-1780526916 This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activit

Re: [I] Google SQL shouldn't require password when using Application Default Credentials and Cloud SQL Auth proxy [airflow]

2023-10-26 Thread via GitHub
github-actions[bot] commented on issue #20660: URL: https://github.com/apache/airflow/issues/20660#issuecomment-1780527021 This issue has been closed because it has not received response from the issue author. -- This is an automated message from the Apache Git Service. To respond to the

Re: [I] Google SQL shouldn't require password when using Application Default Credentials and Cloud SQL Auth proxy [airflow]

2023-10-26 Thread via GitHub
github-actions[bot] closed issue #20660: Google SQL shouldn't require password when using Application Default Credentials and Cloud SQL Auth proxy URL: https://github.com/apache/airflow/issues/20660 -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] The sanitize_for_serialization(cls, obj) function outputs an incorret time format [airflow-client-python]

2023-10-26 Thread via GitHub
pierrejeambrun commented on PR #99: URL: https://github.com/apache/airflow-client-python/pull/99#issuecomment-1780534125 Hello, Can you try with a timezone aware datetime in the client? (For instance datetime.utcnow()) -- This is an automated message from the Apache Git Servic

[PR] Combined the `in_container_utils.sh with run_docs_build.sh [airflow]

2023-10-26 Thread via GitHub
KesharwaniArpita opened a new pull request, #35192: URL: https://github.com/apache/airflow/pull/35192 Combined the `in_container_utils.sh with run_docs_build.sh` to eliminate the dependency of the latter on the former in relation with the issue #26019 . The file is still in bash which will

Re: [PR] Combined the `in_container_utils.sh with run_docs_build.sh [airflow]

2023-10-26 Thread via GitHub
potiuk commented on PR #35192: URL: https://github.com/apache/airflow/pull/35192#issuecomment-1780574428 I do not think this is needed at alll 1) even if that script includes the common file - it does not seem to use any of it so at most i'd just **REMOVE** using the common script (an

[airflow-site] branch smtp created (now c092cefcda)

2023-10-26 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a change to branch smtp in repository https://gitbox.apache.org/repos/asf/airflow-site.git at c092cefcda Add SMTP provider to _index.md This branch includes the following new commits: new c092cefcda A

[PR] Add SMTP provider to _index.md [airflow-site]

2023-10-26 Thread via GitHub
eladkal opened a new pull request, #881: URL: https://github.com/apache/airflow-site/pull/881 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e

[airflow-site] 01/01: Add SMTP provider to _index.md

2023-10-26 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a commit to branch smtp in repository https://gitbox.apache.org/repos/asf/airflow-site.git commit c092cefcda4ff1a1e054f7aaf201e934d14a5b91 Author: Elad Kalif <45845474+elad...@users.noreply.github.com> AuthorDate:

Re: [PR] Add SMTP provider to _index.md [airflow-site]

2023-10-26 Thread via GitHub
eladkal merged PR #881: URL: https://github.com/apache/airflow-site/pull/881 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

[airflow-site] branch main updated: Add SMTP provider to _index.md (#881)

2023-10-26 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow-site.git The following commit(s) were added to refs/heads/main by this push: new 2962fa248c Add SMTP provider to _index.md (#8

[airflow-site] branch smtp deleted (was c092cefcda)

2023-10-26 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a change to branch smtp in repository https://gitbox.apache.org/repos/asf/airflow-site.git was c092cefcda Add SMTP provider to _index.md The revisions that were on this branch are still contained in other re

Re: [PR] Fix local k8s test venv installation to accomodate for local changes [airflow]

2023-10-26 Thread via GitHub
potiuk merged PR #35191: URL: https://github.com/apache/airflow/pull/35191 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[airflow] branch main updated: Fix local k8s test venv installation to accomodate for local changes (#35191)

2023-10-26 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new ba28334621 Fix local k8s test venv installation to

Re: [PR] Replace blocking IO with async IO in AsyncKubernetesHook [airflow]

2023-10-26 Thread via GitHub
potiuk commented on PR #35162: URL: https://github.com/apache/airflow/pull/35162#issuecomment-1780709943 Fixed in main. I rebased to see if it helps -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[PR] Add TriggerRule missing in rest API [airflow]

2023-10-26 Thread via GitHub
pankajastro opened a new pull request, #35194: URL: https://github.com/apache/airflow/pull/35194 closes: https://github.com/apache/airflow/issues/35186 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apac

Re: [PR] Add TriggerRule missing value in rest API [airflow]

2023-10-26 Thread via GitHub
pankajastro commented on code in PR #35194: URL: https://github.com/apache/airflow/pull/35194#discussion_r1372840086 ## airflow/api_connexion/openapi/v1.yaml: ## @@ -4757,19 +4757,29 @@ components: description: | Trigger rule. -*Changed in version 2.2.0

Re: [PR] Add TriggerRule missing value in rest API [airflow]

2023-10-26 Thread via GitHub
pankajastro commented on code in PR #35194: URL: https://github.com/apache/airflow/pull/35194#discussion_r1372840727 ## airflow/www/static/js/types/api-generated.ts: ## @@ -2207,21 +2207,31 @@ export interface components { /** * @description Trigger rule. * -

[airflow-site] branch gh-pages updated (7c741879be -> 121646f264)

2023-10-26 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch gh-pages in repository https://gitbox.apache.org/repos/asf/airflow-site.git discard 7c741879be Rewritten history to remove past gh-pages deployments new 121646f264 Rewritten history to

Re: [I] Airflow-2.6.3 Mysql: Deadlock found when trying to get lock; try restarting transaction [airflow]

2023-10-26 Thread via GitHub
langfu54 commented on issue #35144: URL: https://github.com/apache/airflow/issues/35144#issuecomment-1780813288 had the exactly same problem , once the task instance increase ,the schelduler shut down. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Update datasets.rst issue with running example code [airflow]

2023-10-26 Thread via GitHub
pankajastro merged PR #35035: URL: https://github.com/apache/airflow/pull/35035 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airf

Re: [PR] Update datasets.rst issue with running example code [airflow]

2023-10-26 Thread via GitHub
boring-cyborg[bot] commented on PR #35035: URL: https://github.com/apache/airflow/pull/35035#issuecomment-1780816125 Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional contributions.

[airflow] branch main updated: Update datasets.rst issue with running example code (#35035)

2023-10-26 Thread pankaj
This is an automated email from the ASF dual-hosted git repository. pankaj pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 8e26894073 Update datasets.rst issue with running e

Re: [PR] Add TriggerRule missing value in rest API [airflow]

2023-10-26 Thread via GitHub
potiuk merged PR #35194: URL: https://github.com/apache/airflow/pull/35194 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

Re: [I] Airflow REST API `Get tasks for DAG` returns error if DAG task has trigger rule `one_done` [airflow]

2023-10-26 Thread via GitHub
potiuk closed issue #35186: Airflow REST API `Get tasks for DAG` returns error if DAG task has trigger rule `one_done` URL: https://github.com/apache/airflow/issues/35186 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

[airflow] branch main updated: Add TriggerRule missing value in rest API (#35194)

2023-10-26 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new e3b3d78678 Add TriggerRule missing value in rest AP

Re: [I] Add HttpToS3Operator [airflow]

2023-10-26 Thread via GitHub
potiuk closed issue #35137: Add HttpToS3Operator URL: https://github.com/apache/airflow/issues/35137 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: com

Re: [PR] Add http to s3 operator [airflow]

2023-10-26 Thread via GitHub
potiuk merged PR #35176: URL: https://github.com/apache/airflow/pull/35176 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[airflow] branch main updated: Add Http to s3 operator (#35176)

2023-10-26 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 86640d166c Add Http to s3 operator (#35176) 86640d1

Re: [I] Airflow-2.6.3 Mysql: Deadlock found when trying to get lock; try restarting transaction [airflow]

2023-10-26 Thread via GitHub
Taragolis commented on issue #35144: URL: https://github.com/apache/airflow/issues/35144#issuecomment-1780900233 > once the task instance increase You mean increased number of simultaneous Task Instances? If so, what numbers of TI are we talking about? 10-100-1000? Any chan

Re: [I] Airflow-2.6.3 Mysql: Deadlock found when trying to get lock; try restarting transaction [airflow]

2023-10-26 Thread via GitHub
Taragolis commented on issue #35144: URL: https://github.com/apache/airflow/issues/35144#issuecomment-1780904035 Sorry I couldn't resist --- **X**: Mom can we have a solution for that deadlock issues in Airflow on MySQL? **Mom**: We have solution at home Solutio

Re: [I] Airflow-2.6.3 Mysql: Deadlock found when trying to get lock; try restarting transaction [airflow]

2023-10-26 Thread via GitHub
langfu54 commented on issue #35144: URL: https://github.com/apache/airflow/issues/35144#issuecomment-1780907043 > > once the task instance increase > You mean increased number of simultaneous Task Instances? If so, what numbers of TI are we talking about? 10-100-1000? less than 100

Re: [PR] The sanitize_for_serialization(cls, obj) function outputs an incorret time format [airflow-client-python]

2023-10-26 Thread via GitHub
junqilv commented on PR #99: URL: https://github.com/apache/airflow-client-python/pull/99#issuecomment-1780921520 > Hello, > > Can you try with a timezone aware datetime in the client? (For instance datetime.utcnow()) Thank for your attention to this pull request. This has noth

Re: [I] Airflow-2.6.3 Mysql: Deadlock found when trying to get lock; try restarting transaction [airflow]

2023-10-26 Thread via GitHub
Taragolis commented on issue #35144: URL: https://github.com/apache/airflow/issues/35144#issuecomment-1780931054 @langfu54 In your case you could try to set [`[core] max_num_rendered_ti_fields_per_task`](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#max-num-re

Re: [PR] Validate DAG task_instance.priority_weight column overflow value during import [airflow]

2023-10-26 Thread via GitHub
Taragolis commented on code in PR #34168: URL: https://github.com/apache/airflow/pull/34168#discussion_r1373022607 ## airflow/utils/dag_parameters_overflow.py: ## @@ -0,0 +1,56 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreem

Re: [PR] Replace blocking IO with async IO in AsyncKubernetesHook [airflow]

2023-10-26 Thread via GitHub
potiuk commented on PR #35162: URL: https://github.com/apache/airflow/pull/35162#issuecomment-1780950799 > Fixed in main. I rebased to see if it helps It does :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] AIP-58: Add Airflow ObjectStore (AFS) [airflow]

2023-10-26 Thread via GitHub
potiuk commented on PR #34729: URL: https://github.com/apache/airflow/pull/34729#issuecomment-1780955847 You will need to rebase again @bolkedebruin (this time pre-commit should automatically fix things for you for the breeze help images). also I've fixed the k8s_requirements.txt problem d

Re: [PR] The sanitize_for_serialization(cls, obj) function outputs an incorret time format [airflow-client-python]

2023-10-26 Thread via GitHub
junqilv commented on PR #99: URL: https://github.com/apache/airflow-client-python/pull/99#issuecomment-1780957830 My test steps are as follows: First, when I tried to pass data in datetime format to logical_date, I found that the http request returned showed that the format of logical_da

[PR] Add `YandexCloud` Managed Airflow service in `Ecosystem` page [airflow-site]

2023-10-26 Thread via GitHub
Piatachock opened a new pull request, #882: URL: https://github.com/apache/airflow-site/pull/882 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

Re: [PR] Replace blocking IO with async IO in AsyncKubernetesHook [airflow]

2023-10-26 Thread via GitHub
hussein-awala commented on PR #35162: URL: https://github.com/apache/airflow/pull/35162#issuecomment-1780979659 My problem was with the CI image too, I thought it was a conflict issue. For the change, I'm using a similar solution in my operators without any issues, so it's safe to mer

Re: [PR] Replace blocking IO with async IO in AsyncKubernetesHook [airflow]

2023-10-26 Thread via GitHub
hussein-awala merged PR #35162: URL: https://github.com/apache/airflow/pull/35162 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@ai

[airflow] branch main updated (86640d166c -> d4002261b5)

2023-10-26 Thread husseinawala
This is an automated email from the ASF dual-hosted git repository. husseinawala pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 86640d166c Add Http to s3 operator (#35176) add d4002261b5 Replace blocking IO with async IO in AsyncKuberne

[I] Marking task as "up_for_retry" when it is still running [airflow]

2023-10-26 Thread via GitHub
romanzdk opened a new issue, #35195: URL: https://github.com/apache/airflow/issues/35195 ### Apache Airflow version 2.7.2 ### What happened I have PythonOperator task that uses Paramiko to SSH into some machine, run docker container there and stream logs back to airflow.

[airflow-site] branch main updated: Add `YandexCloud` Managed Airflow service in `Ecosystem` page (#882)

2023-10-26 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow-site.git The following commit(s) were added to refs/heads/main by this push: new c405cb9e20 Add `YandexCloud` Managed Airflow s

Re: [PR] Add `YandexCloud` Managed Airflow service in `Ecosystem` page [airflow-site]

2023-10-26 Thread via GitHub
potiuk merged PR #882: URL: https://github.com/apache/airflow-site/pull/882 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.

Re: [PR] AIP-58: Add Airflow ObjectStore (AFS) [airflow]

2023-10-26 Thread via GitHub
bolkedebruin commented on PR #34729: URL: https://github.com/apache/airflow/pull/34729#issuecomment-1781038669 Afk at the moment but will do @potiuk . 🙏 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[airflow-site] branch gh-pages updated (121646f264 -> cbd15eea16)

2023-10-26 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch gh-pages in repository https://gitbox.apache.org/repos/asf/airflow-site.git discard 121646f264 Rewritten history to remove past gh-pages deployments new cbd15eea16 Rewritten history to

Re: [I] Tuples in xcom not being parsed as native objects in airflow 2.6.3 [airflow]

2023-10-26 Thread via GitHub
boring-cyborg[bot] commented on issue #35197: URL: https://github.com/apache/airflow/issues/35197#issuecomment-1781105267 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for ap

[I] Tuples in xcom not being parsed as native objects in airflow 2.6.3 [airflow]

2023-10-26 Thread via GitHub
gustavohwulee opened a new issue, #35197: URL: https://github.com/apache/airflow/issues/35197 ### Apache Airflow version Other Airflow 2 version (please specify below) ### What happened We upgraded from airflow 2.5.1 to airflow 2.6.3 and noticed that tuples returned in O

Re: [I] Marking task as "up_for_retry" when it is still running [airflow]

2023-10-26 Thread via GitHub
utkarsharma2 commented on issue #35195: URL: https://github.com/apache/airflow/issues/35195#issuecomment-1781124572 @romanzdk I looked at the code for Airflow 2.7.2 which handles task failure/retry and the only reason I see that the task would be retried is if it had failed. So, I suspect t

Re: [I] Invalid kube-config file. Expected key current-context in kube-config when using deferrable=True [airflow]

2023-10-26 Thread via GitHub
apilaskowski commented on issue #34644: URL: https://github.com/apache/airflow/issues/34644#issuecomment-1781149724 @tommyhutcheson what do you think about avoiding those files and providing Kube Config in JSON, i think it should be possible. Having blocking operations (eg. files handling)

Re: [PR] Remove offset-based pagination from `list_jobs` function in `DatabricksHook` [airflow]

2023-10-26 Thread via GitHub
oleksiidav commented on code in PR #34926: URL: https://github.com/apache/airflow/pull/34926#discussion_r1373212596 ## airflow/providers/databricks/CHANGELOG.rst: ## @@ -61,6 +61,13 @@ Misc 4.5.0 . +Breaking changes + + +The ``offset`` parameter has been

Re: [PR] Remove offset-based pagination from `list_jobs` function in `DatabricksHook` [airflow]

2023-10-26 Thread via GitHub
oleksiidav commented on PR #34926: URL: https://github.com/apache/airflow/pull/34926#issuecomment-1781171372 Thanks @eladkal ! Applied changes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Add OpenAI Provider [airflow]

2023-10-26 Thread via GitHub
utkarsharma2 commented on code in PR #35023: URL: https://github.com/apache/airflow/pull/35023#discussion_r1373220747 ## airflow/providers/openai/operators/openai.py: ## @@ -0,0 +1,86 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

Re: [PR] Improve testing harness to separate DB and non-db tests [airflow]

2023-10-26 Thread via GitHub
potiuk commented on code in PR #35160: URL: https://github.com/apache/airflow/pull/35160#discussion_r1373249661 ## Dockerfile.ci: ## @@ -1020,273 +1019,31 @@ if [[ ${DOWNGRADE_SQLALCHEMY=} == "true" ]]; then pip check fi +pip uninstall --root-user-action ignore "pytest-c

Re: [PR] Improve testing harness to separate DB and non-db tests [airflow]

2023-10-26 Thread via GitHub
potiuk commented on code in PR #35160: URL: https://github.com/apache/airflow/pull/35160#discussion_r1373249661 ## Dockerfile.ci: ## @@ -1020,273 +1019,31 @@ if [[ ${DOWNGRADE_SQLALCHEMY=} == "true" ]]; then pip check fi +pip uninstall --root-user-action ignore "pytest-c

Re: [PR] Improve testing harness to separate DB and non-db tests [airflow]

2023-10-26 Thread via GitHub
potiuk commented on code in PR #35160: URL: https://github.com/apache/airflow/pull/35160#discussion_r1373253135 ## Dockerfile.ci: ## @@ -1020,273 +1019,31 @@ if [[ ${DOWNGRADE_SQLALCHEMY=} == "true" ]]; then pip check fi +pip uninstall --root-user-action ignore "pytest-c

Re: [PR] Improve testing harness to separate DB and non-db tests [airflow]

2023-10-26 Thread via GitHub
potiuk commented on code in PR #35160: URL: https://github.com/apache/airflow/pull/35160#discussion_r1373258762 ## scripts/ci/docker-compose/backend-none.yml: ## @@ -15,19 +14,11 @@ # KIND, either express or implied. See the License for the # specific language governing permi

Re: [PR] Improve testing harness to separate DB and non-db tests [airflow]

2023-10-26 Thread via GitHub
potiuk commented on code in PR #35160: URL: https://github.com/apache/airflow/pull/35160#discussion_r1373260982 ## setup.cfg: ## @@ -101,7 +101,7 @@ install_requires = flask-session>=0.4.0 flask-wtf>=0.15 google-re2>=1.0 -graphviz>=0.12 +#graphviz>=0.12 R

Re: [PR] Improve testing harness to separate DB and non-db tests [airflow]

2023-10-26 Thread via GitHub
potiuk commented on code in PR #35160: URL: https://github.com/apache/airflow/pull/35160#discussion_r1373260407 ## setup.cfg: ## @@ -101,7 +101,7 @@ install_requires = flask-session>=0.4.0 flask-wtf>=0.15 google-re2>=1.0 -graphviz>=0.12 +#graphviz>=0.12 R

Re: [PR] Improve testing harness to separate DB and non-db tests [airflow]

2023-10-26 Thread via GitHub
potiuk commented on code in PR #35160: URL: https://github.com/apache/airflow/pull/35160#discussion_r1373261758 ## setup.py: ## @@ -310,7 +310,7 @@ def write_version(filename: str = str(AIRFLOW_SOURCES_ROOT / "airflow" / "git_ve # By adding a lot of whitespace separation.

Re: [PR] Improve testing harness to separate DB and non-db tests [airflow]

2023-10-26 Thread via GitHub
potiuk commented on code in PR #35160: URL: https://github.com/apache/airflow/pull/35160#discussion_r1373264647 ## tests/api_connexion/endpoints/test_dag_run_endpoint.py: ## @@ -599,12 +601,12 @@ class TestGetDagRunsEndDateFilters(TestDagRunEndpoint): [ (

Re: [PR] Improve testing harness to separate DB and non-db tests [airflow]

2023-10-26 Thread via GitHub
potiuk commented on code in PR #35160: URL: https://github.com/apache/airflow/pull/35160#discussion_r1373266881 ## tests/callbacks/test_callback_requests.py: ## @@ -32,11 +32,7 @@ from airflow.utils import timezone from airflow.utils.state import State -TI = TaskInstance( R

Re: [PR] Improve testing harness to separate DB and non-db tests [airflow]

2023-10-26 Thread via GitHub
potiuk commented on code in PR #35160: URL: https://github.com/apache/airflow/pull/35160#discussion_r1373270353 ## tests/providers/amazon/aws/hooks/test_base_aws.py: ## @@ -590,40 +590,40 @@ def import_mock(name, *args): [mock.call.get_default_id_token_credentials(

Re: [PR] Improve testing harness to separate DB and non-db tests [airflow]

2023-10-26 Thread via GitHub
potiuk commented on code in PR #35160: URL: https://github.com/apache/airflow/pull/35160#discussion_r1373270353 ## tests/providers/amazon/aws/hooks/test_base_aws.py: ## @@ -590,40 +590,40 @@ def import_mock(name, *args): [mock.call.get_default_id_token_credentials(

Re: [PR] Improve testing harness to separate DB and non-db tests [airflow]

2023-10-26 Thread via GitHub
potiuk commented on code in PR #35160: URL: https://github.com/apache/airflow/pull/35160#discussion_r1373274323 ## tests/providers/amazon/aws/hooks/test_batch_waiters.py: ## @@ -241,7 +241,7 @@ def describe_jobs_response(job_id: str = "mock-job-id", status: str = INTERMEDIA

Re: [PR] Improve testing harness to separate DB and non-db tests [airflow]

2023-10-26 Thread via GitHub
potiuk commented on code in PR #35160: URL: https://github.com/apache/airflow/pull/35160#discussion_r1373276572 ## tests/providers/apache/druid/hooks/test_druid.py: ## @@ -26,7 +26,8 @@ from airflow.providers.apache.druid.hooks.druid import DruidDbApiHook, DruidHook, Ingestion

Re: [PR] Improve testing harness to separate DB and non-db tests [airflow]

2023-10-26 Thread via GitHub
potiuk commented on code in PR #35160: URL: https://github.com/apache/airflow/pull/35160#discussion_r1373276572 ## tests/providers/apache/druid/hooks/test_druid.py: ## @@ -26,7 +26,8 @@ from airflow.providers.apache.druid.hooks.druid import DruidDbApiHook, DruidHook, Ingestion

[PR] docs: add Double.Cloud to list of Airflow as a Services [airflow-site]

2023-10-26 Thread via GitHub
epikhinm opened a new pull request, #883: URL: https://github.com/apache/airflow-site/pull/883 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] Improve testing harness to separate DB and non-db tests [airflow]

2023-10-26 Thread via GitHub
potiuk commented on code in PR #35160: URL: https://github.com/apache/airflow/pull/35160#discussion_r1373305830 ## tests/providers/amazon/aws/utils/test_connection_wrapper.py: ## @@ -27,14 +28,19 @@ from airflow.models import Connection from airflow.providers.amazon.aws.utils.

[I] Grid view shows wrong task status [airflow]

2023-10-26 Thread via GitHub
atacank opened a new issue, #35198: URL: https://github.com/apache/airflow/issues/35198 ### Apache Airflow version Other Airflow 2 version (please specify below) ### What happened Apache Airflow version==2.6.3 When a DAG runs, grid view doesn't show the right statu

Re: [I] Grid view shows wrong task status [airflow]

2023-10-26 Thread via GitHub
boring-cyborg[bot] commented on issue #35198: URL: https://github.com/apache/airflow/issues/35198#issuecomment-1781294181 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for ap

Re: [I] Changelog omits important change to `dag.test()` behavior in 2.7.0 [airflow]

2023-10-26 Thread via GitHub
benbuckman commented on issue #34490: URL: https://github.com/apache/airflow/issues/34490#issuecomment-1781299141 Bump Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] Improve testing harness to separate DB and non-db tests [airflow]

2023-10-26 Thread via GitHub
vincbeck commented on code in PR #35160: URL: https://github.com/apache/airflow/pull/35160#discussion_r1373274314 ## dev/breeze/src/airflow_breeze/commands/testing_commands.py: ## @@ -250,6 +276,35 @@ def _run_tests_in_pool( skip_docker_compose_down: bool, ): escaped_

Re: [I] Trigger Rule ONE_FAILED does not work in task group with mapped tasks [airflow]

2023-10-26 Thread via GitHub
benbuckman commented on issue #34023: URL: https://github.com/apache/airflow/issues/34023#issuecomment-1781299862 Any updates on this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] Track SparkSubmitHook Yarn Cluster application with Yarn CLI [airflow]

2023-10-26 Thread via GitHub
tokoko commented on issue #24171: URL: https://github.com/apache/airflow/issues/24171#issuecomment-1781312438 Depends on what you mean exactly. YARN always exposes rest api, it just might or might not be accessible from airflow worker because of network security policies. The address can be

Re: [PR] Improve testing harness to separate DB and non-db tests [airflow]

2023-10-26 Thread via GitHub
potiuk commented on code in PR #35160: URL: https://github.com/apache/airflow/pull/35160#discussion_r1373345164 ## .github/workflows/ci.yml: ## @@ -1027,7 +1025,7 @@ jobs: uses: ./.github/actions/migration_tests - name: > Tests: ${{matrix.python-versi

Re: [PR] Improve testing harness to separate DB and non-db tests [airflow]

2023-10-26 Thread via GitHub
potiuk commented on code in PR #35160: URL: https://github.com/apache/airflow/pull/35160#discussion_r1373384808 ## dev/breeze/src/airflow_breeze/commands/testing_commands.py: ## @@ -250,6 +276,35 @@ def _run_tests_in_pool( skip_docker_compose_down: bool, ): escaped_te

Re: [PR] Improve testing harness to separate DB and non-db tests [airflow]

2023-10-26 Thread via GitHub
potiuk commented on code in PR #35160: URL: https://github.com/apache/airflow/pull/35160#discussion_r1373385808 ## dev/breeze/src/airflow_breeze/global_constants.py: ## @@ -44,7 +44,7 @@ ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS = ["3.8", "3.9", "3.10", "3.11"] DEFAULT_PYTHON_MAJOR_

Re: [PR] fix(bigquery.py): pass correct project_id to triggerer [airflow]

2023-10-26 Thread via GitHub
boring-cyborg[bot] commented on PR #35200: URL: https://github.com/apache/airflow/pull/35200#issuecomment-1781408625 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution

[PR] fix(bigquery.py): pass correct project_id to triggerer [airflow]

2023-10-26 Thread via GitHub
mokshasoul opened a new pull request, #35200: URL: https://github.com/apache/airflow/pull/35200 The triggerer uses a different project than the InsertJobOperator when the `project_id` passed to the operator = None -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] Improve testing harness to separate DB and non-db tests [airflow]

2023-10-26 Thread via GitHub
potiuk commented on PR #35160: URL: https://github.com/apache/airflow/pull/35160#issuecomment-1781422687 (addressed all comments @vincbeck - still working on getting it all green but with the feedback it already got a bit better). -- This is an automated message from the Apache Git Servic

[I] [helm] default broker url doesn't match Redis service name [airflow]

2023-10-26 Thread via GitHub
jiajie-chen opened a new issue, #35201: URL: https://github.com/apache/airflow/issues/35201 ### Official Helm Chart version 1.11.0 (latest released) ### Apache Airflow version v2.7.1 ### Kubernetes Version v1.27.4 (minikube v1.31.2) ### Helm Chart con

Re: [I] [helm] default broker url doesn't match Redis service name [airflow]

2023-10-26 Thread via GitHub
boring-cyborg[bot] commented on issue #35201: URL: https://github.com/apache/airflow/issues/35201#issuecomment-1781440310 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for ap

[I] The "log" attribute appears to be missing from BaseOperator [airflow]

2023-10-26 Thread via GitHub
gwerbin opened a new issue, #35202: URL: https://github.com/apache/airflow/issues/35202 ### What do you see as an issue? In the doc page for [BaseOperator](https://airflow.apache.org/docs/apache-airflow/2.7.2/_api/airflow/models/baseoperator/index.html), the `logger` property is docu

Re: [I] The "log" attribute appears to be missing from BaseOperator [airflow]

2023-10-26 Thread via GitHub
boring-cyborg[bot] commented on issue #35202: URL: https://github.com/apache/airflow/issues/35202#issuecomment-1781459804 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for ap

Re: [I] The "log" attribute appears to be missing from BaseOperator [airflow]

2023-10-26 Thread via GitHub
jens-scheffler-bosch commented on issue #35202: URL: https://github.com/apache/airflow/issues/35202#issuecomment-1781502682 I do not understand whar you mean by `log` property. But APIdocs are generated from source, if you want to fix or extend them just open a PR on https://github.com/apac

Re: [I] Node label/capability based scheduling (for CeleryExecutor) [airflow]

2023-10-26 Thread via GitHub
jens-scheffler-bosch commented on issue #35174: URL: https://github.com/apache/airflow/issues/35174#issuecomment-1781515443 Such feature is indirectly supported with a single "label" using the "queue" feature. You can assign tasks to "queues" and workers can pull work from one or multiple q

Re: [I] Trigger Rule ONE_FAILED does not work in task group with mapped tasks [airflow]

2023-10-26 Thread via GitHub
ephraimbuddy commented on issue #34023: URL: https://github.com/apache/airflow/issues/34023#issuecomment-1781531694 > Any updates on this? The PR is awaiting reviews -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

[PR] Move methods used only in FAB from security managers to auth manager [airflow]

2023-10-26 Thread via GitHub
vincbeck opened a new pull request, #35203: URL: https://github.com/apache/airflow/pull/35203 Move methods used only in FAB from security managers to auth manager. I also simplified the way the security manager is instantiated. --- **^ Add meaningful des

[I] Mysterious hanging of the test_retry_handling_job for sqlite on self-hosted/local env [airflow]

2023-10-26 Thread via GitHub
potiuk opened a new issue, #35204: URL: https://github.com/apache/airflow/issues/35204 ### Body ## Problem The test in question: ``` tests/jobs/test_scheduler_job.py::TestSchedulerJob::test_retry_handling_job ``` Started to timeout - mysteriously - on Octobe

Re: [I] Tuples in xcom not being parsed as native objects in airflow 2.6.3 (breaks DmsDescribeTasksOperator) [airflow]

2023-10-26 Thread via GitHub
Taragolis commented on issue #35197: URL: https://github.com/apache/airflow/issues/35197#issuecomment-1781565583 Unable to reproduce locally on main branch. ```console [2023-10-26, 17:41:02 UTC] {logging_mixin.py:154} INFO - Dict Type: [2023-10-26, 17:41:02 UTC] {logging_mixin.p

[PR] Temporary disable test_retry_handling_job test for sqlite [airflow]

2023-10-26 Thread via GitHub
potiuk opened a new pull request, #35206: URL: https://github.com/apache/airflow/pull/35206 This test started To fail ONLY on sqlite but only on self-hosted runners and locally (not on public runners) We should uncomment it when we figure out what's going on. Related: #35204

[PR] Use `requires_access_custom_view` for user and roles API endpoints [airflow]

2023-10-26 Thread via GitHub
vincbeck opened a new pull request, #35207: URL: https://github.com/apache/airflow/pull/35207 This PR sits on top of #35000, please review the last commit only. This PR removes the deprecation warnings you get when you start Airflow: ``` /opt/airflow/airflow/auth/managers/fab

Re: [PR] Temporary disable test_retry_handling_job test for sqlite [airflow]

2023-10-26 Thread via GitHub
potiuk merged PR #35206: URL: https://github.com/apache/airflow/pull/35206 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[airflow] branch main updated: Temporary disable test_retry_handling_job test for sqlite (#35206)

2023-10-26 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new fe360cb11d Temporary disable test_retry_handling_jo

Re: [I] Node label/capability based scheduling (for CeleryExecutor) [airflow]

2023-10-26 Thread via GitHub
jens-scheffler-bosch commented on issue #35174: URL: https://github.com/apache/airflow/issues/35174#issuecomment-1781601318 To map the current queue features to your demand, a workaround to implement the demand you are raising your be: * Node 1: Listen to queue `linux` and `x86` *

Re: [I] Tuples in xcom not being parsed as native objects in airflow 2.6.3 (breaks DmsDescribeTasksOperator) [airflow]

2023-10-26 Thread via GitHub
Taragolis commented on issue #35197: URL: https://github.com/apache/airflow/issues/35197#issuecomment-1781641755 The same example works fine on 2.6.3 ```console [2023-10-26, 18:28:42 UTC] {logging_mixin.py:150} INFO - Dict Type: [2023-10-26, 18:28:42 UTC] {logging_mixin.py:150}

Re: [I] Tuples in xcom not being parsed as native objects in airflow 2.6.3 (breaks DmsDescribeTasksOperator) [airflow]

2023-10-26 Thread via GitHub
Taragolis closed issue #35197: Tuples in xcom not being parsed as native objects in airflow 2.6.3 (breaks DmsDescribeTasksOperator) URL: https://github.com/apache/airflow/issues/35197 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

Re: [PR] Prevent assignment of non JSON serializable values to DagRun.conf dict [airflow]

2023-10-26 Thread via GitHub
jens-scheffler-bosch commented on PR #35096: URL: https://github.com/apache/airflow/pull/35096#issuecomment-1781676216 > > This looks good. Thanks for the contribution. > > I was thinging (but have no strong opinion) as whether it makes sense to extract this to a utility, but keeping it h

  1   2   >