[GitHub] [airflow] tanelk commented on pull request #21316: Fix race condition between triggerer and scheduler

2022-04-11 Thread GitBox
tanelk commented on PR #21316: URL: https://github.com/apache/airflow/pull/21316#issuecomment-1094625615 A very late "review". `CeleryExecutor` overwrites the `trigger_tasks` and this fix does not help when using `celery`. -- This is an automated message from th

[GitHub] [airflow] boring-cyborg[bot] commented on issue #22895: Airflow health API is returning always 200

2022-04-11 Thread GitBox
boring-cyborg[bot] commented on issue #22895: URL: https://github.com/apache/airflow/issues/22895#issuecomment-1094642017 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the mess

[GitHub] [airflow] gitruhul opened a new issue, #22895: Airflow health API is returning always 200

2022-04-11 Thread GitBox
gitruhul opened a new issue, #22895: URL: https://github.com/apache/airflow/issues/22895 ### Apache Airflow version 2.2.4 ### What happened The health endpoint of airflow (base_url/health) is always returning 200 in all the scenarios (healthy+unhealthy). Same info

[GitHub] [airflow] malthe commented on pull request #21316: Fix race condition between triggerer and scheduler

2022-04-11 Thread GitBox
malthe commented on PR #21316: URL: https://github.com/apache/airflow/pull/21316#issuecomment-1094643953 @tanelk nice find – it seems like the Celery executor has its own retrying logic in `task_publish_retries` which combined with this pull request ends up still solving the problem.

[GitHub] [airflow] uranusjr commented on issue #22895: Airflow health API is returning always 200

2022-04-11 Thread GitBox
uranusjr commented on issue #22895: URL: https://github.com/apache/airflow/issues/22895#issuecomment-1094648857 I don’t think we can change this, if we return anything other than 200 it risks breaking existing usages. Also, what would you propose using instead? Honestly 200 feels like the m

[GitHub] [airflow] gitruhul commented on issue #22895: Airflow health API is returning always 200

2022-04-11 Thread GitBox
gitruhul commented on issue #22895: URL: https://github.com/apache/airflow/issues/22895#issuecomment-1094652723 Creating a template for the health response will help with different standards. For example, some organizations prefer health responses as ``` { "status": "un

[GitHub] [airflow] eskarimov commented on pull request #19736: Add Databricks Deferrable Operators

2022-04-11 Thread GitBox
eskarimov commented on PR #19736: URL: https://github.com/apache/airflow/pull/19736#issuecomment-1094655463 There were some conflicts, resolved them and rebased. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [airflow] gitruhul commented on issue #22895: Airflow health API is returning always 200

2022-04-11 Thread GitBox
gitruhul commented on issue #22895: URL: https://github.com/apache/airflow/issues/22895#issuecomment-1094663558 Hi @uranusjr , Thanks for your response. Current health API is just like API ping. whereas health checks should be superior to that. When I see some standards (Including my o

[GitHub] [airflow] eladkal opened a new pull request, #22896: update INTHEWILD

2022-04-11 Thread GitBox
eladkal opened a new pull request, #22896: URL: https://github.com/apache/airflow/pull/22896 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more inf

[GitHub] [airflow] github-actions[bot] commented on pull request #22896: update INTHEWILD

2022-04-11 Thread GitBox
github-actions[bot] commented on PR #22896: URL: https://github.com/apache/airflow/pull/22896#issuecomment-1094667835 The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full

[airflow] branch main updated: update INTHEWILD (#22896)

2022-04-11 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 4201e6e0da update INTHEWILD (#22896) 4201e6e0da is

[GitHub] [airflow] eladkal merged pull request #22896: update INTHEWILD

2022-04-11 Thread GitBox
eladkal merged PR #22896: URL: https://github.com/apache/airflow/pull/22896 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.

[GitHub] [airflow] tirkarthi opened a new issue, #22897: Invalid JSON metadata in get_logs_with_metadata causes server error.

2022-04-11 Thread GitBox
tirkarthi opened a new issue, #22897: URL: https://github.com/apache/airflow/issues/22897 ### Apache Airflow version 2.2.5 (latest released) ### What happened Invalid JSON metadata in get_logs_with_metadata causes server error. The `json.loads` exception is not handled l

[GitHub] [airflow] boring-cyborg[bot] commented on issue #22897: Invalid JSON metadata in get_logs_with_metadata causes server error.

2022-04-11 Thread GitBox
boring-cyborg[bot] commented on issue #22897: URL: https://github.com/apache/airflow/issues/22897#issuecomment-1094677982 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the mess

[GitHub] [airflow] tirkarthi opened a new pull request, #22898: Handle invalid JSON metadata in get_logs_with_metadata endpoint.

2022-04-11 Thread GitBox
tirkarthi opened a new pull request, #22898: URL: https://github.com/apache/airflow/pull/22898 closes: #22897 related: #22897 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

[GitHub] [airflow] github-actions[bot] commented on pull request #22894: Add test case for clearTaskInstance API call with invalid list of Task IDs.

2022-04-11 Thread GitBox
github-actions[bot] commented on PR #22894: URL: https://github.com/apache/airflow/pull/22894#issuecomment-1094690349 The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the

[GitHub] [airflow] uranusjr commented on issue #21584: xcom table deadlocks (error 1205) due to lack of index

2022-04-11 Thread GitBox
uranusjr commented on issue #21584: URL: https://github.com/apache/airflow/issues/21584#issuecomment-1094697479 Hmm I think the situation does not change much. We do have a separate index now that can make this work better, but using that instead would only flip the problem to if you have a

[GitHub] [airflow] pastushenkoy commented on issue #22842: Environment variables for elastic search logging have no effect

2022-04-11 Thread GitBox
pastushenkoy commented on issue #22842: URL: https://github.com/apache/airflow/issues/22842#issuecomment-1094702538 That's much clearer now. Thant you very much. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [airflow] potiuk closed issue #18578: Allow to set execution_timeout default value in airflow.cfg

2022-04-11 Thread GitBox
potiuk closed issue #18578: Allow to set execution_timeout default value in airflow.cfg URL: https://github.com/apache/airflow/issues/18578 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #22389: make operator's execution_timeout configurable

2022-04-11 Thread GitBox
boring-cyborg[bot] commented on PR #22389: URL: https://github.com/apache/airflow/pull/22389#issuecomment-1094706391 Awesome work, congrats on your first merged pull request! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [airflow] potiuk merged pull request #22389: make operator's execution_timeout configurable

2022-04-11 Thread GitBox
potiuk merged PR #22389: URL: https://github.com/apache/airflow/pull/22389 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[airflow] branch main updated (4201e6e0da -> a111a79c0d)

2022-04-11 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 4201e6e0da update INTHEWILD (#22896) add a111a79c0d make operator's execution_timeout configurable (#22389) No ne

[GitHub] [airflow] potiuk commented on issue #22895: Airflow health API is returning always 200

2022-04-11 Thread GitBox
potiuk commented on issue #22895: URL: https://github.com/apache/airflow/issues/22895#issuecomment-1094708235 We should create a NEW health endpoint for that not change the existing one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

[GitHub] [airflow] potiuk commented on issue #22895: Airflow health API is returning always 200

2022-04-11 Thread GitBox
potiuk commented on issue #22895: URL: https://github.com/apache/airflow/issues/22895#issuecomment-1094709511 @uranusjr is right - this is a breaking change and changing it to fit someone's standard (even if it's a reasonable and desired state) does not justify breaking existing endpoint's

[GitHub] [airflow] boring-cyborg[bot] commented on issue #22899: Airflow not showing StatsD metrics

2022-04-11 Thread GitBox
boring-cyborg[bot] commented on issue #22899: URL: https://github.com/apache/airflow/issues/22899#issuecomment-1094710514 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the mess

[GitHub] [airflow] DibyoGit opened a new issue, #22899: Airflow not showing StatsD metrics

2022-04-11 Thread GitBox
DibyoGit opened a new issue, #22899: URL: https://github.com/apache/airflow/issues/22899 ### Apache Airflow version 2.1.0 ### What happened We are running the Airflow via docker-compose, connecting to a MySQL host. It hs Statd Exporter config in it but we can not see ny

[GitHub] [airflow] talnagar opened a new pull request, #22900: add auto refresh to dags home page

2022-04-11 Thread GitBox
talnagar opened a new pull request, #22900: URL: https://github.com/apache/airflow/pull/22900 Add auto refresh functionality to the home page (dags). closes: #22418 --- refresh every 2 seconds: - last dag runs - last run - recent tasks --- before: ![image](

[GitHub] [airflow] talnagar commented on pull request #22900: add auto refresh to dags home page

2022-04-11 Thread GitBox
talnagar commented on PR #22900: URL: https://github.com/apache/airflow/pull/22900#issuecomment-1094742010 @bbovenzi I would appreciate your feedback on this. In addition, I didn't know whether there are any tests relevant for ui only changes such as this. thanks. -- This is an au

[GitHub] [airflow] uranusjr commented on issue #21584: xcom table deadlocks (error 1205) due to lack of index

2022-04-11 Thread GitBox
uranusjr commented on issue #21584: URL: https://github.com/apache/airflow/issues/21584#issuecomment-1094752046 Let’s see how 2.3 works in the real world. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[GitHub] [airflow] Bowrna commented on issue #21105: Breeze: Setting up local virtualenv

2022-04-11 Thread GitBox
Bowrna commented on issue #21105: URL: https://github.com/apache/airflow/issues/21105#issuecomment-1094760707 @joppevos **Setting up environment for developing breeze:** we set the environment for developing breeze by setting venv as `pip install -e dev/breeze/` from airflow_

[GitHub] [airflow] uranusjr merged pull request #22894: Add test case for clearTaskInstance call with invalid Task IDs.

2022-04-11 Thread GitBox
uranusjr merged PR #22894: URL: https://github.com/apache/airflow/pull/22894 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

[airflow] branch main updated: Add test case for clearTaskInstance call with invalid Task IDs. (#22894)

2022-04-11 Thread uranusjr
This is an automated email from the ASF dual-hosted git repository. uranusjr pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 1a8b8f521c Add test case for clearTaskInstance ca

[GitHub] [airflow] ashb closed issue #22833: Allow mapped Task as input to another mapped task

2022-04-11 Thread GitBox
ashb closed issue #22833: Allow mapped Task as input to another mapped task URL: https://github.com/apache/airflow/issues/22833 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [airflow] ashb merged pull request #22849: Allow using mapped upstream's aggregated XCom

2022-04-11 Thread GitBox
ashb merged PR #22849: URL: https://github.com/apache/airflow/pull/22849 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apa

[airflow] branch main updated: Allow using mapped upstream's aggregated XCom (#22849)

2022-04-11 Thread ash
This is an automated email from the ASF dual-hosted git repository. ash pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 8af77127f1 Allow using mapped upstream's aggregated XC

[GitHub] [airflow] github-actions[bot] commented on pull request #22898: Handle invalid JSON metadata in get_logs_with_metadata endpoint.

2022-04-11 Thread GitBox
github-actions[bot] commented on PR #22898: URL: https://github.com/apache/airflow/pull/22898#issuecomment-1094816150 The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the

[GitHub] [airflow] joppevos opened a new pull request, #22901: Add exception in Breeze when docker is not running

2022-04-11 Thread GitBox
joppevos opened a new pull request, #22901: URL: https://github.com/apache/airflow/pull/22901 Running the new python Breeze environment crashes, if Docker is not running. This PR will give a descriptive error. Traceback of the error: ``` File "/Users/vosjoppe/github/air

[GitHub] [airflow] joppevos commented on pull request #22901: Add exception in Breeze when docker is not running

2022-04-11 Thread GitBox
joppevos commented on PR #22901: URL: https://github.com/apache/airflow/pull/22901#issuecomment-1094825881 We could also stop breeze from starting up with an exception like so ```python raise Exception("No running Docker instance found") ``` But I think it is clear enough to t

[GitHub] [airflow] Bowrna commented on pull request #22876: cache and typo fix

2022-04-11 Thread GitBox
Bowrna commented on PR #22876: URL: https://github.com/apache/airflow/pull/22876#issuecomment-1094830282 @potiuk @uranusjr This is the error in static check. This error doesn't occur when i run pre-commit locally. But this occurs in CI. I am not sure how to fix it. ``` Upd

[GitHub] [airflow] uranusjr commented on pull request #22901: Catch error in Breeze when docker is not running

2022-04-11 Thread GitBox
uranusjr commented on PR #22901: URL: https://github.com/apache/airflow/pull/22901#issuecomment-1094831354 Personally I’d prefer breeze to end early. There’s a bunch of output (including a giant ASCII part) after this check and we could avoid all those easily. -- This is an automated mes

[GitHub] [airflow] ashb commented on issue #22626: The command 'airflow dags test' is deadlocking tasks using AIP-42 features with the PythonVirtualenvOperator

2022-04-11 Thread GitBox
ashb commented on issue #22626: URL: https://github.com/apache/airflow/issues/22626#issuecomment-1094834071 This _almost_ passes now. The Tasks run, but right at the end of the `test` command we have this failure ``` [2022-04-11 10:49:28,894] 2990124 MainProcess {{airflow.models.

[GitHub] [airflow] uranusjr commented on pull request #22876: cache and typo fix

2022-04-11 Thread GitBox
uranusjr commented on PR #22876: URL: https://github.com/apache/airflow/pull/22876#issuecomment-1094835825 This seems to be caused by #22880; I think it’s because breeze is not installed in the CI environment. -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] uranusjr commented on issue #22626: The command 'airflow dags test' is deadlocking tasks using AIP-42 features with the PythonVirtualenvOperator

2022-04-11 Thread GitBox
uranusjr commented on issue #22626: URL: https://github.com/apache/airflow/issues/22626#issuecomment-1094839148 Ah, I think I know what’s going on. We made changes in the scheduler, but didn’t update BackFillJob to match it. I should be able to fix it pretty quickly, but what would be

[GitHub] [airflow] bhirsz opened a new pull request, #22903: Update codeowners for Google provider

2022-04-11 Thread GitBox
bhirsz opened a new pull request, #22903: URL: https://github.com/apache/airflow/pull/22903 @potiuk (PR to test if it's possible to update codeowner with someone who don't have write access) It appears that there are errors when trying to update codeowners with user who don't

[GitHub] [airflow] tirkarthi commented on pull request #22894: Add test case for clearTaskInstance call with invalid Task IDs.

2022-04-11 Thread GitBox
tirkarthi commented on PR #22894: URL: https://github.com/apache/airflow/pull/22894#issuecomment-1094848472 Thanks @ephraimbuddy, @eladkal and @uranusjr -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [airflow] joppevos commented on pull request #22901: Catch error in Breeze when docker is not running

2022-04-11 Thread GitBox
joppevos commented on PR #22901: URL: https://github.com/apache/airflow/pull/22901#issuecomment-1094852169 > Personally I’d prefer breeze to end early. There’s a bunch of output (including a giant ASCII part) after this check and we could avoid all those easily. You are right. It is

[airflow] branch constraints-main updated: Updating constraints. Build id:2147589387

2022-04-11 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 62553a6be0 Updating con

[GitHub] [airflow] uranusjr opened a new pull request, #22904: Call mapped_dependants only on the original task

2022-04-11 Thread GitBox
uranusjr opened a new pull request, #22904: URL: https://github.com/apache/airflow/pull/22904 We've made change on this in the scheduler, but need to match it in the BackfillJob. Should be the last piece to fix #22626. -- This is an automated message from the Apache Git Service. To

[GitHub] [airflow] uranusjr commented on a diff in pull request #22901: Catch error in Breeze when docker is not running

2022-04-11 Thread GitBox
uranusjr commented on code in PR #22901: URL: https://github.com/apache/airflow/pull/22901#discussion_r847173433 ## dev/breeze/src/airflow_breeze/utils/docker_command_utils.py: ## @@ -179,11 +179,7 @@ def check_docker_version(verbose: bool): text=True, )

[GitHub] [airflow] tothandor commented on issue #13542: Task stuck in "scheduled" or "queued" state, pool has all slots queued, nothing is executing

2022-04-11 Thread GitBox
tothandor commented on issue #13542: URL: https://github.com/apache/airflow/issues/13542#issuecomment-1094889441 Hi All! With release 2.2.5 scheduling issues have gone away for me. I guess one of the following resolved issues made it. 1) Fix max_active_runs=1 not scheduling runs

[GitHub] [airflow] uranusjr commented on a diff in pull request #22003: Generate release notes for core and the helm chart with towncrier

2022-04-11 Thread GitBox
uranusjr commented on code in PR #22003: URL: https://github.com/apache/airflow/pull/22003#discussion_r847185910 ## .dockerignore: ## @@ -49,7 +49,7 @@ !.flake8 !.dockerignore !pytest.ini -!CHANGELOG.txt +!RELEASE_NOTES.txt Review Comment: This should be `RELEASE_NOTES.rs

[GitHub] [airflow] uranusjr commented on pull request #21797: Fixed dag level access permissions

2022-04-11 Thread GitBox
uranusjr commented on PR #21797: URL: https://github.com/apache/airflow/pull/21797#issuecomment-1094896455 Spltting out and fixing the first part should be doable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

[GitHub] [airflow] potiuk merged pull request #22898: Handle invalid JSON metadata in get_logs_with_metadata endpoint.

2022-04-11 Thread GitBox
potiuk merged PR #22898: URL: https://github.com/apache/airflow/pull/22898 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[GitHub] [airflow] potiuk closed issue #22897: Invalid JSON metadata in get_logs_with_metadata causes server error.

2022-04-11 Thread GitBox
potiuk closed issue #22897: Invalid JSON metadata in get_logs_with_metadata causes server error. URL: https://github.com/apache/airflow/issues/22897 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[airflow] branch main updated: Handle invalid JSON metadata in get_logs_with_metadata endpoint. (#22898)

2022-04-11 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new a3dd8473e4 Handle invalid JSON metadata in get_logs

[GitHub] [airflow] uranusjr commented on issue #21923: Programmatic customization of run_id for scheduled DagRuns

2022-04-11 Thread GitBox
uranusjr commented on issue #21923: URL: https://github.com/apache/airflow/issues/21923#issuecomment-1094899126 One possible solution would be to allow a DAG argument `run_id_format` for Airflow to use instead, and the user can set this to whatever they want (within some limitations of cour

[GitHub] [airflow] potiuk commented on a diff in pull request #22901: Catch error in Breeze when docker is not running

2022-04-11 Thread GitBox
potiuk commented on code in PR #22901: URL: https://github.com/apache/airflow/pull/22901#discussion_r847195377 ## dev/breeze/src/airflow_breeze/utils/docker_command_utils.py: ## @@ -179,11 +179,7 @@ def check_docker_version(verbose: bool): text=True, )

[GitHub] [airflow] potiuk commented on a diff in pull request #22901: Catch error in Breeze when docker is not running

2022-04-11 Thread GitBox
potiuk commented on code in PR #22901: URL: https://github.com/apache/airflow/pull/22901#discussion_r847195377 ## dev/breeze/src/airflow_breeze/utils/docker_command_utils.py: ## @@ -179,11 +179,7 @@ def check_docker_version(verbose: bool): text=True, )

[GitHub] [airflow] potiuk opened a new issue, #22906: Breeze: Describe approach of communicating with users.

2022-04-11 Thread GitBox
potiuk opened a new issue, #22906: URL: https://github.com/apache/airflow/issues/22906 We should add custom styles for Breeze (Info/error/warning/guidance/etc.) and add an ADR with rules describing when we should use them. See discussion in https://github.com/apache/airflow/pull/2290

[GitHub] [airflow] potiuk commented on a diff in pull request #22901: Catch error in Breeze when docker is not running

2022-04-11 Thread GitBox
potiuk commented on code in PR #22901: URL: https://github.com/apache/airflow/pull/22901#discussion_r847197689 ## dev/breeze/src/airflow_breeze/utils/docker_command_utils.py: ## @@ -179,11 +179,7 @@ def check_docker_version(verbose: bool): text=True, )

[GitHub] [airflow] potiuk commented on pull request #22901: Catch error in Breeze when docker is not running

2022-04-11 Thread GitBox
potiuk commented on PR #22901: URL: https://github.com/apache/airflow/pull/22901#issuecomment-1094905422 BTW. Cool that we start gettiing those :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] potiuk commented on pull request #22876: cache and typo fix

2022-04-11 Thread GitBox
potiuk commented on PR #22876: URL: https://github.com/apache/airflow/pull/22876#issuecomment-1094908101 Rebase. We have now completely different pre-commit in place of updating the help output. Instead we generate nice .svg output (using rich + rich-click + rich built-in svg export

[GitHub] [airflow] potiuk commented on pull request #22903: Update codeowners for Google provider

2022-04-11 Thread GitBox
potiuk commented on PR #22903: URL: https://github.com/apache/airflow/pull/22903#issuecomment-1094909027 Opened a support ticket in GitHub for that one: https://support.github.com/ticket/personal/0/1581140 -- This is an automated message from the Apache Git Service. To respond to the mess

[GitHub] [airflow] mik-laj commented on issue #22895: Airflow health API is returning always 200

2022-04-11 Thread GitBox
mik-laj commented on issue #22895: URL: https://github.com/apache/airflow/issues/22895#issuecomment-1094920008 We shouldn't change the code when the scheduler isn't working. This endpoint should only checks health of webserver and required components - metabase. Otherwise the web server wil

[GitHub] [airflow] ashb commented on a diff in pull request #22867: Add concept doc for Dynamic Task Mapping

2022-04-11 Thread GitBox
ashb commented on code in PR #22867: URL: https://github.com/apache/airflow/pull/22867#discussion_r847223664 ## docs/apache-airflow/concepts/dynamic-task-mapping.rst: ## @@ -0,0 +1,260 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more contributor li

[airflow] branch constraints-main updated: Updating constraints. Build id:2147981766

2022-04-11 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new c795c5509b Updating con

[GitHub] [airflow] kotaaaa opened a new issue, #22907: Add SKIP process in addition to occurring ERROR in SQLCheckOperator

2022-04-11 Thread GitBox
kot opened a new issue, #22907: URL: https://github.com/apache/airflow/issues/22907 ### Description In [SQLCheckOperator](https://airflow.apache.org/docs/apache-airflow/1.10.12/_api/airflow/operators/sql/index.html#airflow.operators.sql.SQLCheckOperator), I propose to add a f

[GitHub] [airflow] boring-cyborg[bot] commented on issue #22907: Add SKIP process in addition to occurring ERROR in SQLCheckOperator

2022-04-11 Thread GitBox
boring-cyborg[bot] commented on issue #22907: URL: https://github.com/apache/airflow/issues/22907#issuecomment-1094948322 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the mess

[GitHub] [airflow] joppevos commented on a diff in pull request #22901: Catch error in Breeze when docker is not running

2022-04-11 Thread GitBox
joppevos commented on code in PR #22901: URL: https://github.com/apache/airflow/pull/22901#discussion_r847235044 ## dev/breeze/src/airflow_breeze/utils/docker_command_utils.py: ## @@ -179,11 +179,7 @@ def check_docker_version(verbose: bool): text=True, )

[GitHub] [airflow] tirkarthi commented on pull request #22898: Handle invalid JSON metadata in get_logs_with_metadata endpoint.

2022-04-11 Thread GitBox
tirkarthi commented on PR #22898: URL: https://github.com/apache/airflow/pull/22898#issuecomment-1094952010 Thanks @potiuk and @uranusjr -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

[GitHub] [airflow] eladkal commented on issue #22907: Add SKIP process in addition to occurring ERROR in SQLCheckOperator

2022-04-11 Thread GitBox
eladkal commented on issue #22907: URL: https://github.com/apache/airflow/issues/22907#issuecomment-1094958083 This feels more like a custom use case that you should solve locally in your deployment. However, should this be something needed then this should be solved in `BaseOprator`.

[GitHub] [airflow] ashb commented on pull request #22904: Call mapped_dependants only on the original task

2022-04-11 Thread GitBox
ashb commented on PR #22904: URL: https://github.com/apache/airflow/pull/22904#issuecomment-1094959233 We should be able to add a test for this in test_backfilljob somewhere? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [airflow] ashb commented on pull request #22904: Call mapped_dependants only on the original task

2022-04-11 Thread GitBox
ashb commented on PR #22904: URL: https://github.com/apache/airflow/pull/22904#issuecomment-1094981510 ```diff diff --git a/tests/dags/test_mapped_classic.py b/tests/dags/test_mapped_classic.py index 3880cc74f..4c0b4b1e7 100644 --- a/tests/dags/test_mapped_classic.py +++ b/tests/

[GitHub] [airflow] kotaaaa commented on issue #22907: Add SKIP process in addition to occurring ERROR in SQLCheckOperator

2022-04-11 Thread GitBox
kot commented on issue #22907: URL: https://github.com/apache/airflow/issues/22907#issuecomment-1094989998 @eladkal, thank you for your reply! For my case, process is folked by the result of `select count(*) from {bigquery's table}` so, I created class like below. In not only my

[GitHub] [airflow] tanelk commented on issue #19957: Airflow crashes with a psycopg2.errors.DeadlockDetected exception

2022-04-11 Thread GitBox
tanelk commented on issue #19957: URL: https://github.com/apache/airflow/issues/19957#issuecomment-1094995228 In yout last log you have: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

[GitHub] [airflow] Bowrna commented on pull request #22876: cache and typo fix

2022-04-11 Thread GitBox
Bowrna commented on PR #22876: URL: https://github.com/apache/airflow/pull/22876#issuecomment-1094995271 Could you explain more about .svg @potiuk ? You could share any relevant docs or point to links to know more about this. -- This is an automated message from the Apache Git Service. T

[GitHub] [airflow] tanelk commented on issue #19957: Airflow crashes with a psycopg2.errors.DeadlockDetected exception

2022-04-11 Thread GitBox
tanelk commented on issue #19957: URL: https://github.com/apache/airflow/issues/19957#issuecomment-1094996707 In your last log you have some long lines, that seem to be truncated. For example: `Process 309021: SELECT task_instance.try_number AS task_instance_try_number, task_instance

[GitHub] [airflow] Bowrna commented on pull request #22876: cache and typo fix

2022-04-11 Thread GitBox
Bowrna commented on PR #22876: URL: https://github.com/apache/airflow/pull/22876#issuecomment-109530 @potiuk I have rebased the code now and still issue persists. Am i missing to understand any part? -- This is an automated message from the Apache Git Service. To respond to the messag

[GitHub] [airflow] ephraimbuddy commented on issue #22869: clearTaskInstances API sets state for all dagruns when given task_ids that don't exist / existed in the DAG

2022-04-11 Thread GitBox
ephraimbuddy commented on issue #22869: URL: https://github.com/apache/airflow/issues/22869#issuecomment-1095002866 @wolfier this issue was fixed in https://github.com/apache/airflow/pull/17961, can you verify that it doesn't exist from airflow 2.1.4 -- This is an automated message from

[GitHub] [airflow] dan-vaughan commented on pull request #22182: Add securityContext config for Redis to helm chart

2022-04-11 Thread GitBox
dan-vaughan commented on PR #22182: URL: https://github.com/apache/airflow/pull/22182#issuecomment-1095008708 > @dan-vaughan, congrats on your first commit! 🎉🚀 Thank you for your help @jedcunningham 🙏 -- This is an automated message from the Apache Git Service. To respond to the mes

[GitHub] [airflow] stablum commented on issue #19957: Airflow crashes with a psycopg2.errors.DeadlockDetected exception

2022-04-11 Thread GitBox
stablum commented on issue #19957: URL: https://github.com/apache/airflow/issues/19957#issuecomment-1095014663 Here is the log of an occurence of the crash even after migrating to 2.2.5: ``` dag_bag Running on host ml1 [2022-04-11 12:45:18,836] {scheduler_job.py:742} ERROR - Exc

[GitHub] [airflow] josh-fell commented on issue #22836: Status of testing Providers that were prepared on April 07, 2022

2022-04-11 Thread GitBox
josh-fell commented on issue #22836: URL: https://github.com/apache/airflow/issues/22836#issuecomment-1095027085 #22519 is good! Just a few docstring changes but confirmed in the package files. -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [airflow] tanelk commented on issue #19957: Airflow crashes with a psycopg2.errors.DeadlockDetected exception

2022-04-11 Thread GitBox
tanelk commented on issue #19957: URL: https://github.com/apache/airflow/issues/19957#issuecomment-1095027253 The python logs allways seem to show the update query in `dagrun.schedule_tis`. It would be most helpful to see the second half of this deadlock. In PostgreSQL log there was a hint

[GitHub] [airflow] subkanthi commented on a diff in pull request #21077: Kubernetes decorator

2022-04-11 Thread GitBox
subkanthi commented on code in PR #21077: URL: https://github.com/apache/airflow/pull/21077#discussion_r847322497 ## airflow/decorators/__init__.pyi: ## @@ -245,5 +245,121 @@ class TaskDecoratorCollection: :param cap_add: Include container capabilities """

[GitHub] [airflow] potiuk commented on issue #22895: Airflow health API is returning always 200

2022-04-11 Thread GitBox
potiuk commented on issue #22895: URL: https://github.com/apache/airflow/issues/22895#issuecomment-1095053074 As mentioned elsevhere - I believe each of our components should have a health endpoint (HTTP) - even if currently only webserver implements API. I know there are differing op

[GitHub] [airflow] potiuk commented on pull request #22876: cache and typo fix

2022-04-11 Thread GitBox
potiuk commented on PR #22876: URL: https://github.com/apache/airflow/pull/22876#issuecomment-1095071539 Ah I see - "Basic checks only"! @uranusjr was right - this CI step should have breeze installation same as the "static checks" - you can copy it from "Static Checks" in "ci.yml"

[GitHub] [airflow] potiuk commented on pull request #22876: cache and typo fix

2022-04-11 Thread GitBox
potiuk commented on PR #22876: URL: https://github.com/apache/airflow/pull/22876#issuecomment-1095072506 I will also add a nicer failure in this check though. It should inform the user that in order to run this check, they need to have breeze installed in the error. -- This is an automat

[GitHub] [airflow] potiuk opened a new issue, #22908: Add nicer error for BREEZE.rst update static check

2022-04-11 Thread GitBox
potiuk opened a new issue, #22908: URL: https://github.com/apache/airflow/issues/22908 If you modify BREEZE.rst or any of the "dev" files there is a static check run that updates screenshots in the docs. If you have no Breeze installed, It should nicely inform the user that they have

[GitHub] [airflow] potiuk commented on pull request #22876: cache and typo fix

2022-04-11 Thread GitBox
potiuk commented on PR #22876: URL: https://github.com/apache/airflow/pull/22876#issuecomment-1095077690 Added issue for "nicer" message https://github.com/apache/airflow/issues/22908 but for now you can turn this one into PR fixing the problem by adding Breeze installation in the CI job.,

[GitHub] [airflow] stablum commented on issue #19957: Airflow crashes with a psycopg2.errors.DeadlockDetected exception

2022-04-11 Thread GitBox
stablum commented on issue #19957: URL: https://github.com/apache/airflow/issues/19957#issuecomment-1095078018 one thing that I noticed is that the crashing query is particularly long, as I have several thousands of tasks in this DAG. And since the query is 22MB, the only way that I have to

[GitHub] [airflow] stablum commented on issue #19957: Airflow crashes with a psycopg2.errors.DeadlockDetected exception

2022-04-11 Thread GitBox
stablum commented on issue #19957: URL: https://github.com/apache/airflow/issues/19957#issuecomment-1095084954 I wonder if the SQL IN statement is the one creating problem: would it be possible to avoid it by using a subquery, end/or a JOIN, maybe? -- This is an automated message from the

[GitHub] [airflow] github-actions[bot] commented on pull request #22901: Catch error in Breeze when docker is not running

2022-04-11 Thread GitBox
github-actions[bot] commented on PR #22901: URL: https://github.com/apache/airflow/pull/22901#issuecomment-1095088933 The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full

[GitHub] [airflow] bhirsz commented on pull request #22811: AIP-47 Migrate ElasticSearch into new system tests

2022-04-11 Thread GitBox
bhirsz commented on PR #22811: URL: https://github.com/apache/airflow/pull/22811#issuecomment-1095089936 @Bowrna @joppevos We used State.None to clear the state of the DAG but it's restricted only to tasks: https://github.com/apache/airflow/blob/a3dd8473e4c5bbea214ebc8d5545b75281166428/a

[GitHub] [airflow] stablum commented on issue #19957: Airflow crashes with a psycopg2.errors.DeadlockDetected exception

2022-04-11 Thread GitBox
stablum commented on issue #19957: URL: https://github.com/apache/airflow/issues/19957#issuecomment-1095093672 Hmm, actually the last recorded query starts with: ``` UPDATE serialized_dag SET data='{"__version": 1, "dag": {"_dag_id": "down load_and_preprocess_sets", "schedule_inter

[GitHub] [airflow] stablum commented on issue #19957: Airflow crashes with a psycopg2.errors.DeadlockDetected exception

2022-04-11 Thread GitBox
stablum commented on issue #19957: URL: https://github.com/apache/airflow/issues/19957#issuecomment-1095096287 Hmm, it seems that the serialization is something that is done at a certain interval. Might it be that a serialization operation can get in conflict with the subsequent one if the

[GitHub] [airflow] stablum commented on issue #19957: Airflow crashes with a psycopg2.errors.DeadlockDetected exception

2022-04-11 Thread GitBox
stablum commented on issue #19957: URL: https://github.com/apache/airflow/issues/19957#issuecomment-1095108047 I will try to increase `min_serialized_dag_update_interval` and `min_serialized_dag_fetch_interval` by a factor of 100 -- This is an automated message from the Apache Git Service

[GitHub] [airflow] MatrixManAtYrService commented on issue #21867: Retry for TaskGroup

2022-04-11 Thread GitBox
MatrixManAtYrService commented on issue #21867: URL: https://github.com/apache/airflow/issues/21867#issuecomment-1095113648 I expect that some of the work necessary to make this possible will be done as part of [AIP-42](https://github.com/apache/airflow/projects/12?card_filter_query=task+gr

[GitHub] [airflow] stablum commented on issue #19957: Airflow crashes with a psycopg2.errors.DeadlockDetected exception

2022-04-11 Thread GitBox
stablum commented on issue #19957: URL: https://github.com/apache/airflow/issues/19957#issuecomment-1095114468 > The python logs allways seem to show the update query in `dagrun.schedule_tis`. It would be most helpful to see the second half of this deadlock. In PostgreSQL log there was a hi

[GitHub] [airflow] ashb merged pull request #22867: Add concept doc for Dynamic Task Mapping

2022-04-11 Thread GitBox
ashb merged PR #22867: URL: https://github.com/apache/airflow/pull/22867 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apa

[airflow] branch main updated: Add concept doc for Dynamic Task Mapping (#22867)

2022-04-11 Thread ash
This is an automated email from the ASF dual-hosted git repository. ash pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 73a87f8885 Add concept doc for Dynamic Task Mapping (#

  1   2   3   >