[GitHub] [airflow] eladkal commented on a diff in pull request #23861: Add default task retry delay config

2022-05-23 Thread GitBox
eladkal commented on code in PR #23861: URL: https://github.com/apache/airflow/pull/23861#discussion_r879085897 ## airflow/config_templates/config.yml: ## @@ -249,6 +249,14 @@ type: string example: ~ default: "0" +- name: default_task_retry_delay +

[GitHub] [airflow] eskarimov commented on pull request #19736: Add Databricks Deferrable Operators

2022-05-23 Thread GitBox
eskarimov commented on PR #19736: URL: https://github.com/apache/airflow/pull/19736#issuecomment-1134269034 Thank you everyone for your reviews, ideas and activity! :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

[GitHub] [airflow] tirkarthi commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'

2022-05-23 Thread GitBox
tirkarthi commented on issue #23727: URL: https://github.com/apache/airflow/issues/23727#issuecomment-1134295021 A related Stackoverflow question : https://stackoverflow.com/questions/67929230/airflow-2-0-1-pod-template-override-not-working-as-expected-for-kubernetesexecu -- This is an au

[GitHub] [airflow] bbovenzi merged pull request #23678: Highlight task states by hovering on legend row

2022-05-23 Thread GitBox
bbovenzi merged PR #23678: URL: https://github.com/apache/airflow/pull/23678 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

[airflow] branch main updated: Highlight task states by hovering on legend row (#23678)

2022-05-23 Thread bbovenzi
This is an automated email from the ASF dual-hosted git repository. bbovenzi pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 637a8b8af1 Highlight task states by hovering on l

[GitHub] [airflow] potiuk commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'

2022-05-23 Thread GitBox
potiuk commented on issue #23727: URL: https://github.com/apache/airflow/issues/23727#issuecomment-1134315657 Yeah - it does look as the deserialisation issue as I suspected. Just a thought - we do have the new `airflow dags reserialize` command in Airflow. I wonder @patryk126p

[GitHub] [airflow] potiuk commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'

2022-05-23 Thread GitBox
potiuk commented on issue #23727: URL: https://github.com/apache/airflow/issues/23727#issuecomment-1134318907 This command is exactly foreseen for this kind of cases where the dags were serialized using older airflow versions. I suspect what could have happened is that the old dags we

[GitHub] [airflow] potiuk commented on issue #23803: More flexible dynamic task mapping

2022-05-23 Thread GitBox
potiuk commented on issue #23803: URL: https://github.com/apache/airflow/issues/23803#issuecomment-1134330504 Also - another thing I usually recommend in this case - maybe this is not really a big problem that you cannot use it in classsic Airflow Operators. I think - personally - we should

[GitHub] [airflow] kouk commented on issue #16828: Cannot Set Index Pattern on Elasticsearch as a Log Handler

2022-05-23 Thread GitBox
kouk commented on issue #16828: URL: https://github.com/apache/airflow/issues/16828#issuecomment-113435 I have started working on this here: https://github.com/apache/airflow/compare/main...kouk:support-es-index-patterns?expand=1 it's still a WIP but any feedback would be helpful. -

[GitHub] [airflow] zqd759592942 opened a new issue, #23863: Does airflow have task configuration based on worker machine cpu or memory load balancing

2022-05-23 Thread GitBox
zqd759592942 opened a new issue, #23863: URL: https://github.com/apache/airflow/issues/23863 ### What do you see as an issue? Does airflow have task configuration based on worker machine cpu or memory load balancing ### Solving the problem _No response_ ### Anythi

[GitHub] [airflow] ashb commented on issue #23803: More flexible dynamic task mapping

2022-05-23 Thread GitBox
ashb commented on issue #23803: URL: https://github.com/apache/airflow/issues/23803#issuecomment-1134392649 One downside to not using (classic) operators but just using PythonOperator (decoratored or not) i sht we loose much of the ability to see what the task is doing from a lineage PoV. (

[GitHub] [airflow] msumit commented on a diff in pull request #23861: Add default task retry delay config

2022-05-23 Thread GitBox
msumit commented on code in PR #23861: URL: https://github.com/apache/airflow/pull/23861#discussion_r879209598 ## airflow/config_templates/config.yml: ## @@ -249,6 +249,14 @@ type: string example: ~ default: "0" +- name: default_task_retry_delay + d

[GitHub] [airflow] mhenc opened a new pull request, #23864: Use '--subdir' argument value for standalone dag processor.

2022-05-23 Thread GitBox
mhenc opened a new pull request, #23864: URL: https://github.com/apache/airflow/pull/23864 The value defaults to `settings.DAGS_FOLDER` if not set: https://github.com/apache/airflow/blob/637a8b8af132e6231756160a3f6ce7ed789abaf6/airflow/cli/cli_parser.py#L193 -- This is an automated me

[GitHub] [airflow] mhenc commented on pull request #23864: Use '--subdir' argument value for standalone dag processor.

2022-05-23 Thread GitBox
mhenc commented on PR #23864: URL: https://github.com/apache/airflow/pull/23864#issuecomment-1134408095 cc: @potiuk -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsu

[GitHub] [airflow] eladkal commented on a diff in pull request #23861: Add default task retry delay config

2022-05-23 Thread GitBox
eladkal commented on code in PR #23861: URL: https://github.com/apache/airflow/pull/23861#discussion_r879213035 ## airflow/config_templates/config.yml: ## @@ -249,6 +249,14 @@ type: string example: ~ default: "0" +- name: default_task_retry_delay +

[GitHub] [airflow] github-actions[bot] commented on pull request #23861: Add default task retry delay config

2022-05-23 Thread GitBox
github-actions[bot] commented on PR #23861: URL: https://github.com/apache/airflow/pull/23861#issuecomment-1134414084 The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk

[GitHub] [airflow] wojsamjan opened a new pull request, #23865: Cloud Storage assets & StorageLink update

2022-05-23 Thread GitBox
wojsamjan opened a new pull request, #23865: URL: https://github.com/apache/airflow/pull/23865 Add links to assets for Cloud Storage Co-authored-by: Wojciech Januszek [janus...@google.com](mailto:janus...@google.com) Co-authored-by: Lukasz Wyszomirski [wyszomir...@google.c

[GitHub] [airflow] cansjt commented on issue #23852: Status of testing of Apache Airflow 2.3.1rc1

2022-05-23 Thread GitBox
cansjt commented on issue #23852: URL: https://github.com/apache/airflow/issues/23852#issuecomment-1134472514 I am sorry, but I am alone this week (my colleagues are off), so it's doubtful I will find the time to check. -- This is an automated message from the Apache Git Service. To respo

[GitHub] [airflow] gmsantos commented on issue #23852: Status of testing of Apache Airflow 2.3.1rc1

2022-05-23 Thread GitBox
gmsantos commented on issue #23852: URL: https://github.com/apache/airflow/issues/23852#issuecomment-1134485019 For #23590 (related to https://github.com/apache/airflow/discussions/23489 and #23580), tasks with `.` are shown again in grid view. I checked that #23521 also fixes the sch

[GitHub] [airflow] parthpalkhiwala-vonage commented on issue #20875: Serialization error after successful DAG run

2022-05-23 Thread GitBox
parthpalkhiwala-vonage commented on issue #20875: URL: https://github.com/apache/airflow/issues/20875#issuecomment-1134486949 Hello All, We are still facing a similar issuing while upgrading our airflow version from 2.1.3 to 2.2.5 Below is the error we are getting - this usually occurs

[GitHub] [airflow] gmsantos commented on issue #23852: Status of testing of Apache Airflow 2.3.1rc1

2022-05-23 Thread GitBox
gmsantos commented on issue #23852: URL: https://github.com/apache/airflow/issues/23852#issuecomment-1134511408 #23625: working fine as well -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

[GitHub] [airflow] uranusjr commented on a diff in pull request #23861: Add default task retry delay config

2022-05-23 Thread GitBox
uranusjr commented on code in PR #23861: URL: https://github.com/apache/airflow/pull/23861#discussion_r879348378 ## airflow/config_templates/config.yml: ## @@ -249,6 +249,14 @@ type: string example: ~ default: "0" +- name: default_task_retry_delay +

[GitHub] [airflow] github-actions[bot] commented on pull request #23864: Use '--subdir' argument value for standalone dag processor.

2022-05-23 Thread GitBox
github-actions[bot] commented on PR #23864: URL: https://github.com/apache/airflow/pull/23864#issuecomment-1134559304 The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the

[GitHub] [airflow] potiuk commented on issue #23863: Does airflow have task configuration based on worker machine cpu or memory load balancing

2022-05-23 Thread GitBox
potiuk commented on issue #23863: URL: https://github.com/apache/airflow/issues/23863#issuecomment-1134561823 No. And please next time follow what you are edlladvised to when you open an issue. You should open Discussion when you have question. NOT issue -- This is an automated message f

[GitHub] [airflow] potiuk closed issue #23863: Does airflow have task configuration based on worker machine cpu or memory load balancing

2022-05-23 Thread GitBox
potiuk closed issue #23863: Does airflow have task configuration based on worker machine cpu or memory load balancing URL: https://github.com/apache/airflow/issues/23863 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [airflow] msumit commented on a diff in pull request #23861: Add default task retry delay config

2022-05-23 Thread GitBox
msumit commented on code in PR #23861: URL: https://github.com/apache/airflow/pull/23861#discussion_r879354885 ## airflow/config_templates/config.yml: ## @@ -249,6 +249,14 @@ type: string example: ~ default: "0" +- name: default_task_retry_delay + d

[GitHub] [airflow] skyboi1233 commented on a diff in pull request #23861: Add default task retry delay config

2022-05-23 Thread GitBox
skyboi1233 commented on code in PR #23861: URL: https://github.com/apache/airflow/pull/23861#discussion_r879356149 ## airflow/config_templates/config.yml: ## @@ -249,6 +249,14 @@ type: string example: ~ default: "0" +- name: default_task_retry_delay +

[GitHub] [airflow] uranusjr commented on a diff in pull request #23861: Add default task retry delay config

2022-05-23 Thread GitBox
uranusjr commented on code in PR #23861: URL: https://github.com/apache/airflow/pull/23861#discussion_r879356526 ## airflow/config_templates/config.yml: ## @@ -249,6 +249,14 @@ type: string example: ~ default: "0" +- name: default_task_retry_delay +

[GitHub] [airflow] uranusjr commented on pull request #23812: Fix UnboundLocalError when sql is empty list in ExasolHook

2022-05-23 Thread GitBox
uranusjr commented on PR #23812: URL: https://github.com/apache/airflow/pull/23812#issuecomment-1134573130 I wonder if we should silently return None when `sql` is empty, instead of failing the task. -- This is an automated message from the Apache Git Service. To respond to the message, p

[GitHub] [airflow] msumit commented on pull request #23861: Add default task retry delay config

2022-05-23 Thread GitBox
msumit commented on PR #23861: URL: https://github.com/apache/airflow/pull/23861#issuecomment-1134574755 > (putting a temporary hold on this) > > Isn't this already achievable using the "Cluster Policy" hooks? I'm wary of "config sprawl" if we add lots of options. Let me try to

[GitHub] [airflow] eladkal commented on pull request #23861: Add default task retry delay config

2022-05-23 Thread GitBox
eladkal commented on PR #23861: URL: https://github.com/apache/airflow/pull/23861#issuecomment-1134580999 > Isn't this already achievable using the "Cluster Policy" hooks? I'm wary of "config sprawl" if we add lots of options. Cluster policy has weakness that it's not exposed to the e

[GitHub] [airflow] ashb commented on pull request #23861: Add default task retry delay config

2022-05-23 Thread GitBox
ashb commented on PR #23861: URL: https://github.com/apache/airflow/pull/23861#issuecomment-1134582393 How is cluster policy any different to a config setting in this regard? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [airflow] eladkal commented on pull request #23861: Add default task retry delay config

2022-05-23 Thread GitBox
eladkal commented on PR #23861: URL: https://github.com/apache/airflow/pull/23861#issuecomment-1134600986 > How is cluster policy any different to a config setting in this regard? The config is picked up by the operator thus in the task details you will see the default value being set

[GitHub] [airflow] potiuk opened a new pull request, #23866: Speed up Breeze experience on Mac OS

2022-05-23 Thread GitBox
potiuk opened a new pull request, #23866: URL: https://github.com/apache/airflow/pull/23866 This change should significantly speed up Breeze experience (and especially iterating over a change in Breeze for MacOS users - independently if you are using x86 or arm architecture. The

[GitHub] [airflow] potiuk commented on pull request #23866: Speed up Breeze experience on Mac OS

2022-05-23 Thread GitBox
potiuk commented on PR #23866: URL: https://github.com/apache/airflow/pull/23866#issuecomment-1134605144 @dstandish @bbovenzi @kaxil @bowrna - I believe you use Mac OS. I hope you will enjoy the 5x speed improvements that this change brings for MacOS Breeze user running MyPy checks. --

[GitHub] [airflow] msumit commented on pull request #23861: Add default task retry delay config

2022-05-23 Thread GitBox
msumit commented on PR #23861: URL: https://github.com/apache/airflow/pull/23861#issuecomment-1134606398 A couple of observations using the CP hook - The `retry_dealy` value is already set when it comes to the hook, so we would need to make a lot of code changes in `Baseoperator` to make it

[GitHub] [airflow] potiuk commented on a diff in pull request #23866: Speed up Breeze experience on Mac OS

2022-05-23 Thread GitBox
potiuk commented on code in PR #23866: URL: https://github.com/apache/airflow/pull/23866#discussion_r879393276 ## scripts/ci/docker-compose/backend-mssql-docker-volume.yml: ## @@ -20,3 +20,5 @@ services: mssql: volumes: - mssql-db-volume:/var/opt/mssql +volumes:

[GitHub] [airflow] jyotidhiman0610 commented on issue #21955: Possibility to filter tasks in `None` state

2022-05-23 Thread GitBox
jyotidhiman0610 commented on issue #21955: URL: https://github.com/apache/airflow/issues/21955#issuecomment-1134626941 Hi, @mgorsk1 are you working on this or can I take this up? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [airflow] potiuk commented on pull request #23812: Fix UnboundLocalError when sql is empty list in ExasolHook

2022-05-23 Thread GitBox
potiuk commented on PR #23812: URL: https://github.com/apache/airflow/pull/23812#issuecomment-1134628063 I think failing is more reasonable. I think explicit `SELECT 1 FROM DUAL` is better one to indicate "empty" query and there are at least few scenarios where empty response might be resul

[GitHub] [airflow] jyotidhiman0610 commented on issue #20829: Update google-cloud-container package to v2.10.1

2022-05-23 Thread GitBox
jyotidhiman0610 commented on issue #20829: URL: https://github.com/apache/airflow/issues/20829#issuecomment-1134628224 Hi, @askmeegs are you working on this or can I take this up? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [airflow] jyotidhiman0610 commented on issue #22632: Missing doc and sample dag for ExasolToS3Operator

2022-05-23 Thread GitBox
jyotidhiman0610 commented on issue #22632: URL: https://github.com/apache/airflow/issues/22632#issuecomment-1134632016 Hi, can I take this up? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] kazanzhy commented on pull request #23812: Fix UnboundLocalError when sql is empty list in ExasolHook

2022-05-23 Thread GitBox
kazanzhy commented on PR #23812: URL: https://github.com/apache/airflow/pull/23812#issuecomment-1134639957 @uranusjr there was a short discussion about the behavior https://github.com/apache/airflow/pull/23767 It seems that raising an error when `sql` parameter is an empty list will be

[GitHub] [airflow] github-actions[bot] commented on pull request #23866: Speed up Breeze experience on Mac OS

2022-05-23 Thread GitBox
github-actions[bot] commented on PR #23866: URL: https://github.com/apache/airflow/pull/23866#issuecomment-1134644917 The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk

[GitHub] [airflow] potiuk commented on pull request #23866: Speed up Breeze experience on Mac OS

2022-05-23 Thread GitBox
potiuk commented on PR #23866: URL: https://github.com/apache/airflow/pull/23866#issuecomment-1134647191 > nice, thank you @potiuk Actually thank Airflow Summit. I thought about it and attempted to fix it while flying to London for the Airflow Summit opening today :) - that was ideal

[GitHub] [airflow] potiuk commented on pull request #23866: Speed up Breeze experience on Mac OS

2022-05-23 Thread GitBox
potiuk commented on PR #23866: URL: https://github.com/apache/airflow/pull/23866#issuecomment-1134649694 BTW. Not that fast - some static checks failing in CI :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [airflow] potiuk commented on pull request #23866: Speed up Breeze experience on Mac OS

2022-05-23 Thread GitBox
potiuk commented on PR #23866: URL: https://github.com/apache/airflow/pull/23866#issuecomment-1134652689 Ah yeah. committed changing Python 3.7 -> 3.8 as I had no 3.7 image handy in the plane after cleanup yesterday 🤦 -- This is an automated message from the Apache Git Service. To respon

[GitHub] [airflow] potiuk commented on issue #22632: Missing doc and sample dag for ExasolToS3Operator

2022-05-23 Thread GitBox
potiuk commented on issue #22632: URL: https://github.com/apache/airflow/issues/22632#issuecomment-1134656471 go ahead -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

[GitHub] [airflow] potiuk commented on issue #20829: Update google-cloud-container package to v2.10.1

2022-05-23 Thread GitBox
potiuk commented on issue #20829: URL: https://github.com/apache/airflow/issues/20829#issuecomment-1134657226 I will add you as assignee already @jyotidhiman0610 :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

[GitHub] [airflow] potiuk commented on issue #21955: Possibility to filter tasks in `None` state

2022-05-23 Thread GitBox
potiuk commented on issue #21955: URL: https://github.com/apache/airflow/issues/21955#issuecomment-1134657709 Feel free. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [airflow] potiuk merged pull request #23597: Clean up f-strings in logging calls

2022-05-23 Thread GitBox
potiuk merged PR #23597: URL: https://github.com/apache/airflow/pull/23597 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[airflow] branch main updated: Clean up f-strings in logging calls (#23597)

2022-05-23 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new ec6761a5c0 Clean up f-strings in logging calls (#23

[GitHub] [airflow] uranusjr opened a new issue, #23867: Don’t prompt for 'breeze build-image'

2022-05-23 Thread GitBox
uranusjr opened a new issue, #23867: URL: https://github.com/apache/airflow/issues/23867 ### Body Currently, running the (new) `breeze build-image` brings up two prompts if any of the meta files are outdated: ``` $ breeze build-image Good version of Docker: 20.10.14. G

[GitHub] [airflow] oriolcmp commented on issue #12985: PythonVirtualenvOperator with provide_context=True does not have 'ti' keyword

2022-05-23 Thread GitBox
oriolcmp commented on issue #12985: URL: https://github.com/apache/airflow/issues/12985#issuecomment-1134718072 Hello, After two years with this ticket open and after adding a fix like #19616 which seems was the same issue but in this case adding the "ti" as serializable would have f

[GitHub] [airflow] uranusjr commented on issue #23792: Dynamic task mapping with KubernetesPodOperator maps too many tasks erroneously

2022-05-23 Thread GitBox
uranusjr commented on issue #23792: URL: https://github.com/apache/airflow/issues/23792#issuecomment-1134719462 I seem to recall there’s a bug on expanding a wrong number of tasks, but can’t find it now. This _may_ have been fixed in main. -- This is an automated message from the Apache G

[GitHub] [airflow] uranusjr commented on issue #12985: PythonVirtualenvOperator with provide_context=True does not have 'ti' keyword

2022-05-23 Thread GitBox
uranusjr commented on issue #12985: URL: https://github.com/apache/airflow/issues/12985#issuecomment-1134723587 Feel free to propose a PR to add `ti` to the list. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [airflow] uranusjr opened a new issue, #23868: Don’t show traceback on 'breeze tests' subprocess returning non-zero

2022-05-23 Thread GitBox
uranusjr opened a new issue, #23868: URL: https://github.com/apache/airflow/issues/23868 ### Body Currently, if any tests fail when `breeze tests` is run, Breeze 2 would emit a traceback pointing to the `docker-compose` subprocess call. This is due to Docker propagating the exit call

[GitHub] [airflow] josh-fell commented on pull request #22938: Enable dbt Cloud provider to interact with single tenant instances

2022-05-23 Thread GitBox
josh-fell commented on PR #22938: URL: https://github.com/apache/airflow/pull/22938#issuecomment-1134740013 > @josh-fell Yep, totally follow. I removed the arg from the Operator. Do you have an opinion whether the tenant should be stored in the connection as the schema or as an extra?

[GitHub] [airflow] jedcunningham merged pull request #23859: update K8S-KIND to 0.14.0

2022-05-23 Thread GitBox
jedcunningham merged PR #23859: URL: https://github.com/apache/airflow/pull/23859 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@ai

[airflow] branch main updated: update K8S-KIND to 0.14.0 (#23859)

2022-05-23 Thread jedcunningham
This is an automated email from the ASF dual-hosted git repository. jedcunningham pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 35620edd4b update K8S-KIND to 0.14.0 (#23859

[GitHub] [airflow] uranusjr commented on pull request #23237: Replaced all days_ago functions with datetime functions

2022-05-23 Thread GitBox
uranusjr commented on PR #23237: URL: https://github.com/apache/airflow/pull/23237#issuecomment-1134750350 Should work now! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [airflow] github-actions[bot] commented on pull request #23237: Replaced all days_ago functions with datetime functions

2022-05-23 Thread GitBox
github-actions[bot] commented on PR #23237: URL: https://github.com/apache/airflow/pull/23237#issuecomment-1134758881 The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk

[GitHub] [airflow] domagojrazum opened a new pull request, #23870: DAG regex flag in backfill command

2022-05-23 Thread GitBox
domagojrazum opened a new pull request, #23870: URL: https://github.com/apache/airflow/pull/23870 When this parameter is set, the dag_id string will be used as a regular expression to match the available DAGs. This enables multiple DAG backfills to be consecutively executed under the sam

[GitHub] [airflow] gmcrocetti commented on pull request #23857: Create `get_partition` and `create_partition` methods in `GlueCatalogHook`

2022-05-23 Thread GitBox
gmcrocetti commented on PR #23857: URL: https://github.com/apache/airflow/pull/23857#issuecomment-1134801928 Hey @vincbeck :). Tagging you as requested o/. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

[GitHub] [airflow] Taragolis opened a new issue, #23871: `dag-processor` failed to start in docker

2022-05-23 Thread GitBox
Taragolis opened a new issue, #23871: URL: https://github.com/apache/airflow/issues/23871 ### Apache Airflow version 2.3.0 (latest released) ### What happened Standalone DagProcessor which run in Apache Airflow Production Docker Image failed with error ``` airf

[GitHub] [airflow] potiuk commented on issue #23867: Don’t prompt for 'breeze build-image'

2022-05-23 Thread GitBox
potiuk commented on issue #23867: URL: https://github.com/apache/airflow/issues/23867#issuecomment-1134824848 Oh absolutely :). It's been like that in the old breeze :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

[GitHub] [airflow] uranusjr merged pull request #23237: Replaced all days_ago functions with datetime functions

2022-05-23 Thread GitBox
uranusjr merged PR #23237: URL: https://github.com/apache/airflow/pull/23237 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

[GitHub] [airflow] uranusjr closed issue #23233: Remove usage of `days_ago()` from test suite

2022-05-23 Thread GitBox
uranusjr closed issue #23233: Remove usage of `days_ago()` from test suite URL: https://github.com/apache/airflow/issues/23233 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[airflow] branch main updated: Replaced all days_ago functions with datetime functions (#23237)

2022-05-23 Thread uranusjr
This is an automated email from the ASF dual-hosted git repository. uranusjr pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new f352ee63a5 Replaced all days_ago functions with d

[GitHub] [airflow] josh-fell commented on a diff in pull request #23704: Add Tabular provider

2022-05-23 Thread GitBox
josh-fell commented on code in PR #23704: URL: https://github.com/apache/airflow/pull/23704#discussion_r879549311 ## airflow/providers/tabular/example_dags/example_tabular.py: ## @@ -0,0 +1,46 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributo

[GitHub] [airflow] Taragolis opened a new pull request, #23872: DagFileProcessorManager: Start a new process group only if current process not a session leader

2022-05-23 Thread GitBox
Taragolis opened a new pull request, #23872: URL: https://github.com/apache/airflow/pull/23872 Not sure if additional tests required because it required mock os methods just for compare PID = SID however let me know and I will add it closes: #23871 --- **^ Add me

[GitHub] [airflow] josh-fell commented on a diff in pull request #23873: Introduce `flake8-implicit-str-concat` plugin to static checks

2022-05-23 Thread GitBox
josh-fell commented on code in PR #23873: URL: https://github.com/apache/airflow/pull/23873#discussion_r879692977 ## airflow/providers/databricks/operators/databricks_repos.py: ## @@ -90,7 +90,7 @@ def __init__( self.git_provider = self.__detect_repo_provider__(git_

[GitHub] [airflow] josh-fell opened a new pull request, #23873: Introduce `flake8-implicit-str-concat` plugin to static checks

2022-05-23 Thread GitBox
josh-fell opened a new pull request, #23873: URL: https://github.com/apache/airflow/pull/23873 Related: #23591 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines

[GitHub] [airflow] alexott commented on a diff in pull request #23873: Introduce `flake8-implicit-str-concat` plugin to static checks

2022-05-23 Thread GitBox
alexott commented on code in PR #23873: URL: https://github.com/apache/airflow/pull/23873#discussion_r879693454 ## airflow/providers/databricks/operators/databricks_repos.py: ## @@ -90,7 +90,7 @@ def __init__( self.git_provider = self.__detect_repo_provider__(git_ur

[GitHub] [airflow] josh-fell commented on a diff in pull request #23873: Introduce `flake8-implicit-str-concat` plugin to static checks

2022-05-23 Thread GitBox
josh-fell commented on code in PR #23873: URL: https://github.com/apache/airflow/pull/23873#discussion_r879694846 ## tests/providers/ssh/hooks/test_ssh.py: ## @@ -742,7 +742,7 @@ def test_openssh_private_key(self): def test_oneline_key(self): with pytest.raises(E

[GitHub] [airflow] snjypl commented on a diff in pull request #23720: Fix backfill queued task getting reset to scheduled state.

2022-05-23 Thread GitBox
snjypl commented on code in PR #23720: URL: https://github.com/apache/airflow/pull/23720#discussion_r879714440 ## airflow/executors/kubernetes_executor.py: ## @@ -530,13 +532,15 @@ def start(self) -> None: self.kube_config.worker_pods_pending_timeout_check_interval,

[GitHub] [airflow] pingzh commented on a diff in pull request #21877: AIP-45 Remove dag parsing in airflow run local

2022-05-23 Thread GitBox
pingzh commented on code in PR #21877: URL: https://github.com/apache/airflow/pull/21877#discussion_r879715014 ## airflow/jobs/scheduler_job.py: ## @@ -764,6 +765,26 @@ def _execute(self) -> None: self.log.exception("Exception when executing DagFileProcesso

[GitHub] [airflow] josh-fell commented on a diff in pull request #23848: The first system test for slack (implemented according to AIP-47)

2022-05-23 Thread GitBox
josh-fell commented on code in PR #23848: URL: https://github.com/apache/airflow/pull/23848#discussion_r879712639 ## tests/system/providers/slack/example_slack.py: ## @@ -0,0 +1,71 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

[GitHub] [airflow] Taragolis commented on pull request #23872: DagFileProcessorManager: Start a new process group only if current process not a session leader

2022-05-23 Thread GitBox
Taragolis commented on PR #23872: URL: https://github.com/apache/airflow/pull/23872#issuecomment-1134954306 Also I found call `os.setpgid(0, 0)` in two other places https://github.com/apache/airflow/blob/f352ee63a5d09546a7997ba8f2f8702a1ddb4af7/airflow/task/task_runner/standard_task_

[GitHub] [airflow] josh-fell commented on pull request #23704: Add Tabular provider

2022-05-23 Thread GitBox
josh-fell commented on PR #23704: URL: https://github.com/apache/airflow/pull/23704#issuecomment-1134963962 Just to toss it out there, should net-new example DAGs live as system tests based on AIP-47 now? Or since this is v0.0.1 of this provider that it can be migrated over as the provider

[GitHub] [airflow] denimalpaca opened a new issue, #23874: Add Templated SQL Check Operators for Columns and Tables

2022-05-23 Thread GitBox
denimalpaca opened a new issue, #23874: URL: https://github.com/apache/airflow/issues/23874 ### Description The new SQL Check Operators would provide templated quality checks so users could quickly and easily add common data quality checks on columns and tables. The resulting output

[GitHub] [airflow] Taragolis commented on issue #23475: AWS rate limiting causes tasks to fail

2022-05-23 Thread GitBox
Taragolis commented on issue #23475: URL: https://github.com/apache/airflow/issues/23475#issuecomment-1134984937 As workaround - set Environment Variables for airflow service to use more "pessimistic" retry strategy, rather than [default one](https://boto3.amazonaws.com/v1/documentation/api

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #23875: Ignore deprecation days ago tests

2022-05-23 Thread GitBox
boring-cyborg[bot] commented on PR #23875: URL: https://github.com/apache/airflow/pull/23875#issuecomment-1134990290 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution

[GitHub] [airflow] alexkruc opened a new pull request, #23875: Ignore deprecation days ago tests

2022-05-23 Thread GitBox
alexkruc opened a new pull request, #23875: URL: https://github.com/apache/airflow/pull/23875 Decreases the deprecation warnings for this test file from 71 to 19 :) This function will be removed anyway in Airflow 3, so it's either to delete the test or to remove the deprecation warnings.

[GitHub] [airflow] github-actions[bot] commented on pull request #23875: Ignore deprecation warnings for days ago tests

2022-05-23 Thread GitBox
github-actions[bot] commented on PR #23875: URL: https://github.com/apache/airflow/pull/23875#issuecomment-1135105307 The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk

[GitHub] [airflow] schattian commented on pull request #23658: Fix k8s pod.execute randomly stuck indefinitely by logs consumption (#23497)

2022-05-23 Thread GitBox
schattian commented on PR #23658: URL: https://github.com/apache/airflow/pull/23658#issuecomment-1135138024 @potiuk could you please take a look at this one? would be nice to have, personally I experience this bug regularly (once a week) on gke. -- This is an automated message from the Ap

[GitHub] [airflow] jedcunningham opened a new pull request, #23876: Document LocalKubernetesExecutor support in chart

2022-05-23 Thread GitBox
jedcunningham opened a new pull request, #23876: URL: https://github.com/apache/airflow/pull/23876 I just noticed we missed adding LocalKubernetesExecutor into the helm chart docs. I've also reordered them to put the "core" executors first, and the "combo" executors last. -- This is an a

[GitHub] [airflow] github-actions[bot] commented on pull request #22793: Support DAGRun type reset through the backfill command

2022-05-23 Thread GitBox
github-actions[bot] commented on PR #22793: URL: https://github.com/apache/airflow/pull/22793#issuecomment-1135261788 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for you

[GitHub] [airflow] github-actions[bot] commented on pull request #22329: Add naming convention to the DB

2022-05-23 Thread GitBox
github-actions[bot] commented on PR #22329: URL: https://github.com/apache/airflow/pull/22329#issuecomment-1135261827 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for you

[GitHub] [airflow] github-actions[bot] commented on pull request #22676: Fix helm name override

2022-05-23 Thread GitBox
github-actions[bot] commented on PR #22676: URL: https://github.com/apache/airflow/pull/22676#issuecomment-1135261807 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for you

[GitHub] [airflow] github-actions[bot] commented on pull request #23876: Document LocalKubernetesExecutor support in chart

2022-05-23 Thread GitBox
github-actions[bot] commented on PR #23876: URL: https://github.com/apache/airflow/pull/23876#issuecomment-1135276893 The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the

[GitHub] [airflow] josh-fell opened a new pull request, #23877: Move string arg evals to `execute()` in `EksCreateClusterOperator`

2022-05-23 Thread GitBox
josh-fell opened a new pull request, #23877: URL: https://github.com/apache/airflow/pull/23877 Currently there are string-value evaluations of `compute`, `nodegroup_role_arn`, and `fargate_pod_execution_role_arn` args in the constructor of `EksCreateClusterOperator`. These args are all li

[GitHub] [airflow] github-actions[bot] commented on pull request #23877: Move string arg evals to `execute()` in `EksCreateClusterOperator`

2022-05-23 Thread GitBox
github-actions[bot] commented on PR #23877: URL: https://github.com/apache/airflow/pull/23877#issuecomment-1135323959 The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the

[GitHub] [airflow] josh-fell commented on a diff in pull request #23873: Introduce `flake8-implicit-str-concat` plugin to static checks

2022-05-23 Thread GitBox
josh-fell commented on code in PR #23873: URL: https://github.com/apache/airflow/pull/23873#discussion_r879694846 ## tests/providers/ssh/hooks/test_ssh.py: ## @@ -742,7 +742,7 @@ def test_openssh_private_key(self): def test_oneline_key(self): with pytest.raises(E

[GitHub] [airflow] josh-fell commented on issue #16506: Replication of Subdag behaviour with Task Groups

2022-05-23 Thread GitBox
josh-fell commented on issue #16506: URL: https://github.com/apache/airflow/issues/16506#issuecomment-1135329841 Does the new [`ignore_downstream_trigger_rules`](https://github.com/apache/airflow/blob/6b772f68db4f80b01ede836ddd47437d4aa1b43d/airflow/operators/python.py#L248) param of the `S

[GitHub] [airflow] epapineau commented on pull request #22938: Enable dbt Cloud provider to interact with single tenant instances

2022-05-23 Thread GitBox
epapineau commented on PR #22938: URL: https://github.com/apache/airflow/pull/22938#issuecomment-1135339271 @josh-fell okay changes pushed 🎉 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

[GitHub] [airflow] uranusjr commented on pull request #23662: Implement PlainCronDataIntervalTimetable

2022-05-23 Thread GitBox
uranusjr commented on PR #23662: URL: https://github.com/apache/airflow/pull/23662#issuecomment-1135359777 Instead of inheriting `CronDataIntervalTimetable` directly, I would prefer the common logic be extracted out to a superclass, which both timetable clases inherit. I feel this cl

[GitHub] [airflow] uranusjr commented on pull request #23662: Implement PlainCronDataIntervalTimetable

2022-05-23 Thread GitBox
uranusjr commented on PR #23662: URL: https://github.com/apache/airflow/pull/23662#issuecomment-1135361440 Or, if this is too big a scope and you really only want to “fix” the first run of cron scheduling. Honestly the current behaviour sounds like a bug to me, and arguably we might want to

[GitHub] [airflow] github-actions[bot] commented on pull request #23452: Fix closing connection dbapi.get_pandas_df

2022-05-23 Thread GitBox
github-actions[bot] commented on PR #23452: URL: https://github.com/apache/airflow/pull/23452#issuecomment-1135362751 The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk

[GitHub] [airflow] uranusjr closed issue #23419: Add a DAG Run clear endpoint to REST API

2022-05-23 Thread GitBox
uranusjr closed issue #23419: Add a DAG Run clear endpoint to REST API URL: https://github.com/apache/airflow/issues/23419 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

[airflow] branch main updated: Add clear DagRun endpoint. (#23451)

2022-05-23 Thread uranusjr
This is an automated email from the ASF dual-hosted git repository. uranusjr pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new b83cc9b5e2 Add clear DagRun endpoint. (#23451) b8

  1   2   >