[GitHub] [airflow] github-actions[bot] closed issue #14572: subdag cannot get {{ next_ds }} {{ prev_ds }} correctly from context in 2.0

2023-07-17 Thread via GitHub
github-actions[bot] closed issue #14572: subdag cannot get {{ next_ds }} {{ prev_ds }} correctly from context in 2.0 URL: https://github.com/apache/airflow/issues/14572 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [airflow] mis98zb commented on a diff in pull request #32122: control task group concurrency: use allow list for mapping index

2023-07-17 Thread via GitHub
mis98zb commented on code in PR #32122: URL: https://github.com/apache/airflow/pull/32122#discussion_r126497 ## airflow/jobs/scheduler_job_runner.py: ## @@ -277,9 +288,51 @@ def __get_concurrency_maps(self, states: Iterable[TaskInstanceState], session: S .where

[GitHub] [airflow] ikholodkov commented on issue #32585: Commit failed: Local: No offset stored while using AwaitMessageTriggerFunctionSensor

2023-07-17 Thread via GitHub
ikholodkov commented on issue #32585: URL: https://github.com/apache/airflow/issues/32585#issuecomment-1637529577 confluent-kafka was installed while installing apache-airflow-providers-apache-kafka==1.1.2 and it's 2.2.0 Kafka cluster version is 3.2.x -- This is an automated message fr

[GitHub] [airflow] uranusjr commented on a diff in pull request #32122: control task group concurrency: use allow list for mapping index

2023-07-17 Thread via GitHub
uranusjr commented on code in PR #32122: URL: https://github.com/apache/airflow/pull/32122#discussion_r1264984045 ## airflow/jobs/scheduler_job_runner.py: ## @@ -277,9 +288,51 @@ def __get_concurrency_maps(self, states: Iterable[TaskInstanceState], session: S .wher

[GitHub] [airflow] mis98zb commented on a diff in pull request #32122: control task group concurrency: use allow list for mapping index

2023-07-17 Thread via GitHub
mis98zb commented on code in PR #32122: URL: https://github.com/apache/airflow/pull/32122#discussion_r126497 ## airflow/jobs/scheduler_job_runner.py: ## @@ -277,9 +288,51 @@ def __get_concurrency_maps(self, states: Iterable[TaskInstanceState], session: S .where

[GitHub] [airflow] mis98zb commented on pull request #32122: control task group concurrency: use allow list for mapping index

2023-07-17 Thread via GitHub
mis98zb commented on PR #32122: URL: https://github.com/apache/airflow/pull/32122#issuecomment-1637548255 Sorry for my mistake. We should not use State.unfinished because it include None... -- This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [airflow] juhai commented on a diff in pull request #32278: Allow creating unique pod and hostname for KubernetesPodOperator from task job_id

2023-07-17 Thread via GitHub
juhai commented on code in PR #32278: URL: https://github.com/apache/airflow/pull/32278#discussion_r1265015291 ## airflow/providers/cncf/kubernetes/operators/pod.py: ## @@ -158,6 +158,8 @@ class KubernetesPodOperator(BaseOperator): suffix if random_name_suffix is True)

[GitHub] [airflow] VladaZakharova commented on pull request #31895: Refactor of links in Dataproc

2023-07-17 Thread via GitHub
VladaZakharova commented on PR #31895: URL: https://github.com/apache/airflow/pull/31895#issuecomment-1637687111 Hi @uranusjr ! Could you please recheck this PR? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [airflow] huymq1710 opened a new issue, #32642: Backfill API endpoint

2023-07-17 Thread via GitHub
huymq1710 opened a new issue, #32642: URL: https://github.com/apache/airflow/issues/32642 ### Description The proposed feature is to enhance the functionality of the Airflow Backfill API endpoint. This improvement would allow users to specify a date range for backfilling DAG runs, en

[GitHub] [airflow] bkossakowska opened a new pull request, #32643: Fix CloudTasksTaskCreateOperator system test

2023-07-17 Thread via GitHub
bkossakowska opened a new pull request, #32643: URL: https://github.com/apache/airflow/pull/32643 Fix CloudTasksTaskCreateOperator system test --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/bl

[GitHub] [airflow] csp33 opened a new pull request, #32644: Fix webserver configMap name not including Release Name

2023-07-17 Thread via GitHub
csp33 opened a new pull request, #32644: URL: https://github.com/apache/airflow/pull/32644 We are not using the `airflow_webserver_config_configmap_name` template when creating the volumeMount. Before the change: https://github.com/apache/airflow/assets/28935464/0a7827e5-a585-4a6e-

[GitHub] [airflow] eladkal commented on a diff in pull request #32278: Allow creating unique pod and hostname for KubernetesPodOperator from task job_id

2023-07-17 Thread via GitHub
eladkal commented on code in PR #32278: URL: https://github.com/apache/airflow/pull/32278#discussion_r1264874060 ## airflow/providers/cncf/kubernetes/operators/pod.py: ## @@ -158,6 +158,8 @@ class KubernetesPodOperator(BaseOperator): suffix if random_name_suffix is True

[GitHub] [airflow] juhai closed pull request #32278: Allow creating unique pod and hostname for KubernetesPodOperator from task job_id

2023-07-17 Thread via GitHub
juhai closed pull request #32278: Allow creating unique pod and hostname for KubernetesPodOperator from task job_id URL: https://github.com/apache/airflow/pull/32278 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [airflow] juhai commented on pull request #32278: Allow creating unique pod and hostname for KubernetesPodOperator from task job_id

2023-07-17 Thread via GitHub
juhai commented on PR #32278: URL: https://github.com/apache/airflow/pull/32278#issuecomment-1637778783 Closing as the current proposal isn't required to achieve what I need -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [airflow] amoghrajesh commented on a diff in pull request #32613: Revising instructions to run back referencing scripts

2023-07-17 Thread via GitHub
amoghrajesh commented on code in PR #32613: URL: https://github.com/apache/airflow/pull/32613#discussion_r1265155947 ## dev/README_RELEASE_PROVIDER_PACKAGES.md: ## @@ -393,8 +393,7 @@ breeze release-management publish-docs \ --override-versioned cd "${AIRFLOW_SITE_DIRECT

[GitHub] [airflow] amoghrajesh commented on a diff in pull request #32613: Revising instructions to run back referencing scripts

2023-07-17 Thread via GitHub
amoghrajesh commented on code in PR #32613: URL: https://github.com/apache/airflow/pull/32613#discussion_r1265156907 ## dev/README_RELEASE_PROVIDER_PACKAGES.md: ## @@ -393,8 +393,7 @@ breeze release-management publish-docs \ --override-versioned cd "${AIRFLOW_SITE_DIRECT

[GitHub] [airflow] xgao1023 commented on a diff in pull request #32520: allows users to write dag_id and task_id in their national characters, added display name for dag / task

2023-07-17 Thread via GitHub
xgao1023 commented on code in PR #32520: URL: https://github.com/apache/airflow/pull/32520#discussion_r1265163469 ## airflow/www/templates/airflow/task_instance.html: ## @@ -25,7 +25,7 @@ -Task Instance: {{ task_id }} +Task Instance: {{display_name}} Review

[GitHub] [airflow] xgao1023 commented on a diff in pull request #32520: allows users to write dag_id and task_id in their national characters, added display name for dag / task

2023-07-17 Thread via GitHub
xgao1023 commented on code in PR #32520: URL: https://github.com/apache/airflow/pull/32520#discussion_r1265165492 ## airflow/models/dag.py: ## @@ -3380,6 +3388,8 @@ class DagModel(Base): processor_subdir = Column(String(2000), nullable=True) # String representing the o

[GitHub] [airflow] phanikumv opened a new pull request, #32645: Refactor Sqlalchemy queries to 2.0 style (Part 6)

2023-07-17 Thread via GitHub
phanikumv opened a new pull request, #32645: URL: https://github.com/apache/airflow/pull/32645 This is a continuation of the effort to refactor the queries to sqlalchemy 2.0 related: https://github.com/apache/airflow/issues/28723 --- **^ Add meaningful descri

[GitHub] [airflow] pankajkoti opened a new pull request, #32646: Add arbitrary log shipper

2023-07-17 Thread via GitHub
pankajkoti opened a new pull request, #32646: URL: https://github.com/apache/airflow/pull/32646 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** f

[GitHub] [airflow] da-ekta-sharma commented on issue #32552: Airflow DAG is unable to create Task and skipping all remaining tasks with '0' downstream

2023-07-17 Thread via GitHub
da-ekta-sharma commented on issue #32552: URL: https://github.com/apache/airflow/issues/32552#issuecomment-1637926902 We created trigger_Fill_rate_QC_report task using triggerDagRunOperator and it previous(build_activate) and downstream task (send_notification) is PythonOperator. Then gener

[GitHub] [airflow] bkossakowska opened a new pull request, #32647: Dataplex dq - docs

2023-07-17 Thread via GitHub
bkossakowska opened a new pull request, #32647: URL: https://github.com/apache/airflow/pull/32647 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**

[GitHub] [airflow] bkossakowska closed pull request #32647: Dataplex dq - docs

2023-07-17 Thread via GitHub
bkossakowska closed pull request #32647: Dataplex dq - docs URL: https://github.com/apache/airflow/pull/32647 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

[GitHub] [airflow] otamxfcz opened a new issue, #32648: Tableau provider - AttributeError

2023-07-17 Thread via GitHub
otamxfcz opened a new issue, #32648: URL: https://github.com/apache/airflow/issues/32648 ### Apache Airflow version 2.6.3 ### What happened After updating to Airflow 2.6.3-python3.8 I have started to get this error message from Tableau provider/hook. I am not passing

[GitHub] [airflow] amoghrajesh opened a new pull request, #32649: Fixing breeze add-back-references command with doc

2023-07-17 Thread via GitHub
amoghrajesh opened a new pull request, #32649: URL: https://github.com/apache/airflow/pull/32649 A failure was reported in using this command. This PR intends to fix it. Reference: https://github.com/apache/airflow/pull/32613#discussion_r1264883314 --- **^ Add meaningfu

[GitHub] [airflow] amoghrajesh commented on pull request #32649: Fixing breeze add-back-references command with doc

2023-07-17 Thread via GitHub
amoghrajesh commented on PR #32649: URL: https://github.com/apache/airflow/pull/32649#issuecomment-1638074922 @eladkal this should fix the issue you ran into earlier today. Can you help with a review when you have some time? -- This is an automated message from the Apache Git Service. To

[GitHub] [airflow] A-Costa opened a new issue, #32650: GCS Sensor for matching glob

2023-07-17 Thread via GitHub
A-Costa opened a new issue, #32650: URL: https://github.com/apache/airflow/issues/32650 ### Description Since the new version of [`GCSHook.list`](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/_api/airflow/providers/google/cloud/hooks/gcs/index.html#airflow.pro

[GitHub] [airflow] A-Costa commented on issue #32650: GCS Sensor for matching glob

2023-07-17 Thread via GitHub
A-Costa commented on issue #32650: URL: https://github.com/apache/airflow/issues/32650#issuecomment-1638122053 I am currently working on a PR. Starting with [GCSObjectsWithPrefixExistenceSensor](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/_api/airflow/providers/goo

[GitHub] [airflow] uranusjr commented on issue #27399: CronTriggerTimetable lost one task occasionally

2023-07-17 Thread via GitHub
uranusjr commented on issue #27399: URL: https://github.com/apache/airflow/issues/27399#issuecomment-1638152940 I spent some time to take a closer look at the implementation. The problem with > I think the reason is because CronDataIntervalTimetable runs a last DAG run if it missed e

[GitHub] [airflow] uranusjr commented on issue #27399: CronTriggerTimetable lost one task occasionally

2023-07-17 Thread via GitHub
uranusjr commented on issue #27399: URL: https://github.com/apache/airflow/issues/27399#issuecomment-1638156248 Forgot to mentioned the one-line change. This line https://github.com/apache/airflow/blob/4a44343731144a7a7dc7fff7e3ed01663d4dd2e1/airflow/timetables/trigger.py#L92 s

[GitHub] [airflow] phanikumv closed pull request #32645: Refactor Sqlalchemy queries to 2.0 style (Part 6)

2023-07-17 Thread via GitHub
phanikumv closed pull request #32645: Refactor Sqlalchemy queries to 2.0 style (Part 6) URL: https://github.com/apache/airflow/pull/32645 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [airflow] eladkal commented on issue #32650: GCS Sensor for matching glob

2023-07-17 Thread via GitHub
eladkal commented on issue #32650: URL: https://github.com/apache/airflow/issues/32650#issuecomment-1638197464 I suggest to double check why we have `GCSObjectsWithPrefixExistenceSensor` & `GCSObjectExistenceSensor` to begin with? `prefix` should be a parameter of `GCSObjectExistenceSens

[GitHub] [airflow] potiuk commented on a diff in pull request #32613: Revising instructions to run back referencing scripts

2023-07-17 Thread via GitHub
potiuk commented on code in PR #32613: URL: https://github.com/apache/airflow/pull/32613#discussion_r1265406284 ## dev/README_RELEASE_PROVIDER_PACKAGES.md: ## @@ -393,8 +393,7 @@ breeze release-management publish-docs \ --override-versioned cd "${AIRFLOW_SITE_DIRECTORY}"

[GitHub] [airflow] amoghrajesh commented on a diff in pull request #32613: Revising instructions to run back referencing scripts

2023-07-17 Thread via GitHub
amoghrajesh commented on code in PR #32613: URL: https://github.com/apache/airflow/pull/32613#discussion_r1265407600 ## dev/README_RELEASE_PROVIDER_PACKAGES.md: ## @@ -393,8 +393,7 @@ breeze release-management publish-docs \ --override-versioned cd "${AIRFLOW_SITE_DIRECT

[GitHub] [airflow] potiuk commented on pull request #32649: Fixing breeze add-back-references command with doc

2023-07-17 Thread via GitHub
potiuk commented on PR #32649: URL: https://github.com/apache/airflow/pull/32649#issuecomment-1638217318 Somehow it has not been fixed in the previous `--airflow-site-directory DIRECTORY` -> this is what I referred to before - just remove it altoghether from all commands (including `--airf

[GitHub] [airflow] potiuk commented on a diff in pull request #32649: Fixing breeze add-back-references command with doc

2023-07-17 Thread via GitHub
potiuk commented on code in PR #32649: URL: https://github.com/apache/airflow/pull/32649#discussion_r1265419303 ## dev/README_RELEASE_AIRFLOW.md: ## @@ -802,10 +802,11 @@ Documentation for providers can be found in the ``/docs/apache-airflow`` directo ``` - Copy the doc

[GitHub] [airflow] mobuchowski commented on pull request #32604: Allow configuration to be contributed by providers

2023-07-17 Thread via GitHub
mobuchowski commented on PR #32604: URL: https://github.com/apache/airflow/pull/32604#issuecomment-1638226188 I like the approach, it definitely will be cleaner to have provider-related config in provider itself. Getting rid of confusing redundancy in `default_airflow.cfg` is also nice.

[GitHub] [airflow] amoghrajesh commented on pull request #32649: Fixing breeze add-back-references command with doc

2023-07-17 Thread via GitHub
amoghrajesh commented on PR #32649: URL: https://github.com/apache/airflow/pull/32649#issuecomment-1638231495 > --airflow-site-directory Hi @potiuk, just to understand better, should I not make any command in release management accept the `airflow-site-dierectory` parameter at all?

[GitHub] [airflow] amoghrajesh commented on pull request #32649: Fixing breeze add-back-references command with doc

2023-07-17 Thread via GitHub
amoghrajesh commented on PR #32649: URL: https://github.com/apache/airflow/pull/32649#issuecomment-1638231498 > --airflow-site-directory Hi @potiuk, just to understand better, should I not make any command in release management accept the `airflow-site-dierectory` parameter at all?

[GitHub] [airflow] potiuk commented on pull request #32604: Allow configuration to be contributed by providers

2023-07-17 Thread via GitHub
potiuk commented on PR #32604: URL: https://github.com/apache/airflow/pull/32604#issuecomment-1638232431 > I like the approach, it definitely will be cleaner to have provider-related config in provider itself. Getting rid of confusing redundancy in default_airflow.cfg is also nice. Y

[GitHub] [airflow] eladkal commented on a diff in pull request #32649: Fixing breeze add-back-references command with doc

2023-07-17 Thread via GitHub
eladkal commented on code in PR #32649: URL: https://github.com/apache/airflow/pull/32649#discussion_r1265426635 ## dev/README_RELEASE_PROVIDER_PACKAGES.md: ## @@ -392,8 +393,7 @@ breeze release-management publish-docs \ --package-filter 'apache-airflow-providers-*' \

[GitHub] [airflow] vincbeck commented on a diff in pull request #32573: Fix bug in prune_dict where empty dict and list would be removed even in strict mode

2023-07-17 Thread via GitHub
vincbeck commented on code in PR #32573: URL: https://github.com/apache/airflow/pull/32573#discussion_r1265444127 ## airflow/providers/amazon/aws/utils/__init__.py: ## @@ -18,16 +18,29 @@ import logging import re +import warnings from datetime import datetime from enum imp

[GitHub] [airflow] amoghrajesh commented on a diff in pull request #32649: Fixing breeze add-back-references command with doc

2023-07-17 Thread via GitHub
amoghrajesh commented on code in PR #32649: URL: https://github.com/apache/airflow/pull/32649#discussion_r1265444961 ## dev/README_RELEASE_PROVIDER_PACKAGES.md: ## @@ -392,8 +393,7 @@ breeze release-management publish-docs \ --package-filter 'apache-airflow-providers-*' \

[GitHub] [airflow] amoghrajesh commented on a diff in pull request #32649: Fixing breeze add-back-references command with doc

2023-07-17 Thread via GitHub
amoghrajesh commented on code in PR #32649: URL: https://github.com/apache/airflow/pull/32649#discussion_r1265445748 ## dev/README_RELEASE_PROVIDER_PACKAGES.md: ## @@ -381,6 +381,7 @@ If you have providers as list of provider ids because you just released them, yo ``` - Cop

[GitHub] [airflow] amoghrajesh commented on a diff in pull request #32649: Fixing breeze add-back-references command with doc

2023-07-17 Thread via GitHub
amoghrajesh commented on code in PR #32649: URL: https://github.com/apache/airflow/pull/32649#discussion_r1265446460 ## dev/README_RELEASE_PROVIDER_PACKAGES.md: ## @@ -392,8 +393,7 @@ breeze release-management publish-docs \ --package-filter 'apache-airflow-providers-*' \

[GitHub] [airflow] vincbeck commented on a diff in pull request #28900: Convert DagFileProcessor.execute_callbacks to Internal API

2023-07-17 Thread via GitHub
vincbeck commented on code in PR #28900: URL: https://github.com/apache/airflow/pull/28900#discussion_r1265453160 ## airflow/models/dagrun.py: ## @@ -100,6 +102,29 @@ def _creator_note(val): return DagRunNote(*val) +def _get_task_instance( Review Comment: You ar

[GitHub] [airflow] vincbeck commented on pull request #32626: Replace cassandra hack with fixture to fail badly named test files

2023-07-17 Thread via GitHub
vincbeck commented on PR #32626: URL: https://github.com/apache/airflow/pull/32626#issuecomment-1638316534 All our system tests are failing because of this: https://aws-mwaa.github.io/open-source/system-tests/dashboard.html ``` ERROR tests/system/providers/amazon/aws/example_cloudf

[GitHub] [airflow] vincbeck commented on pull request #32626: Replace cassandra hack with fixture to fail badly named test files

2023-07-17 Thread via GitHub
vincbeck commented on PR #32626: URL: https://github.com/apache/airflow/pull/32626#issuecomment-1638316931 All our system tests are failing because of this: https://aws-mwaa.github.io/open-source/system-tests/dashboard.html ``` ERROR tests/system/providers/amazon/aws/example_cloudf

[GitHub] [airflow] Lee-W commented on a diff in pull request #32513: EMR serverless Create/Start/Stop/Delete Application deferrable mode

2023-07-17 Thread via GitHub
Lee-W commented on code in PR #32513: URL: https://github.com/apache/airflow/pull/32513#discussion_r1265491042 ## airflow/providers/amazon/aws/hooks/emr.py: ## @@ -256,9 +256,14 @@ def __init__(self, *args: Any, **kwargs: Any) -> None: kwargs["client_type"] = "emr-serve

[GitHub] [airflow] Lee-W commented on a diff in pull request #32534: Add Deferrable mode for EMR Serverless Start Job Operator

2023-07-17 Thread via GitHub
Lee-W commented on code in PR #32534: URL: https://github.com/apache/airflow/pull/32534#discussion_r1265499110 ## airflow/providers/amazon/aws/operators/emr.py: ## @@ -1227,6 +1256,39 @@ def execute(self, context: Context) -> str | None: return self.job_id +def

[GitHub] [airflow] potiuk commented on pull request #32626: Replace cassandra hack with fixture to fail badly named test files

2023-07-17 Thread via GitHub
potiuk commented on PR #32626: URL: https://github.com/apache/airflow/pull/32626#issuecomment-1638332159 Yeah we should add exclusion in https://github.com/apache/airflow/pull/32626/files#diff-e52e4ddd58b7ef887ab03c04116e676f6280b824ab7469d5d3080e5cba4f2128R911 the check should allow "examp

[GitHub] [airflow] vincbeck commented on pull request #32626: Replace cassandra hack with fixture to fail badly named test files

2023-07-17 Thread via GitHub
vincbeck commented on PR #32626: URL: https://github.com/apache/airflow/pull/32626#issuecomment-163843 Yep, I can do that -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [airflow] ieunea1128 opened a new issue, #32652: AWS hook test_connection also needs endpoint_url

2023-07-17 Thread via GitHub
ieunea1128 opened a new issue, #32652: URL: https://github.com/apache/airflow/issues/32652 ### Apache Airflow version 2.6.3 ### What happened The initial code for testing an aws connection in my project had a bug. The bug caused the connection to be established with an i

[GitHub] [airflow] pankajkoti opened a new pull request, #32653: Update providers count

2023-07-17 Thread via GitHub
pankajkoti opened a new pull request, #32653: URL: https://github.com/apache/airflow/pull/32653 I was going through the document and realised that we now have almost 80+ providers. So felt a bit odd to read that it said only 60+. So quickly updating these values to 80 instead of 6

[GitHub] [airflow] Lee-W commented on a diff in pull request #32437: add deferrable mode to rds start & stop DB

2023-07-17 Thread via GitHub
Lee-W commented on code in PR #32437: URL: https://github.com/apache/airflow/pull/32437#discussion_r1265520231 ## airflow/providers/amazon/aws/operators/rds.py: ## @@ -791,23 +829,47 @@ def __init__( db_identifier: str, db_type: RdsDbType | str = RdsDbType.INST

[GitHub] [airflow] vincbeck opened a new pull request, #32655: Fix refuse_to_run_test_from_wrongly_named_files to handle system tests

2023-07-17 Thread via GitHub
vincbeck opened a new pull request, #32655: URL: https://github.com/apache/airflow/pull/32655 A check has been added in #32626 to enforce all test file names to be `test_*`. System tests are not named `test_*` but `example_*`. As a result, any attempt to execute a system test will fail. See

[GitHub] [airflow] A-Costa commented on issue #32650: GCS Sensor for matching glob

2023-07-17 Thread via GitHub
A-Costa commented on issue #32650: URL: https://github.com/apache/airflow/issues/32650#issuecomment-1638399128 I agree! It would make a lot of sense to generalise `GCSObjectExistenceSensor` and add both `prefix` and `glob` there as parameters -- This is an automated message from the Apach

[GitHub] [airflow] phanikumv closed pull request #32645: Refactor Sqlalchemy queries to 2.0 style (Part 6)

2023-07-17 Thread via GitHub
phanikumv closed pull request #32645: Refactor Sqlalchemy queries to 2.0 style (Part 6) URL: https://github.com/apache/airflow/pull/32645 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [airflow] dirrao commented on pull request #32249: Schedulers are issuing abrupt pod deletes when there is a delay in schedulers' heartbeat

2023-07-17 Thread via GitHub
dirrao commented on PR #32249: URL: https://github.com/apache/airflow/pull/32249#issuecomment-1638445760 @hussein-awala I have rebased the code with main branch and added unit test cases. Can you review it? -- This is an automated message from the Apache Git Service. To respond to th

[GitHub] [airflow] jedcunningham commented on a diff in pull request #32646: Add arbitrary log shipper

2023-07-17 Thread via GitHub
jedcunningham commented on code in PR #32646: URL: https://github.com/apache/airflow/pull/32646#discussion_r1265638391 ## airflow/config_templates/config.yml: ## @@ -913,6 +913,18 @@ logging: type: boolean example: ~ default: "False" +enable_arbitrary_lo

[GitHub] [airflow] ferruzzi commented on pull request #32591: D205 Support - Utils

2023-07-17 Thread via GitHub
ferruzzi commented on PR #32591: URL: https://github.com/apache/airflow/pull/32591#issuecomment-1638510448 Thanks @shubham22 , those sound great to me. I'll get those implemented some time today and push them along with the merge-conflict fix. :+1: -- This is an automated message from t

[GitHub] [airflow] pankajkoti commented on a diff in pull request #32646: Add arbitrary log shipper

2023-07-17 Thread via GitHub
pankajkoti commented on code in PR #32646: URL: https://github.com/apache/airflow/pull/32646#discussion_r1265644759 ## airflow/config_templates/config.yml: ## @@ -913,6 +913,18 @@ logging: type: boolean example: ~ default: "False" +enable_arbitrary_log_s

[GitHub] [airflow] ferruzzi commented on pull request #32575: D205 Support - Models

2023-07-17 Thread via GitHub
ferruzzi commented on PR #32575: URL: https://github.com/apache/airflow/pull/32575#issuecomment-1638547027 @uranusjr - Could you have another look when you get time? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [airflow] vincbeck commented on pull request #32655: Fix refuse_to_run_test_from_wrongly_named_files to handle system tests

2023-07-17 Thread via GitHub
vincbeck commented on PR #32655: URL: https://github.com/apache/airflow/pull/32655#issuecomment-1638547225 Closing the PR and re-open it with `use-public-runners` label. Tests are not picked up -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [airflow] vincbeck closed pull request #32655: Fix refuse_to_run_test_from_wrongly_named_files to handle system tests

2023-07-17 Thread via GitHub
vincbeck closed pull request #32655: Fix refuse_to_run_test_from_wrongly_named_files to handle system tests URL: https://github.com/apache/airflow/pull/32655 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[GitHub] [airflow] ferruzzi closed pull request #32591: D205 Support - Utils

2023-07-17 Thread via GitHub
ferruzzi closed pull request #32591: D205 Support - Utils URL: https://github.com/apache/airflow/pull/32591 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-ma

[GitHub] [airflow] vincbeck commented on pull request #28900: Convert DagFileProcessor.execute_callbacks to Internal API

2023-07-17 Thread via GitHub
vincbeck commented on PR #28900: URL: https://github.com/apache/airflow/pull/28900#issuecomment-1638554724 Closing the PR and re-open it with `use-public-runners` label. Tests are not picked up -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [airflow] vincbeck closed pull request #28900: Convert DagFileProcessor.execute_callbacks to Internal API

2023-07-17 Thread via GitHub
vincbeck closed pull request #28900: Convert DagFileProcessor.execute_callbacks to Internal API URL: https://github.com/apache/airflow/pull/28900 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] vincbeck opened a new pull request, #28900: Convert DagFileProcessor.execute_callbacks to Internal API

2023-07-17 Thread via GitHub
vincbeck opened a new pull request, #28900: URL: https://github.com/apache/airflow/pull/28900 Closes #28269 Closes #28784 Add the following methods to the internal API component: - DAG.fetch_callback - DAG.fetch_dagrun - SerializedDagModel.get_serialized_dag - TaskInstan

[GitHub] [airflow] Sighery opened a new issue, #32657: Increase connections HTTP login length to 5000 characters

2023-07-17 Thread via GitHub
Sighery opened a new issue, #32657: URL: https://github.com/apache/airflow/issues/32657 ### Description The current length limit for the `login` parameter in an HTTP connection is 500 characters. It'd be nice if this was 5000 characters like the `password` parameter. ### Use c

[GitHub] [airflow] o-nikolas commented on a diff in pull request #32589: BUGFIX: AWS ECS trigger returned "success" after max_attempts instead of failing

2023-07-17 Thread via GitHub
o-nikolas commented on code in PR #32589: URL: https://github.com/apache/airflow/pull/32589#discussion_r1265677100 ## airflow/providers/amazon/aws/triggers/ecs.py: ## @@ -179,8 +181,7 @@ async def run(self) -> AsyncIterator[TriggerEvent]: finally:

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #32573: Fix bug in prune_dict where empty dict and list would be removed even in strict mode

2023-07-17 Thread via GitHub
vandonr-amz commented on code in PR #32573: URL: https://github.com/apache/airflow/pull/32573#discussion_r1265684530 ## airflow/providers/amazon/aws/utils/__init__.py: ## @@ -18,16 +18,29 @@ import logging import re +import warnings from datetime import datetime from enum

[GitHub] [airflow] o-nikolas commented on a diff in pull request #32151: Get failure information on EMR job failure

2023-07-17 Thread via GitHub
o-nikolas commented on code in PR #32151: URL: https://github.com/apache/airflow/pull/32151#discussion_r1265684845 ## airflow/providers/amazon/aws/hooks/emr.py: ## @@ -169,16 +172,21 @@ def add_job_flow_steps( if wait_for_completion: waiter = self.get_conn(

[GitHub] [airflow] vincbeck commented on a diff in pull request #32573: Fix bug in prune_dict where empty dict and list would be removed even in strict mode

2023-07-17 Thread via GitHub
vincbeck commented on code in PR #32573: URL: https://github.com/apache/airflow/pull/32573#discussion_r1265687195 ## airflow/providers/amazon/aws/utils/__init__.py: ## @@ -18,16 +18,29 @@ import logging import re +import warnings from datetime import datetime from enum imp

[GitHub] [airflow] potiuk commented on a diff in pull request #32649: Fixing breeze add-back-references command with doc

2023-07-17 Thread via GitHub
potiuk commented on code in PR #32649: URL: https://github.com/apache/airflow/pull/32649#discussion_r1265687711 ## dev/README_RELEASE_PROVIDER_PACKAGES.md: ## @@ -392,8 +393,7 @@ breeze release-management publish-docs \ --package-filter 'apache-airflow-providers-*' \ -

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #32573: Fix bug in prune_dict where empty dict and list would be removed even in strict mode

2023-07-17 Thread via GitHub
vandonr-amz commented on code in PR #32573: URL: https://github.com/apache/airflow/pull/32573#discussion_r1265689489 ## airflow/providers/amazon/aws/utils/__init__.py: ## @@ -18,16 +18,29 @@ import logging import re +import warnings from datetime import datetime from enum

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #32573: Fix bug in prune_dict where empty dict and list would be removed even in strict mode

2023-07-17 Thread via GitHub
vandonr-amz commented on code in PR #32573: URL: https://github.com/apache/airflow/pull/32573#discussion_r1265690429 ## airflow/providers/amazon/aws/utils/__init__.py: ## @@ -18,16 +18,29 @@ import logging import re +import warnings from datetime import datetime from enum

[GitHub] [airflow] potiuk commented on pull request #32655: Fix refuse_to_run_test_from_wrongly_named_files to handle system tests

2023-07-17 Thread via GitHub
potiuk commented on PR #32655: URL: https://github.com/apache/airflow/pull/32655#issuecomment-1638594888 Full tests needed is the right label for it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [airflow] potiuk closed pull request #32655: Fix refuse_to_run_test_from_wrongly_named_files to handle system tests

2023-07-17 Thread via GitHub
potiuk closed pull request #32655: Fix refuse_to_run_test_from_wrongly_named_files to handle system tests URL: https://github.com/apache/airflow/pull/32655 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #32437: add deferrable mode to rds start & stop DB

2023-07-17 Thread via GitHub
vandonr-amz commented on code in PR #32437: URL: https://github.com/apache/airflow/pull/32437#discussion_r1265719265 ## airflow/providers/amazon/aws/triggers/rds.py: ## @@ -18,11 +18,17 @@ from typing import Any +from deprecated import deprecated + +from airflow.providers.a

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #32437: add deferrable mode to rds start & stop DB

2023-07-17 Thread via GitHub
vandonr-amz commented on code in PR #32437: URL: https://github.com/apache/airflow/pull/32437#discussion_r1265725694 ## airflow/providers/amazon/aws/utils/rds.py: ## @@ -22,5 +22,5 @@ class RdsDbType(Enum): """Only available types for the RDS.""" -INSTANCE: str = "in

[GitHub] [airflow] o-nikolas commented on pull request #32655: Fix refuse_to_run_test_from_wrongly_named_files to handle system tests

2023-07-17 Thread via GitHub
o-nikolas commented on PR #32655: URL: https://github.com/apache/airflow/pull/32655#issuecomment-1638639473 > > > Full tests needed is the right label for it > > > > > > @potiuk Does "Full tests needed" run on public runners? The issue we were trying to avoid is that it's actually

[GitHub] [airflow] o-nikolas commented on a diff in pull request #32604: Allow configuration to be contributed by providers

2023-07-17 Thread via GitHub
o-nikolas commented on code in PR #32604: URL: https://github.com/apache/airflow/pull/32604#discussion_r1265730580 ## TESTING.rst: ## @@ -55,8 +55,33 @@ Follow the guidelines when writing unit tests: tests, so we run Pytest with ``--disable-warnings`` but instead we have ``p

[GitHub] [airflow] ferruzzi commented on pull request #32655: Fix refuse_to_run_test_from_wrongly_named_files to handle system tests

2023-07-17 Thread via GitHub
ferruzzi commented on PR #32655: URL: https://github.com/apache/airflow/pull/32655#issuecomment-1638669108 FWIW I ran into this over [here](https://github.com/apache/airflow/pull/32591) as well, where the tests just stayed queued. I bumped them to a public runner since it's a small docs c

[GitHub] [airflow] amoghrajesh commented on a diff in pull request #32649: Fixing breeze add-back-references command with doc

2023-07-17 Thread via GitHub
amoghrajesh commented on code in PR #32649: URL: https://github.com/apache/airflow/pull/32649#discussion_r1265760108 ## dev/README_RELEASE_PROVIDER_PACKAGES.md: ## @@ -392,8 +393,7 @@ breeze release-management publish-docs \ --package-filter 'apache-airflow-providers-*' \

[GitHub] [airflow] vijayasarathib opened a new pull request, #32658: Documentation update to increase readability

2023-07-17 Thread via GitHub
vijayasarathib opened a new pull request, #32658: URL: https://github.com/apache/airflow/pull/32658 **Readability change in modules_management.rst** --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/

[GitHub] [airflow] amoghrajesh commented on pull request #32649: Fixing breeze add-back-references command with doc

2023-07-17 Thread via GitHub
amoghrajesh commented on PR #32649: URL: https://github.com/apache/airflow/pull/32649#issuecomment-1638684711 @potiuk @eladkal tested all the scenarios. They seem to work well with the current PR state now. -- This is an automated message from the Apache Git Service. To respond to the mes

[GitHub] [airflow] vijayasarathib opened a new issue, #32660: Fix the broken link in Apache git hub

2023-07-17 Thread via GitHub
vijayasarathib opened a new issue, #32660: URL: https://github.com/apache/airflow/issues/32660 ### What do you see as an issue? The "Suggest a change on this Page" link in the provided Airflow documentation at https://airflow.apache.org/docs/apache-airflow/2.3.2/modules_management.ht

[GitHub] [airflow] zachik250 opened a new issue, #32661: airflow 2.6.2 works from curl but not work when generate javaclient with same user password got 401, used to work on 2.3.0

2023-07-17 Thread via GitHub
zachik250 opened a new issue, #32661: URL: https://github.com/apache/airflow/issues/32661 ### Apache Airflow version Other Airflow 2 version (please specify below) ### What happened airflow 2.6.2 works from curl but not work when generate javaclient with same user passwo

[GitHub] [airflow] boring-cyborg[bot] commented on issue #32661: airflow 2.6.2 works from curl but not work when generate javaclient with same user password got 401, used to work on 2.3.0

2023-07-17 Thread via GitHub
boring-cyborg[bot] commented on issue #32661: URL: https://github.com/apache/airflow/issues/32661#issuecomment-1638686738 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for ap

[GitHub] [airflow] potiuk commented on a diff in pull request #32649: Fixing breeze add-back-references command with doc

2023-07-17 Thread via GitHub
potiuk commented on code in PR #32649: URL: https://github.com/apache/airflow/pull/32649#discussion_r1265773973 ## dev/breeze/src/airflow_breeze/commands/release_management_commands.py: ## @@ -821,7 +821,6 @@ def publish_docs( "--airflow-site-directory", envvar="AIRFLO

[GitHub] [airflow] potiuk commented on pull request #32655: Fix refuse_to_run_test_from_wrongly_named_files to handle system tests

2023-07-17 Thread via GitHub
potiuk commented on PR #32655: URL: https://github.com/apache/airflow/pull/32655#issuecomment-1638694545 Still waiting for someone to migrate to k8s operator if GitHub -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

[GitHub] [airflow] ferruzzi merged pull request #32655: Fix refuse_to_run_test_from_wrongly_named_files to handle system tests

2023-07-17 Thread via GitHub
ferruzzi merged PR #32655: URL: https://github.com/apache/airflow/pull/32655 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

[airflow] branch main updated: Fix refuse_to_run_test_from_wrongly_named_files to handle system tests (#32655)

2023-07-17 Thread ferruzzi
This is an automated email from the ASF dual-hosted git repository. ferruzzi pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 6064a6e3e7 Fix refuse_to_run_test_from_wrongly_na

[GitHub] [airflow] ferruzzi commented on a diff in pull request #32604: Allow configuration to be contributed by providers

2023-07-17 Thread via GitHub
ferruzzi commented on code in PR #32604: URL: https://github.com/apache/airflow/pull/32604#discussion_r1265768024 ## TESTING.rst: ## @@ -55,8 +55,33 @@ Follow the guidelines when writing unit tests: tests, so we run Pytest with ``--disable-warnings`` but instead we have ``py

[GitHub] [airflow] ferruzzi commented on a diff in pull request #32604: Allow configuration to be contributed by providers

2023-07-17 Thread via GitHub
ferruzzi commented on code in PR #32604: URL: https://github.com/apache/airflow/pull/32604#discussion_r1265782734 ## docs/apache-airflow-providers/index.rst: ## @@ -51,6 +51,15 @@ provider package and re-start Airflow, those become automatically available to A The summary of t

[GitHub] [airflow] ferruzzi commented on a diff in pull request #32604: Allow configuration to be contributed by providers

2023-07-17 Thread via GitHub
ferruzzi commented on code in PR #32604: URL: https://github.com/apache/airflow/pull/32604#discussion_r1265783122 ## docs/apache-airflow-providers/index.rst: ## @@ -51,6 +51,15 @@ provider package and re-start Airflow, those become automatically available to A The summary of t

[GitHub] [airflow] ferruzzi commented on a diff in pull request #32604: Allow configuration to be contributed by providers

2023-07-17 Thread via GitHub
ferruzzi commented on code in PR #32604: URL: https://github.com/apache/airflow/pull/32604#discussion_r1265784486 ## docs/exts/logging.rst.jinja2: ## @@ -22,7 +22,7 @@ :doc:`{{ provider }}:logging/index` -{% for handler in provider_dict['handlers'] -%} +{% for handler in pr

[GitHub] [airflow] ferruzzi commented on a diff in pull request #32604: Allow configuration to be contributed by providers

2023-07-17 Thread via GitHub
ferruzzi commented on code in PR #32604: URL: https://github.com/apache/airflow/pull/32604#discussion_r1265785329 ## docs/exts/operators_and_hooks_ref.py: ## @@ -341,10 +204,24 @@ def render_content(self, *, tags: set[str] | None, header_separator: str = DEFAU raise No

[GitHub] [airflow] amoghrajesh commented on a diff in pull request #32649: Fixing breeze add-back-references command with doc

2023-07-17 Thread via GitHub
amoghrajesh commented on code in PR #32649: URL: https://github.com/apache/airflow/pull/32649#discussion_r1265788557 ## dev/breeze/src/airflow_breeze/commands/release_management_commands.py: ## @@ -821,7 +821,6 @@ def publish_docs( "--airflow-site-directory", envvar="A

  1   2   >