[GitHub] [airflow] ephraimbuddy commented on issue #29199: TaskFlow AirflowSkipException causes downstream step to fail when multiple_outputs is true

2023-02-21 Thread via GitHub
ephraimbuddy commented on issue #29199: URL: https://github.com/apache/airflow/issues/29199#issuecomment-1439556135 > Where does the query go? It may (or may not) make sense depending on the situation. > > I think another problem is that _technically_ the error is correct in that

[GitHub] [airflow] ecodina commented on issue #28647: Intermitent log on deferrable operator

2023-02-21 Thread via GitHub
ecodina commented on issue #28647: URL: https://github.com/apache/airflow/issues/28647#issuecomment-1439555165 Hi, We have not written any custom log backend: we use ``self.log`` from within the custom operator and the only related configuration we've changed from ``airflow.cfg`` is

[GitHub] [airflow] ephraimbuddy commented on issue #29199: TaskFlow AirflowSkipException causes downstream step to fail when multiple_outputs is true

2023-02-21 Thread via GitHub
ephraimbuddy commented on issue #29199: URL: https://github.com/apache/airflow/issues/29199#issuecomment-1439554856 Here's the full diff: ```diff diff --git a/airflow/models/xcom_arg.py b/airflow/models/xcom_arg.py index 133fd4280b..046bc4a6e2 100644 ---

[GitHub] [airflow] eladkal commented on a diff in pull request #29548: Syedahsn/ec2 create terminate operators

2023-02-21 Thread via GitHub
eladkal commented on code in PR #29548: URL: https://github.com/apache/airflow/pull/29548#discussion_r1113927353 ## airflow/providers/amazon/aws/operators/ec2.py: ## @@ -116,3 +116,139 @@ def execute(self, context: Context): target_state="stopped",

[GitHub] [airflow] uranusjr commented on a diff in pull request #29548: Syedahsn/ec2 create terminate operators

2023-02-21 Thread via GitHub
uranusjr commented on code in PR #29548: URL: https://github.com/apache/airflow/pull/29548#discussion_r1113923418 ## airflow/providers/amazon/aws/operators/ec2.py: ## @@ -116,3 +116,139 @@ def execute(self, context: Context): target_state="stopped",

[GitHub] [airflow] uranusjr commented on issue #29366: default_args feature incompatible with Dynamic Task Mapping

2023-02-21 Thread via GitHub
uranusjr commented on issue #29366: URL: https://github.com/apache/airflow/issues/29366#issuecomment-1439541245 Intuitively it feels possible, except `default_args` is much more volatile to get right since it deals with unspecified arguments from indefinite operator classes. A poc would

[GitHub] [airflow] uranusjr commented on issue #29199: TaskFlow AirflowSkipException causes downstream step to fail when multiple_outputs is true

2023-02-21 Thread via GitHub
uranusjr commented on issue #29199: URL: https://github.com/apache/airflow/issues/29199#issuecomment-1439532491 Where does the query go? It may (or may not) make sense depending on the situation. I think another problem is that _technically_ the error is correct in that the XCom

[GitHub] [airflow] uranusjr commented on a diff in pull request #29245: fix code checking job names in sagemaker

2023-02-21 Thread via GitHub
uranusjr commented on code in PR #29245: URL: https://github.com/apache/airflow/pull/29245#discussion_r1113909968 ## airflow/providers/amazon/aws/operators/sagemaker.py: ## @@ -106,6 +108,41 @@ def _create_integer_fields(self) -> None: """ self.integer_fields

[GitHub] [airflow] github-actions[bot] commented on issue #19221: OpenAPI Generator generated python wrapper with a bug. It returns ApiValueError when calling get_variables().

2023-02-21 Thread via GitHub
github-actions[bot] commented on issue #19221: URL: https://github.com/apache/airflow/issues/19221#issuecomment-1439526971 This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last

[GitHub] [airflow] github-actions[bot] commented on issue #19511: With large number of DAGS, dag_run state takes long time to update after last task

2023-02-21 Thread via GitHub
github-actions[bot] commented on issue #19511: URL: https://github.com/apache/airflow/issues/19511#issuecomment-1439526941 This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last

[GitHub] [airflow] github-actions[bot] commented on issue #19934: Reloading of Logging and ORM in Dag Processing can be disruptive in spawn mode

2023-02-21 Thread via GitHub
github-actions[bot] commented on issue #19934: URL: https://github.com/apache/airflow/issues/19934#issuecomment-1439526909 This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last

[GitHub] [airflow] github-actions[bot] commented on issue #20125: Airflow Gantt View is improper

2023-02-21 Thread via GitHub
github-actions[bot] commented on issue #20125: URL: https://github.com/apache/airflow/issues/20125#issuecomment-1439526872 This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last

[GitHub] [airflow] ephraimbuddy commented on issue #29199: TaskFlow AirflowSkipException causes downstream step to fail when multiple_outputs is true

2023-02-21 Thread via GitHub
ephraimbuddy commented on issue #29199: URL: https://github.com/apache/airflow/issues/29199#issuecomment-1439526904 > THis would change the behaviour of `zip`. `ArgNotSet` is `fillvalue`’s default value, and it needs to cause `XComNotFound` for the mechanism to work correctly. So either

[GitHub] [airflow] potiuk commented on issue #29679: Fix Quarantined `test_cli_internal_api_background`

2023-02-21 Thread via GitHub
potiuk commented on issue #29679: URL: https://github.com/apache/airflow/issues/29679#issuecomment-1439524562 cc: @mhenc -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [airflow] uranusjr commented on pull request #29644: Remove <2.0.0 limit on google-cloud-bigtable

2023-02-21 Thread via GitHub
uranusjr commented on PR #29644: URL: https://github.com/apache/airflow/pull/29644#issuecomment-1439520760 The error is from a bug in our CI setup code. I opened a PR (linked above) to fix it. -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [airflow] uranusjr opened a new pull request, #29685: Do not crash when a version fails to parse

2023-02-21 Thread via GitHub
uranusjr opened a new pull request, #29685: URL: https://github.com/apache/airflow/pull/29685 Old files on PyPI may contain version numbers that are non-standard and can't be properly parsed. Those are no longer allowed for new versions, so we can safely ignore those versions since they

[GitHub] [airflow] uranusjr commented on a diff in pull request #29599: fix do_xcom_push=False bug in SnowflakeOperator

2023-02-21 Thread via GitHub
uranusjr commented on code in PR #29599: URL: https://github.com/apache/airflow/pull/29599#discussion_r1113899583 ## tests/providers/common/sql/operators/test_sql.py: ## @@ -89,6 +92,8 @@ def test_dont_xcom_push(self, mock_get_db_hook): handler=None,

[GitHub] [airflow] s0neq commented on a diff in pull request #29635: YandexCloud provider: support Yandex SDK feature "endpoint"

2023-02-21 Thread via GitHub
s0neq commented on code in PR #29635: URL: https://github.com/apache/airflow/pull/29635#discussion_r1113895749 ## airflow/providers/yandex/hooks/yandex.py: ## @@ -78,6 +78,11 @@ def get_connection_form_widgets() -> dict[str, Any]: description="Optional. This

[GitHub] [airflow] sangrpatil2 commented on issue #29555: Dag Fails while creating a Dynamic Tasks using airflow variables.

2023-02-21 Thread via GitHub
sangrpatil2 commented on issue #29555: URL: https://github.com/apache/airflow/issues/29555#issuecomment-1439511002 @hussein-awala / @ephraimbuddy Dag is only failing while creating dynamic tasks using airflow variables, the rest of the tasks/flow is working fine. You can refer to

[GitHub] [airflow] nervoussidd commented on pull request #29683: this file adds a on_kill method to Dataproc operator.

2023-02-21 Thread via GitHub
nervoussidd commented on PR #29683: URL: https://github.com/apache/airflow/pull/29683#issuecomment-1439486888 Thanks, I'll check it out. On Wed, Feb 22, 2023, 11:35 AM Tzu-ping Chung ***@***.***> wrote: > Start with reading one of the CONTRIBUTING guides listed here: >

[GitHub] [airflow] uranusjr commented on pull request #29683: this file adds a on_kill method to Dataproc operator.

2023-02-21 Thread via GitHub
uranusjr commented on PR #29683: URL: https://github.com/apache/airflow/pull/29683#issuecomment-1439485650 Start with reading one of the CONTRIBUTION guides listed here: https://github.com/apache/airflow/ -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] boring-cyborg[bot] commented on issue #29684: Not able to submit FlinkDeployment to Azure AKS Kubernetes Cluster

2023-02-21 Thread via GitHub
boring-cyborg[bot] commented on issue #29684: URL: https://github.com/apache/airflow/issues/29684#issuecomment-1439479817 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] tusharg1993 opened a new issue, #29684: Not able to submit FlinkDeployment to Azure AKS Kubernetes Cluster

2023-02-21 Thread via GitHub
7+', 'var': {'json': None, 'value': None}, 'conn': None, 'yesterday_ds': '2023-02-21', 'yesterday_ds_nodash': '20230221'} [2023-02-22T05:46:11.474+] {taskinstance.py:1851} ERROR - Task failed with exception Traceback (most recent call last): File "/home/airflow/.local/li

[GitHub] [airflow] nervoussidd commented on pull request #29683: this file adds a on_kill method to Dataproc operator.

2023-02-21 Thread via GitHub
nervoussidd commented on PR #29683: URL: https://github.com/apache/airflow/pull/29683#issuecomment-143947 Hello Tzu, No I didn't tested. I was looking for the guidance but I didn't knew whom to approach so I just learn some basic stuff like about the libraries and about the

[GitHub] [airflow] uranusjr commented on pull request #29683: this file adds a on_kill method to Dataproc operator.

2023-02-21 Thread via GitHub
uranusjr commented on PR #29683: URL: https://github.com/apache/airflow/pull/29683#issuecomment-1439463713 This is not valid Python. Did you test it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [airflow] nervoussidd closed pull request #29683: this file adds a on_kill method to Dataproc operator.

2023-02-21 Thread via GitHub
nervoussidd closed pull request #29683: this file adds a on_kill method to Dataproc operator. URL: https://github.com/apache/airflow/pull/29683 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #29683: this file adds a on_kill method to Dataproc operator.

2023-02-21 Thread via GitHub
boring-cyborg[bot] commented on PR #29683: URL: https://github.com/apache/airflow/pull/29683#issuecomment-1439451836 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution

[GitHub] [airflow] nervoussidd opened a new pull request, #29683: this file adds a on_kill method to Dataproc operator.

2023-02-21 Thread via GitHub
nervoussidd opened a new pull request, #29683: URL: https://github.com/apache/airflow/pull/29683 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for

[GitHub] [airflow] uranusjr commented on issue #29199: TaskFlow AirflowSkipException causes downstream step to fail when multiple_outputs is true

2023-02-21 Thread via GitHub
uranusjr commented on issue #29199: URL: https://github.com/apache/airflow/issues/29199#issuecomment-1439451086 THis would change the behaviour of `zip`. `ArgNotSet` is `fillvalue`’s default value, and it needs to cause `XComNotFound` for the mechanism to work correctly. So either further

[GitHub] [airflow] nervoussidd commented on issue #10385: Add on_kill method to DataprocWorkflowTemplateInstantiate[Inline]Operator

2023-02-21 Thread via GitHub
nervoussidd commented on issue #10385: URL: https://github.com/apache/airflow/issues/10385#issuecomment-1439417565 .take-issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [airflow] boushphong commented on issue #25024: Database connection with get_sqlalchemy_engine() results in an error because of a quoted uri

2023-02-21 Thread via GitHub
boushphong commented on issue #25024: URL: https://github.com/apache/airflow/issues/25024#issuecomment-1439413393 @shubham0473 You're still working on this? you mind if I take-over this issue? -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [airflow] josh-fell commented on pull request #29549: Fix and augment `check-for-inclusive-language` CI check

2023-02-21 Thread via GitHub
josh-fell commented on PR #29549: URL: https://github.com/apache/airflow/pull/29549#issuecomment-1439398230 Force pushing to hopefully get passed the test_cli_internal_api_background test now that it's quarantined. -- This is an automated message from the Apache Git Service. To respond

[GitHub] [airflow] r-richmond commented on issue #24730: Google CloudRun job operator

2023-02-21 Thread via GitHub
r-richmond commented on issue #24730: URL: https://github.com/apache/airflow/issues/24730#issuecomment-1439340843 > It also currently won't build with apache-airflow-providers-google because of incompatible protobuf support (see https://github.com/googleapis/python-run/issues/70).

[GitHub] [airflow] josh-fell commented on pull request #29678: FIx formatting of Dataset inlet/outlet note in TaskFlow concepts

2023-02-21 Thread via GitHub
josh-fell commented on PR #29678: URL: https://github.com/apache/airflow/pull/29678#issuecomment-1439340653 > Can it make sense to replace all (or almost all) such "Note: " places in ".rst" with the corresponding formatting? > > For instance, > >

[GitHub] [airflow] dimberman commented on pull request #29680: Google Cloud Providers - Introduce GoogleCloudBaseOperator

2023-02-21 Thread via GitHub
dimberman commented on PR #29680: URL: https://github.com/apache/airflow/pull/29680#issuecomment-1439325320 LGTM if the tests pass we can merge. That said if you could please add some context at the top. I realize that there is more info in the ticket but it would be nice to have a few

[GitHub] [airflow] 1yangsh commented on a diff in pull request #29378: Add gitSync optional env description

2023-02-21 Thread via GitHub
1yangsh commented on code in PR #29378: URL: https://github.com/apache/airflow/pull/29378#discussion_r1113741287 ## chart/values.yaml: ## @@ -1919,6 +1919,19 @@ dags: extraVolumeMounts: [] env: [] + # Change permissions on the checked-out files to the specified

[GitHub] [airflow] Taragolis commented on pull request #29681: Small Extension Fix of Internal API Configuration Path in init_views.py

2023-02-21 Thread via GitHub
Taragolis commented on PR #29681: URL: https://github.com/apache/airflow/pull/29681#issuecomment-1439266217 Is this fix also solve https://github.com/apache/airflow/issues/29679? [Tests / Quarantined

[GitHub] [airflow] bugraoz93 opened a new pull request, #29682: Migrate BaseJob.heartbeat to InternalAPI

2023-02-21 Thread via GitHub
bugraoz93 opened a new pull request, #29682: URL: https://github.com/apache/airflow/pull/29682 Migration of `BaseJob.heartbeat` to Internal API. closes: #29315 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [airflow] Taragolis commented on pull request #29661: Disable unixodbc and related packages from Microsoft APT repo

2023-02-21 Thread via GitHub
Taragolis commented on PR #29661: URL: https://github.com/apache/airflow/pull/29661#issuecomment-1439255547 At the moment when you decide not to wait anymore https://github.com/microsoft/linux-package-repositories/issues/36#issuecomment-1439222008 藍 藍 >Closing as the

[GitHub] [airflow] bugraoz93 opened a new pull request, #29681: Fix Extension of Internal API Configuration Path in init_views.py

2023-02-21 Thread via GitHub
bugraoz93 opened a new pull request, #29681: URL: https://github.com/apache/airflow/pull/29681 It is just a tiny extension change for the configuration path. **Context:** While working on [BaseJob.heartbeat to InternalAPI#29315](https://github.com/apache/airflow/issues/29315), I

[airflow] branch revert-29408-docker-compose-change-example updated (2b07e01cdc -> ab54df3039)

2023-02-21 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a change to branch revert-29408-docker-compose-change-example in repository https://gitbox.apache.org/repos/asf/airflow.git discard 2b07e01cdc Revert "Improve health checks in example docker-compose and

[airflow] branch main updated (9de301da2a -> 6c13f04365)

2023-02-21 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 9de301da2a FIx formatting of Dataset inlet/outlet note in TaskFlow concepts (#29678) add 6c13f04365 AWS Glue

[GitHub] [airflow] eladkal merged pull request #29659: AWS Glue job hook: Make s3_bucket parameter optional

2023-02-21 Thread via GitHub
eladkal merged PR #29659: URL: https://github.com/apache/airflow/pull/29659 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] eladkal closed issue #29423: GlueJobOperator throws error after migration to newest version of Airflow

2023-02-21 Thread via GitHub
eladkal closed issue #29423: GlueJobOperator throws error after migration to newest version of Airflow URL: https://github.com/apache/airflow/issues/29423 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [airflow] IKholopov commented on pull request #29518: Google Cloud Providers - Fix _MethodDefault deepcopy failure

2023-02-21 Thread via GitHub
IKholopov commented on PR #29518: URL: https://github.com/apache/airflow/pull/29518#issuecomment-1439217827 > Sounds good, created https://github.com/apache/airflow/pull/29680 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [airflow] IKholopov commented on a diff in pull request #29518: Google Cloud Providers - Fix _MethodDefault deepcopy failure

2023-02-21 Thread via GitHub
IKholopov commented on code in PR #29518: URL: https://github.com/apache/airflow/pull/29518#discussion_r1113667482 ## airflow/providers/google/cloud/operators/cloud_base.py: ## @@ -0,0 +1,34 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more

[GitHub] [airflow] IKholopov opened a new pull request, #29680: Google Cloud Providers - Introduce GoogleCloudBaseOperator

2023-02-21 Thread via GitHub
IKholopov opened a new pull request, #29680: URL: https://github.com/apache/airflow/pull/29680 As part of addressing #28751 the GoogleCloudBaseOperator is introduced to have common fix for all Google Cloud operators. Refactored to a separate PR per @potiuk request. -- This is an

[GitHub] [airflow] Taragolis commented on pull request #26946: Fix assume role if user explicit set credentials

2023-02-21 Thread via GitHub
Taragolis commented on PR #26946: URL: https://github.com/apache/airflow/pull/26946#issuecomment-1439190448 > Hi, currently I'm using airflow 2.4.3 with apache-airflow-providers-amazon 6.0.0, the issue that I'm facing is that I add the keys of some aws user and also add the role_arn in the

[GitHub] [airflow] crua-godaddy commented on pull request #26946: Fix assume role if user explicit set credentials

2023-02-21 Thread via GitHub
crua-godaddy commented on PR #26946: URL: https://github.com/apache/airflow/pull/26946#issuecomment-1439178281 Hi, currently I'm using airflow 2.4.3 with apache-airflow-providers-amazon 6.0.0, the issue that I'm facing is that I add the keys of some aws user and also add the role_arn in

[airflow] branch constraints-main updated: Updating constraints. Github run id:4237083783

2023-02-21 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new d6d0562b70 Updating

[GitHub] [airflow] Taragolis commented on pull request #29661: Disable unixodbc and related packages from Microsoft APT repo

2023-02-21 Thread via GitHub
Taragolis commented on PR #29661: URL: https://github.com/apache/airflow/pull/29661#issuecomment-1439171423 For comparison output from latest official Airflow Docker. By default PROD image also contains Debian's packages, however if user upgrade packages then `unixodbc` related packages

[airflow] branch revert-29408-docker-compose-change-example updated (a8124e3bbe -> 2b07e01cdc)

2023-02-21 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a change to branch revert-29408-docker-compose-change-example in repository https://gitbox.apache.org/repos/asf/airflow.git discard a8124e3bbe Revert "Improve health checks in example docker-compose and

[GitHub] [airflow] steren commented on a diff in pull request #28525: Add CloudRunExecuteJobOperator

2023-02-21 Thread via GitHub
steren commented on code in PR #28525: URL: https://github.com/apache/airflow/pull/28525#discussion_r1113612570 ## airflow/providers/google/cloud/operators/cloud_run.py: ## @@ -0,0 +1,115 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor

[GitHub] [airflow] steren commented on a diff in pull request #28525: Add CloudRunExecuteJobOperator

2023-02-21 Thread via GitHub
steren commented on code in PR #28525: URL: https://github.com/apache/airflow/pull/28525#discussion_r1113612282 ## airflow/providers/google/cloud/hooks/cloud_run.py: ## @@ -0,0 +1,245 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor

[GitHub] [airflow] Taragolis opened a new issue, #29679: Fix Quarantined `test_cli_internal_api_background`

2023-02-21 Thread via GitHub
Taragolis opened a new issue, #29679: URL: https://github.com/apache/airflow/issues/29679 ### Body Recently, [this test](https://github.com/apache/airflow/blob/9de301da2a44385f57be5407e80e16ee376f3d39/tests/cli/commands/test_internal_api_command.py#L134-L137) began to failed with

[GitHub] [airflow] eladkal commented on a diff in pull request #29635: YandexCloud provider: support Yandex SDK feature "endpoint"

2023-02-21 Thread via GitHub
eladkal commented on code in PR #29635: URL: https://github.com/apache/airflow/pull/29635#discussion_r1113600318 ## airflow/providers/yandex/hooks/yandex.py: ## @@ -78,6 +78,11 @@ def get_connection_form_widgets() -> dict[str, Any]: description="Optional. This

[GitHub] [airflow] eladkal closed issue #29607: Status of testing Providers that were prepared on February 18, 2023

2023-02-21 Thread via GitHub
eladkal closed issue #29607: Status of testing Providers that were prepared on February 18, 2023 URL: https://github.com/apache/airflow/issues/29607 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] eladkal commented on issue #29607: Status of testing Providers that were prepared on February 18, 2023

2023-02-21 Thread via GitHub
eladkal commented on issue #29607: URL: https://github.com/apache/airflow/issues/29607#issuecomment-1439130411 Thank you everyone. Providers are released I invite everyone to help improve providers for the next release, a list of open issues can be found

[GitHub] [airflow] vemikhaylov commented on pull request #29678: FIx formatting of Dataset inlet/outlet note in TaskFlow concepts

2023-02-21 Thread via GitHub
vemikhaylov commented on PR #29678: URL: https://github.com/apache/airflow/pull/29678#issuecomment-1439128327 Can it make sense to replace all (or almost all) such "Note: " places in ".rst" with the corresponding formatting? For instance,

[airflow] branch main updated (e8aa957439 -> 9de301da2a)

2023-02-21 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from e8aa957439 AWS system test sagemaker-endpoint: archive logs (#29581) add 9de301da2a FIx formatting of Dataset

[GitHub] [airflow] Taragolis merged pull request #29678: FIx formatting of Dataset inlet/outlet note in TaskFlow concepts

2023-02-21 Thread via GitHub
Taragolis merged PR #29678: URL: https://github.com/apache/airflow/pull/29678 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow-site] eladkal merged pull request #737: Add documentation for packages - 2023-02-18

2023-02-21 Thread via GitHub
eladkal merged PR #737: URL: https://github.com/apache/airflow-site/pull/737 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[airflow-site] branch add-documentation-2023-02-18 updated (c873115a9f -> 9f492b15f4)

2023-02-21 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a change to branch add-documentation-2023-02-18 in repository https://gitbox.apache.org/repos/asf/airflow-site.git omit c873115a9f Add documentation for packages - 2023-02-18 add 9f492b15f4 Add

svn commit: r60245 - /dev/airflow/providers/ /release/airflow/providers/

2023-02-21 Thread eladkal
Author: eladkal Date: Tue Feb 21 21:19:07 2023 New Revision: 60245 Log: Release Airflow Providers on 2023-02-21 Added: release/airflow/providers/apache-airflow-providers-amazon-7.2.1.tar.gz - copied unchanged from r60244,

[airflow] branch main updated (6699f953e6 -> e8aa957439)

2023-02-21 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 6699f953e6 Avoid modifying PRs in Recheck old bug reports workflow (#29653) add e8aa957439 AWS system test

[GitHub] [airflow] Taragolis merged pull request #29581: AWS system test sagemaker-endpoint: archive logs

2023-02-21 Thread via GitHub
Taragolis merged PR #29581: URL: https://github.com/apache/airflow/pull/29581 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] fritz-astronomer commented on pull request #29599: fix do_xcom_push=False bug in SnowflakeOperator

2023-02-21 Thread via GitHub
fritz-astronomer commented on PR #29599: URL: https://github.com/apache/airflow/pull/29599#issuecomment-1439063340 Did a quick rebase just to clean up commits. Tests are added and passing in breeze ✅ -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] josh-fell opened a new pull request, #29678: FIx formatting of Dataset inlet/outlet note in TaskFlow concepts

2023-02-21 Thread via GitHub
josh-fell opened a new pull request, #29678: URL: https://github.com/apache/airflow/pull/29678 **Before** https://user-images.githubusercontent.com/48934154/220452755-2f674ada-b405-4bc7-ab12-77215c8bcfd7.png;> **After**

[airflow] branch josh-fell-patch-1 created (now 773f21d95e)

2023-02-21 Thread joshfell
This is an automated email from the ASF dual-hosted git repository. joshfell pushed a change to branch josh-fell-patch-1 in repository https://gitbox.apache.org/repos/asf/airflow.git at 773f21d95e FIx formatting of Dataset inlet/outlet note in TaskFlow concepts This branch includes the

[airflow] 01/01: FIx formatting of Dataset inlet/outlet note in TaskFlow concepts

2023-02-21 Thread joshfell
This is an automated email from the ASF dual-hosted git repository. joshfell pushed a commit to branch josh-fell-patch-1 in repository https://gitbox.apache.org/repos/asf/airflow.git commit 773f21d95e3f3281f9d4ddfb4e3280c7553a1006 Author: Josh Fell <48934154+josh-f...@users.noreply.github.com>

[GitHub] [airflow] vemikhaylov commented on pull request #29608: Enable passing --xcom-args to tasks test CLI command

2023-02-21 Thread via GitHub
vemikhaylov commented on PR #29608: URL: https://github.com/apache/airflow/pull/29608#issuecomment-1439054387 Regarding the JSON direction potentially a better way would be to make it more explicitly structured like: ``` {"get_python_echo_message": {"key": "return_value", "value":

[GitHub] [airflow] r-richmond commented on pull request #29644: Remove <2.0.0 limit on google-cloud-bigtable

2023-02-21 Thread via GitHub
r-richmond commented on PR #29644: URL: https://github.com/apache/airflow/pull/29644#issuecomment-1439042843 This hit the same error that my local run of find newer dependencies did. Not sure what to do from here.. -- This is an automated message from the Apache Git Service. To respond

[GitHub] [airflow] vandonr-amz commented on pull request #29245: fix code checking job names in sagemaker

2023-02-21 Thread via GitHub
vandonr-amz commented on PR #29245: URL: https://github.com/apache/airflow/pull/29245#issuecomment-1439042642 hello @dimberman we just had a new system test failure because of this issue (getting throttled on too many listing requests). Any chance you can re-assess your review ? It'd be

[GitHub] [airflow] vincbeck commented on a diff in pull request #29627: Clarify `service_config` in AWS Connection

2023-02-21 Thread via GitHub
vincbeck commented on code in PR #29627: URL: https://github.com/apache/airflow/pull/29627#discussion_r1113526645 ## docs/apache-airflow-providers-amazon/connections/aws.rst: ## @@ -332,6 +317,29 @@ The following settings may be used within the ``assume_role_with_saml``

[GitHub] [airflow] vincbeck commented on a diff in pull request #29657: Add `wait_for_completion` param in `RedshiftCreateClusterOperator`

2023-02-21 Thread via GitHub
vincbeck commented on code in PR #29657: URL: https://github.com/apache/airflow/pull/29657#discussion_r1113524200 ## tests/providers/amazon/aws/operators/test_redshift_cluster.py: ## @@ -76,6 +77,10 @@ def test_create_single_node_cluster(self, mock_get_conn):

[airflow] branch main updated (1f1f97e666 -> 6699f953e6)

2023-02-21 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 1f1f97e666 Add Maciej Obuchowski to triage to help with AIP-53 issues (#29668) add 6699f953e6 Avoid modifying

[GitHub] [airflow] eladkal merged pull request #29653: Avoid modifying PRs in Recheck old bug reports workflow

2023-02-21 Thread via GitHub
eladkal merged PR #29653: URL: https://github.com/apache/airflow/pull/29653 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] vincbeck commented on pull request #29580: Allow to specify which connection, variable or config are being looked up in the backend using *_lookup_pattern parameters

2023-02-21 Thread via GitHub
vincbeck commented on PR #29580: URL: https://github.com/apache/airflow/pull/29580#issuecomment-1439024161 Thanks @dimberman for the feedbacks. I addressed them, feel free to review when you get a chance :) -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] potiuk commented on issue #29656: Generate URI/JSON for a connection from the Airflow Connection UI

2023-02-21 Thread via GitHub
potiuk commented on issue #29656: URL: https://github.com/apache/airflow/issues/29656#issuecomment-1439013943 Sure. If you think so. Without sensitive info it might be fine - the 'get as uri' is kinda cumbersome though :) -- This is an automated message from the Apache Git Service. To

[GitHub] [airflow] romibuzi commented on pull request #29659: AWS Glue job hook: Make s3_bucket parameter optional

2023-02-21 Thread via GitHub
romibuzi commented on PR #29659: URL: https://github.com/apache/airflow/pull/29659#issuecomment-1438999584 @vincbeck I have pushed fixes, checks are currently running it should be ok after -- This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [airflow] vincbeck commented on a diff in pull request #28241: Create Lambda create operator and sensor

2023-02-21 Thread via GitHub
vincbeck commented on code in PR #28241: URL: https://github.com/apache/airflow/pull/28241#discussion_r1113495450 ## airflow/providers/amazon/aws/operators/lambda_function.py: ## @@ -20,35 +20,122 @@ import json from typing import TYPE_CHECKING, Sequence +from

[GitHub] [airflow] vincbeck opened a new issue, #29677: Rename AWS lambda related resources

2023-02-21 Thread via GitHub
vincbeck opened a new issue, #29677: URL: https://github.com/apache/airflow/issues/29677 ### Apache Airflow Provider(s) amazon ### Versions of Apache Airflow Providers _No response_ ### Apache Airflow version 2.5.0 ### Operating System MacOS

[GitHub] [airflow] vincbeck commented on pull request #29659: AWS Glue job hook: Make s3_bucket parameter optional

2023-02-21 Thread via GitHub
vincbeck commented on PR #29659: URL: https://github.com/apache/airflow/pull/29659#issuecomment-1438972523 There are some static check failures though. Please read [documentation](https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#id4) in order to fix them. -- This is an

[GitHub] [airflow] mobuchowski opened a new issue, #29676: Implement system tests that confirm OpenLineage integration in selected provider works

2023-02-21 Thread via GitHub
mobuchowski opened a new issue, #29676: URL: https://github.com/apache/airflow/issues/29676 ### Body Airflow uses system tests as a way to test interaction of providers with external systems. We should use this mechanism to confirm lineage collection works as designed ###

[GitHub] [airflow] mobuchowski opened a new issue, #29675: Implement native OpenLineage integration for chosen other Provider

2023-02-21 Thread via GitHub
mobuchowski opened a new issue, #29675: URL: https://github.com/apache/airflow/issues/29675 ### Body In a first phase, we want to choose one specific provider, implement OpenLineage integration with it and make sure it plays well with prepared framework. It seems to me that

[GitHub] [airflow] mobuchowski opened a new issue, #29674: Migrate OpenLineage integration unit tests to Airflow test framework

2023-02-21 Thread via GitHub
mobuchowski opened a new issue, #29674: URL: https://github.com/apache/airflow/issues/29674 ### Body As part of this task, we also need to make sure they are properly run at CI. ### Committer - [X] I acknowledge that I am a maintainer/committer of the Apache Airflow

[GitHub] [airflow] mobuchowski opened a new issue, #29673: Provide OL SQL parser as internal OpenLineage provider API

2023-02-21 Thread via GitHub
mobuchowski opened a new issue, #29673: URL: https://github.com/apache/airflow/issues/29673 ### Body To easily provide OpenLineage events from SQL-based operators, we want to add SQL parser API that other providers can use. The API should be stable. There are two proposed

[GitHub] [airflow] vincbeck commented on a diff in pull request #29580: Allow to specify which connection, variable or config are being looked up in the backend using *_lookup_pattern parameters

2023-02-21 Thread via GitHub
vincbeck commented on code in PR #29580: URL: https://github.com/apache/airflow/pull/29580#discussion_r1113446539 ## airflow/providers/amazon/aws/secrets/secrets_manager.py: ## @@ -264,14 +286,14 @@ def get_conn_uri(self, conn_id: str) -> str | None: def

[GitHub] [airflow] mobuchowski opened a new issue, #29672: Adapt OpenLineage extractor framework code to make sure externally defined extractors work

2023-02-21 Thread via GitHub
mobuchowski opened a new issue, #29672: URL: https://github.com/apache/airflow/issues/29672 ### Body The OpenLineage provider should respect externally defined extractors, even if they are based on openlineage-airflow data types and classes. ### Committer - [X] I

[GitHub] [airflow] mobuchowski opened a new issue, #29671: Adapt OpenLineage default extractor to properly accept all OL implementation

2023-02-21 Thread via GitHub
mobuchowski opened a new issue, #29671: URL: https://github.com/apache/airflow/issues/29671 ### Body Adapt default extractor to accept any valid type returned from Operators `get_openlineage_facets_*` method. This needs to ensure compatibility with operators made with external

[GitHub] [airflow] eladkal commented on a diff in pull request #28241: Create Lambda create operator and sensor

2023-02-21 Thread via GitHub
eladkal commented on code in PR #28241: URL: https://github.com/apache/airflow/pull/28241#discussion_r1113441726 ## airflow/providers/amazon/aws/operators/lambda_function.py: ## @@ -20,35 +20,122 @@ import json from typing import TYPE_CHECKING, Sequence +from

[GitHub] [airflow] mobuchowski opened a new issue, #29670: Integrate OpenLineage provider configuration with general Airflow configuration

2023-02-21 Thread via GitHub
mobuchowski opened a new issue, #29670: URL: https://github.com/apache/airflow/issues/29670 ### Body Currently OL-Airflow is configured either with env variables (some things) or `openlineage.yml` file. We need to add a way to configure provider using Airflow config the same way

[GitHub] [airflow] ephraimbuddy commented on issue #29199: TaskFlow AirflowSkipException causes downstream step to fail when multiple_outputs is true

2023-02-21 Thread via GitHub
ephraimbuddy commented on issue #29199: URL: https://github.com/apache/airflow/issues/29199#issuecomment-1438935777 This fixes it: ```diff diff --git a/airflow/models/xcom_arg.py b/airflow/models/xcom_arg.py index 133fd4280b..1c7290f794 100644 --- a/airflow/models/xcom_arg.py

[GitHub] [airflow] mobuchowski opened a new issue, #29669: Create OpenLineage provider and migrate existing code to it.

2023-02-21 Thread via GitHub
mobuchowski opened a new issue, #29669: URL: https://github.com/apache/airflow/issues/29669 ### Body As first step in AIP-53 implementation, we want to migrate [existing OpenLineage Airflow integration](https://github.com/OpenLineage/OpenLineage/tree/main/integration/airflow), to

[GitHub] [airflow] shubham22 commented on pull request #29659: AWS Glue job hook: Make s3_bucket parameter optional

2023-02-21 Thread via GitHub
shubham22 commented on PR #29659: URL: https://github.com/apache/airflow/pull/29659#issuecomment-1438930972 cc: @vincbeck @syedahsn @ferruzzi @vandonr-amz - can you please review this if you've a chance? -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] Taragolis commented on pull request #29665: Quarantine `test_cli_internal_api_background`

2023-02-21 Thread via GitHub
Taragolis commented on PR #29665: URL: https://github.com/apache/airflow/pull/29665#issuecomment-1438930914 If I do not forget (50/50) I will create Task in Issues -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [airflow] fritz-astronomer commented on issue #29656: Generate URI/JSON for a connection from the Airflow Connection UI

2023-02-21 Thread via GitHub
fritz-astronomer commented on issue #29656: URL: https://github.com/apache/airflow/issues/29656#issuecomment-1438926634 There is an API route to get the connection - and as mentioned a CLI route. I think this request is reasonable to expose the same functionality in the UI - even

[GitHub] [airflow] jedcunningham commented on a diff in pull request #29378: Add gitSync optional env description

2023-02-21 Thread via GitHub
jedcunningham commented on code in PR #29378: URL: https://github.com/apache/airflow/pull/29378#discussion_r1113416556 ## chart/values.yaml: ## @@ -1919,6 +1919,19 @@ dags: extraVolumeMounts: [] env: [] + # Change permissions on the checked-out files to the

[GitHub] [airflow] pollynesterovich commented on pull request #29347: SSH Provider: Add cmd_timeout to ssh connection extra

2023-02-21 Thread via GitHub
pollynesterovich commented on PR #29347: URL: https://github.com/apache/airflow/pull/29347#issuecomment-1438908550 Please add these changes to the helm chart of the latest version 1.8.0 so it affects the work of dags. -- This is an automated message from the Apache Git Service. To

[airflow] branch main updated (66a8d102fc -> 1f1f97e666)

2023-02-21 Thread jedcunningham
This is an automated email from the ASF dual-hosted git repository. jedcunningham pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 66a8d102fc Quarantine `test_cli_internal_api_background` (#29665) add 1f1f97e666 Add Maciej Obuchowski to

  1   2   3   >