date:20221213



eladkal commented on PR #710:
URL: https://github.com/apache/airflow-site/pull/710#issuecomment-1350577796

   cc @ephraimbuddy 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated: Add `renamed` and `previous_name` in config sections (#28324)

2022-12-13 Thread ephraimanierobi

This is an automated email from the ASF dual-hosted git repository.

ephraimanierobi pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 9e5879576d Add `renamed` and `previous_name` in config sections 
(#28324)
9e5879576d is described below

commit 9e5879576d7427517dc33d2891f58d791db6c7bf
Author: Ephraim Anierobi 
AuthorDate: Wed Dec 14 08:51:26 2022 +0100

Add `renamed` and `previous_name` in config sections (#28324)

This helps us retain the version-added value of config options when
the config section is renamed.
This information of rename is also made available in the config 
documentation
---
 airflow/config_templates/config.yml|  3 +++
 dev/validate_version_added_fields_in_config.py | 14 ++
 docs/apache-airflow/configurations-ref.rst |  4 
 3 files changed, 21 insertions(+)

diff --git a/airflow/config_templates/config.yml 
b/airflow/config_templates/config.yml
index 2759eef2ff..962de33d9d 100644
--- a/airflow/config_templates/config.yml
+++ b/airflow/config_templates/config.yml
@@ -2368,6 +2368,9 @@
   default: "True"
 - name: kubernetes_executor
   description: ~
+  renamed:
+previous_name: kubernetes
+version: 2.5.0
   options:
 - name: pod_template_file
   description: |
diff --git a/dev/validate_version_added_fields_in_config.py 
b/dev/validate_version_added_fields_in_config.py
index a43e149175..e7920620d6 100755
--- a/dev/validate_version_added_fields_in_config.py
+++ b/dev/validate_version_added_fields_in_config.py
@@ -34,6 +34,9 @@ KNOWN_FALSE_DETECTIONS = {
 ("logging", "extra_logger_names", "2.2.0")
 }
 
+# Renamed sections: (new_section, old_section, version_before_renaming)
+RENAMED_SECTIONS = [("kubernetes_executor", "kubernetes", "2.4.3")]
+
 
 def fetch_pypi_versions() -> list[str]:
 r = requests.get("https://pypi.org/pypi/apache-airflow/json;)
@@ -71,6 +74,14 @@ def read_local_config_options() -> set[tuple[str, str, str]]:
 return config_options
 
 
+computed_option_new_section = set()
+for new_section, old_section, version_before_renaming in RENAMED_SECTIONS:
+options = fetch_config_options_for_version(version_before_renaming)
+options = {
+(new_section, option_name) for section_name, option_name in options if 
section_name == old_section
+}
+computed_option_new_section.update(options)
+
 # 1. Prepare versions to checks
 airflow_version = fetch_pypi_versions()
 airflow_version = sorted(airflow_version, key=semver.VersionInfo.parse)
@@ -83,6 +94,9 @@ for prev_version, curr_version in zip(to_check_versions[:-1], 
to_check_versions[
 options_1 = fetch_config_options_for_version(prev_version)
 options_2 = fetch_config_options_for_version(curr_version)
 new_options = options_2 - options_1
+# Remove existing options in new section
+new_options -= computed_option_new_section
+# Update expected options with version added field
 expected_computed_options.update(
 {(section_name, option_name, curr_version) for section_name, 
option_name in new_options}
 )
diff --git a/docs/apache-airflow/configurations-ref.rst 
b/docs/apache-airflow/configurations-ref.rst
index c77a33cd22..ab0ccd5f64 100644
--- a/docs/apache-airflow/configurations-ref.rst
+++ b/docs/apache-airflow/configurations-ref.rst
@@ -48,6 +48,10 @@ that you run airflow components on is synchronized (for 
example using ntpd) othe
 [{{ section["name"] }}]
 {{ "=" * (section["name"]|length + 2) }}
 
+{% if 'renamed' in section %}
+*Renamed in version {{ section['renamed']['version'] }}, previous name was 
{{ section['renamed']['previous_name'] }}*
+{% endif %}
+
 {% if section["description"] %}
 {{ section["description"] }}
 {% endif %}

[GitHub] [airflow] ephraimbuddy closed pull request #28076: Update main with 2.5.0 config values



ephraimbuddy closed pull request #28076: Update main with 2.5.0 config values
URL: https://github.com/apache/airflow/pull/28076


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ephraimbuddy merged pull request #28324: Add `version_renamed` and `previous_name` in config sections



ephraimbuddy merged PR #28324:
URL: https://github.com/apache/airflow/pull/28324


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated: Fix template rendered bucket_key in S3KeySensor (#28340)

This is an automated email from the ASF dual-hosted git repository.

eladkal pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 381160c0f6 Fix template rendered bucket_key in S3KeySensor (#28340)
381160c0f6 is described below

commit 381160c0f63a15957a631da9db875f98bb8e9d64
Author: Sung Yun <107272191+syu...@users.noreply.github.com>
AuthorDate: Wed Dec 14 02:47:46 2022 -0500

Fix template rendered bucket_key in S3KeySensor (#28340)
---
 airflow/providers/amazon/aws/sensors/s3.py|  7 --
 tests/providers/amazon/aws/sensors/test_s3_key.py | 27 +++
 2 files changed, 32 insertions(+), 2 deletions(-)

diff --git a/airflow/providers/amazon/aws/sensors/s3.py 
b/airflow/providers/amazon/aws/sensors/s3.py
index 40e31596d7..57c9393c0c 100644
--- a/airflow/providers/amazon/aws/sensors/s3.py
+++ b/airflow/providers/amazon/aws/sensors/s3.py
@@ -86,7 +86,7 @@ class S3KeySensor(BaseSensorOperator):
 ):
 super().__init__(**kwargs)
 self.bucket_name = bucket_name
-self.bucket_key = [bucket_key] if isinstance(bucket_key, str) else 
bucket_key
+self.bucket_key = bucket_key
 self.wildcard_match = wildcard_match
 self.check_fn = check_fn
 self.aws_conn_id = aws_conn_id
@@ -125,7 +125,10 @@ class S3KeySensor(BaseSensorOperator):
 return True
 
 def poke(self, context: Context):
-return all(self._check_key(key) for key in self.bucket_key)
+if isinstance(self.bucket_key, str):
+return self._check_key(self.bucket_key)
+else:
+return all(self._check_key(key) for key in self.bucket_key)
 
 def get_hook(self) -> S3Hook:
 """Create and return an S3Hook"""
diff --git a/tests/providers/amazon/aws/sensors/test_s3_key.py 
b/tests/providers/amazon/aws/sensors/test_s3_key.py
index 8d560e2c82..f0832d3df9 100644
--- a/tests/providers/amazon/aws/sensors/test_s3_key.py
+++ b/tests/providers/amazon/aws/sensors/test_s3_key.py
@@ -126,6 +126,33 @@ class TestS3KeySensor:
 
 mock_head_object.assert_called_once_with("key", "bucket")
 
+@mock.patch("airflow.providers.amazon.aws.sensors.s3.S3Hook.head_object")
+def test_parse_list_of_bucket_keys_from_jinja(self, mock_head_object):
+mock_head_object.return_value = None
+mock_head_object.side_effect = [{"ContentLength": 0}, 
{"ContentLength": 0}]
+
+Variable.set("test_bucket_key", ["s3://bucket/file1", 
"s3://bucket/file2"])
+
+execution_date = timezone.datetime(2020, 1, 1)
+
+dag = DAG("test_s3_key", start_date=execution_date, 
render_template_as_native_obj=True)
+op = S3KeySensor(
+task_id="s3_key_sensor",
+bucket_key="{{ var.value.test_bucket_key }}",
+bucket_name=None,
+dag=dag,
+)
+
+dag_run = DagRun(dag_id=dag.dag_id, execution_date=execution_date, 
run_id="test")
+ti = TaskInstance(task=op)
+ti.dag_run = dag_run
+context = ti.get_template_context()
+ti.render_templates(context)
+op.poke(None)
+
+mock_head_object.assert_any_call("file1", "bucket")
+mock_head_object.assert_any_call("file2", "bucket")
+
 @mock.patch("airflow.providers.amazon.aws.sensors.s3.S3Hook.head_object")
 def test_poke(self, mock_head_object):
 op = S3KeySensor(task_id="s3_key_sensor", 
bucket_key="s3://test_bucket/file")

[GitHub] [airflow] eladkal closed issue #28272: S3KeySensor 'bucket_key' instantiates as a nested list when rendered as a templated_field



eladkal closed issue #28272: S3KeySensor 'bucket_key' instantiates as a nested 
list when rendered as a templated_field
URL: https://github.com/apache/airflow/issues/28272


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #28340: Fix template rendered bucket_key in S3KeySensor



boring-cyborg[bot] commented on PR #28340:
URL: https://github.com/apache/airflow/pull/28340#issuecomment-1350563468

   Awesome work, congrats on your first merged pull request!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] eladkal merged pull request #28340: Fix template rendered bucket_key in S3KeySensor



eladkal merged PR #28340:
URL: https://github.com/apache/airflow/pull/28340


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] hugochinchilla commented on a diff in pull request #28333: Allow a DAG to be scheduled when any related DataSet is updated



hugochinchilla commented on code in PR #28333:
URL: https://github.com/apache/airflow/pull/28333#discussion_r1048098434


##
airflow/migrations/versions/0123_2_6_0_add_dataset_schedule_mode_to_dag_model.py:
##
@@ -0,0 +1,45 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""add dataset schedule mode to DAG model
+
+Revision ID: 7afd6d4021a9
+Revises: 290244fb8b83
+Create Date: 2022-12-13 11:34:26.648465
+
+"""
+
+from __future__ import annotations
+
+import sqlalchemy as sa
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision = "7afd6d4021a9"
+down_revision = "290244fb8b83"
+branch_labels = None
+depends_on = None
+airflow_version = "2.6.0"
+
+
+def upgrade():
+op.add_column("dag", sa.Column("run_on_any_dataset_changed", sa.Boolean(), 
nullable=False, default=False))

Review Comment:
   @shubham22 for the quorum case you would also neeed the amount of datasets 
updated over the total to aquire quorum, so you also will need a new column. 
Having full extensibility is ot that simple as making a column "more flexible".



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated (552d13be56 -> 9d9b15989a)

2022-12-13 Thread taragolis

This is an automated email from the ASF dual-hosted git repository.

taragolis pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


from 552d13be56 Add support for k8s 1.26 (#28320)
 add 9d9b15989a Create `LambdaCreateFunctionOperator` and sensor (#28241)

No new revisions were added by this update.

Summary of changes:
 .../providers/amazon/aws/hooks/lambda_function.py  |   7 +-
 .../amazon/aws/operators/lambda_function.py| 120 ++---
 .../aws/sensors/{athena.py => lambda_function.py}  |  67 +-
 airflow/providers/amazon/provider.yaml |   3 +
 .../operators/lambda.rst   |  35 -
 .../amazon/aws/hooks/test_lambda_function.py   | 148 +++--
 .../{test_lambda.py => test_lambda_function.py}|  45 ++-
 .../amazon/aws/sensors/test_lambda_function.py |  71 ++
 .../system/providers/amazon/aws/example_lambda.py  |  57 
 9 files changed, 396 insertions(+), 157 deletions(-)
 copy airflow/providers/amazon/aws/sensors/{athena.py => lambda_function.py} 
(52%)
 rename tests/providers/amazon/aws/operators/{test_lambda.py => 
test_lambda_function.py} (73%)
 create mode 100644 tests/providers/amazon/aws/sensors/test_lambda_function.py

[GitHub] [airflow] Taragolis merged pull request #28241: Create Lambda create operator and sensor



Taragolis merged PR #28241:
URL: https://github.com/apache/airflow/pull/28241


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] malthe commented on issue #28349: Status of testing Providers that were prepared on December 14, 2022



malthe commented on issue #28349:
URL: https://github.com/apache/airflow/issues/28349#issuecomment-1350522767

   We’ve tested the Azure log patch in our system, works great.
   
   ons. 14. dec. 2022 kl. 00.46 skrev eladkal ***@***.***>:
   
   > Body
   >
   > I have a kind request for all the contributors to the latest provider
   > packages release.
   > Could you please help us to test the RC versions of the providers?
   >
   > Let us know in the comment, whether the issue is addressed.
   >
   > Those are providers that require testing as there were some substantial
   > changes introduced:
   > Provider apache.hive: 5.0.0rc1
   > 
   >
   >- Move hive_cli_params to hook parameters (#28101)
   >: @potiuk
   >
   >- Improve filtering for invalid schemas in Hive hook (#27808)
   >: @PApostol
   >
   >
   > Provider microsoft.azure: 5.0.1rc1
   > 

   >
   >- Make arguments 'offset' and 'length' not required (#28234)
   >: @malthe
   >
   >
   > The guidelines on how to test providers can be found in
   >
   > Verify providers by contributors
   > 

   > Committer
   >
   >- I acknowledge that I am a maintainer/committer of the Apache Airflow
   >project.
   >
   > —
   > Reply to this email directly, view it on GitHub
   > , or unsubscribe
   > 

   > .
   > You are receiving this because you were mentioned.Message ID:
   > ***@***.***>
   >
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] amoghrajesh commented on pull request #28325: Adding an example dag for dynamic task mapping



amoghrajesh commented on PR #28325:
URL: https://github.com/apache/airflow/pull/28325#issuecomment-1350508583

   @potiuk @uranusjr The UTs are failing in the `validate_deserialized_task`, I 
am not sure how to debug it. Can someone guide me here? 
https://github.com/apache/airflow/actions/runs/3692197454/jobs/6251170704


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] amoghrajesh commented on pull request #28325: Adding an example dag for dynamic task mapping



amoghrajesh commented on PR #28325:
URL: https://github.com/apache/airflow/pull/28325#issuecomment-1350435244

   @potiuk @uranusjr I am running into this doc build error:
   ```
   ## Start docs build errors summary 
##
   
   == apache-airflow ==
   -- Error   1 
ERROR: Unexpected indentation.
   
   File path: apache-airflow/concepts/dynamic-task-mapping.rst (35)
   
 30 | 
 31 | In its simplest form you can map over a list defined directly in your 
DAG file using the ``expand()`` function instead of calling your task directly.
 32 | 
 33 | If you want to see a simple usage of Dynamic Task Mapping, you can 
look below:
 34 | .. exampleinclude:: 
/../../airflow/example_dags/example_dynamic_task_mapping.py
   > 35 | :language: python
 36 | :dedent: 4
 37 | 
 38 | This will show ``Total was 9`` in the task logs when executed.
 39 | 
 40 | This is the resulting DAG structure:
   
   ```
   
   I am not really able to figure out whats wrong here. The indent looks about 
right to me. Can i get some hint?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] shubham22 commented on a diff in pull request #28333: Allow a DAG to be scheduled when any related DataSet is updated



shubham22 commented on code in PR #28333:
URL: https://github.com/apache/airflow/pull/28333#discussion_r1047998173


##
airflow/migrations/versions/0123_2_6_0_add_dataset_schedule_mode_to_dag_model.py:
##
@@ -0,0 +1,45 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""add dataset schedule mode to DAG model
+
+Revision ID: 7afd6d4021a9
+Revises: 290244fb8b83
+Create Date: 2022-12-13 11:34:26.648465
+
+"""
+
+from __future__ import annotations
+
+import sqlalchemy as sa
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision = "7afd6d4021a9"
+down_revision = "290244fb8b83"
+branch_labels = None
+depends_on = None
+airflow_version = "2.6.0"
+
+
+def upgrade():
+op.add_column("dag", sa.Column("run_on_any_dataset_changed", sa.Boolean(), 
nullable=False, default=False))

Review Comment:
   I agree with @allebacco's suggestion. Yes, you should aim for simplest 
implementation that can be changed easily in the future when there is clarity 
on the full scope. However, your design should be extensible enough that it 
doesn't break customer's workflows/setup. Adding a boolean column may be too 
limiting. Instead, using `dataset_scheduling_mode` as a string would be a 
better option because it can be easily extended. For the "quorum" case, you 
could use a new `dataset_scheduling_mode` value of "QUORUM", which is generic 
enough to not require changes in the near future.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch constraints-main updated: Updating constraints. Build id:

2022-12-13 Thread github-bot

This is an automated email from the ASF dual-hosted git repository.

github-bot pushed a commit to branch constraints-main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/constraints-main by this push:
 new 208186901c Updating constraints. Build id:
208186901c is described below

commit 208186901cebbef04a774d3343037bfddcd88411
Author: Automated GitHub Actions commit 
AuthorDate: Wed Dec 14 03:36:52 2022 +

Updating constraints. Build id:

This update in constraints is automatically committed by the CI 
'constraints-push' step based on
HEAD of '' in ''
with commit sha .

All tests passed in this build so we determined we can push the updated 
constraints.

See 
https://github.com/apache/airflow/blob/main/README.md#installing-from-pypi for 
details.
---
 constraints-3.10.txt  | 19 ++-
 constraints-3.7.txt   | 19 ++-
 constraints-3.8.txt   | 19 ++-
 constraints-3.9.txt   | 19 ++-
 constraints-no-providers-3.10.txt |  2 +-
 constraints-no-providers-3.7.txt  |  2 +-
 constraints-no-providers-3.8.txt  |  2 +-
 constraints-no-providers-3.9.txt  |  2 +-
 constraints-source-providers-3.10.txt | 20 ++--
 constraints-source-providers-3.7.txt  | 20 ++--
 constraints-source-providers-3.8.txt  | 20 ++--
 constraints-source-providers-3.9.txt  | 20 ++--
 12 files changed, 84 insertions(+), 80 deletions(-)

diff --git a/constraints-3.10.txt b/constraints-3.10.txt
index 780cd49c8e..b28d4bb086 100644
--- a/constraints-3.10.txt
+++ b/constraints-3.10.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-12-12T20:21:40Z
+# This constraints file was automatically generated on 2022-12-14T03:36:10Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'apache-airflow' but installs
 # the providers from PIP-released packages at the moment of the constraint 
generation.
@@ -173,9 +173,9 @@ billiard==3.6.4.0
 black==22.12.0
 bleach==5.0.1
 blinker==1.5
-boto3==1.26.28
+boto3==1.26.29
 boto==2.49.0
-botocore==1.29.28
+botocore==1.29.29
 bowler==0.9.0
 cachelib==0.9.0
 cachetools==4.2.2
@@ -258,7 +258,7 @@ google-ads==18.0.0
 google-api-core==2.8.2
 google-api-python-client==1.12.11
 google-auth-httplib2==0.1.0
-google-auth-oauthlib==0.7.1
+google-auth-oauthlib==0.8.0
 google-auth==2.15.0
 google-cloud-aiplatform==1.16.1
 google-cloud-appengine-logging==1.1.3
@@ -327,10 +327,10 @@ incremental==22.10.0
 inflection==0.5.1
 influxdb-client==1.35.0
 iniconfig==1.1.1
-ipdb==0.13.9
+ipdb==0.13.11
 ipython==8.7.0
 isodate==0.6.1
-isort==5.10.1
+isort==5.11.2
 itsdangerous==2.1.2
 jaraco.classes==3.2.3
 jedi==0.18.2
@@ -359,7 +359,7 @@ linkify-it-py==2.0.0
 locket==1.0.0
 lockfile==0.12.2
 looker-sdk==22.20.0
-lxml==4.9.1
+lxml==4.9.2
 lz4==4.0.2
 markdown-it-py==2.1.0
 marshmallow-enum==1.5.1
@@ -383,7 +383,7 @@ msrestazure==0.6.4
 multi-key-dict==2.0.3
 multidict==6.0.3
 mypy-boto3-appflow==1.26.15
-mypy-boto3-rds==1.26.24
+mypy-boto3-rds==1.26.29
 mypy-boto3-redshift-data==1.26.0.post1
 mypy-extensions==0.4.3
 mypy==0.971
@@ -437,6 +437,7 @@ prompt-toolkit==3.0.36
 proto-plus==1.19.6
 protobuf==3.20.0
 psutil==5.9.4
+psycopg2-binary==2.9.5
 psycopg2==2.9.5
 ptyprocess==0.7.0
 pure-eval==0.2.2
@@ -478,7 +479,7 @@ pytest-rerunfailures==9.1.1
 pytest-timeouts==1.2.1
 pytest-xdist==3.1.0
 pytest==6.2.5
-python-arango==7.5.3
+python-arango==7.5.4
 python-daemon==2.3.2
 python-dateutil==2.8.2
 python-dotenv==0.21.0
diff --git a/constraints-3.7.txt b/constraints-3.7.txt
index 41b9391c7c..635314b1fd 100644
--- a/constraints-3.7.txt
+++ b/constraints-3.7.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-12-12T20:21:51Z
+# This constraints file was automatically generated on 2022-12-14T03:36:49Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'apache-airflow' but installs
 # the providers from PIP-released packages at the moment of the constraint 
generation.
@@ -173,9 +173,9 @@ billiard==3.6.4.0
 black==22.12.0
 bleach==5.0.1
 blinker==1.5
-boto3==1.26.28
+boto3==1.26.29
 boto==2.49.0
-botocore==1.29.28
+botocore==1.29.29
 bowler==0.9.0
 cached-property==1.5.2
 cachelib==0.9.0
@@ -258,7 +258,7 @@ google-ads==18.0.0
 google-api-core==2.8.2
 google-api-python-client==1.12.11
 google-auth-httplib2==0.1.0
-google-auth-oauthlib==0.7.1
+google-auth-oauthlib==0.8.0
 google-auth==2.15.0
 google-cloud-aiplatform==1.16.1
 google-cloud-appengine-logging==1.1.3
@@ -328,10 +328,10 @@ incremental==22.10.0
 inflection==0.5.1
 influxdb-client==1.35.0
 iniconfig==1.1.1
-ipdb==0.13.9

[GitHub] [airflow] rkarish commented on issue #27930: AIP-51 - Misc. Compatibility Checks



rkarish commented on issue #27930:
URL: https://github.com/apache/airflow/issues/27930#issuecomment-1350341087

   Hey Niko, I'll take this ticket if it's still available.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] doiken commented on pull request #26968: Fix sla checks being skipped when the preceding task instance misses the sla



doiken commented on PR #26968:
URL: https://github.com/apache/airflow/pull/26968#issuecomment-1350336810

   It has been about a month since the first approve.
   Is there anything I should do?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a diff in pull request #28208: Show mapped task groups in grid view



uranusjr commented on code in PR #28208:
URL: https://github.com/apache/airflow/pull/28208#discussion_r1047972569


##
airflow/www/views.py:
##
@@ -389,6 +391,65 @@ def get_summary(dag_run, children):
 "end_date": group_end_date,
 }
 
+def get_mapped_group_summaries(run_ids, children):
+child_ids = [child["id"] for child in children]
+mappedGroupQuery = session.query(
+TaskInstance.task_id, TaskInstance.state, TaskInstance.run_id, 
TaskInstance.map_index
+).filter(
+TaskInstance.dag_id == dag.dag_id,
+TaskInstance.task_id.in_(child_ids),
+TaskInstance.run_id.in_(run_ids),
+)
+
+mapped_instances = list(mappedGroupQuery)

Review Comment:
   ```suggestion
   mapped_instances = (
   session
   .query(
   TaskInstance.task_id, TaskInstance.state, 
TaskInstance.run_id, TaskInstance.map_index
   )
   .filter(
   TaskInstance.dag_id == dag.dag_id,
   TaskInstance.task_id.in_(child["id"] for child in 
children),
   TaskInstance.run_id.in_(run_ids),
   )
   )
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] BobDu commented on pull request #28198: fix: find zombies orm DetachedInstanceError when standalone dag processor mode



BobDu commented on PR #28198:
URL: https://github.com/apache/airflow/pull/28198#issuecomment-1350297325

   ping @potiuk  @kaxil @ashb @XD-DENG 
   sorry, anyone have time to look it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #28353: Update pipeline.rst

boring-cyborg[bot] commented on PR #28353:
URL: https://github.com/apache/airflow/pull/28353#issuecomment-1350293566

Congratulations on your first Pull Request and welcome to the Apache Airflow
community! If you have any issues or are unsure about any anything please check
our Contribution Guide
(https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, mypy and type
annotations). Our [pre-commits](
https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks)
will help you with that.
- In case of a new feature add useful documentation (in docstrings or in
`docs/` directory). Adding a new operator? Check this short
[guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst)
Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze
environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for
testing locally, it's a heavy docker but it ships with a working Airflow and a
lot of integrations.
- Be patient and persistent. It might take some time to get a review or get
the final approval from Committers.
- Please follow [ASF Code of
Conduct](https://www.apache.org/foundation/policies/conduct) for all
communication including (but not limited to) comments on Pull Requests, Mailing
list and Slack.
- Be sure to read the [Airflow Coding style](
https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it
better .
In case of doubts contact the developers at:
Mailing List: d...@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] avicol opened a new pull request, #28353: Update pipeline.rst



avicol opened a new pull request, #28353:
URL: https://github.com/apache/airflow/pull/28353

   The sql syntax in merge_data is broken for all the postgres versions. The 
try/catch block should be removed as somehow does not report the error
   
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a diff in pull request #28246: Better table name validation for db clean



uranusjr commented on code in PR #28246:
URL: https://github.com/apache/airflow/pull/28246#discussion_r1047949602


##
airflow/utils/db_cleanup.py:
##
@@ -337,8 +337,14 @@ def run_cleanup(
 :param session: Session representing connection to the metadata database.
 """
 clean_before_timestamp = timezone.coerce_datetime(clean_before_timestamp)
-effective_table_names = table_names if table_names else 
list(config_dict.keys())
-effective_config_dict = {k: v for k, v in config_dict.items() if k in 
effective_table_names}
+desired_table_names = set(table_names if table_names else 
config_dict.keys())
+effective_config_dict = {k: v for k, v in config_dict.items() if k in 
desired_table_names}
+effective_table_names = set(effective_config_dict.keys())
+if desired_table_names != effective_table_names:
+outliers = desired_table_names - effective_table_names
+logger.warning("The following table(s) are not valid choices and will 
be skipped: %s", outliers)

Review Comment:
   ```suggestion
   logger.warning("The following table(s) are not valid choices and 
will be skipped: %s", sorted(outliers))
   ```



##
airflow/utils/db_cleanup.py:
##
@@ -347,7 +353,7 @@ def run_cleanup(
 )
 _print_config(configs=effective_config_dict)
 if not dry_run and confirm:
-_confirm_delete(date=clean_before_timestamp, 
tables=list(effective_config_dict.keys()))
+_confirm_delete(date=clean_before_timestamp, 
tables=effective_table_names)

Review Comment:
   ```suggestion
   _confirm_delete(date=clean_before_timestamp, 
tables=sorted(effective_table_names))
   ```



##
airflow/utils/db_cleanup.py:
##
@@ -337,8 +337,14 @@ def run_cleanup(
 :param session: Session representing connection to the metadata database.
 """
 clean_before_timestamp = timezone.coerce_datetime(clean_before_timestamp)
-effective_table_names = table_names if table_names else 
list(config_dict.keys())
-effective_config_dict = {k: v for k, v in config_dict.items() if k in 
effective_table_names}
+desired_table_names = set(table_names if table_names else 
config_dict.keys())
+effective_config_dict = {k: v for k, v in config_dict.items() if k in 
desired_table_names}
+effective_table_names = set(effective_config_dict.keys())

Review Comment:
   ```suggestion
   effective_table_names = set(effective_config_dict)
   ```



##
airflow/utils/db_cleanup.py:
##
@@ -337,8 +337,14 @@ def run_cleanup(
 :param session: Session representing connection to the metadata database.
 """
 clean_before_timestamp = timezone.coerce_datetime(clean_before_timestamp)
-effective_table_names = table_names if table_names else 
list(config_dict.keys())
-effective_config_dict = {k: v for k, v in config_dict.items() if k in 
effective_table_names}
+desired_table_names = set(table_names if table_names else 
config_dict.keys())

Review Comment:
   ```suggestion
   desired_table_names = set(table_names or config_dict)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated (b8aae46aec -> 552d13be56)

2022-12-13 Thread jedcunningham

This is an automated email from the ASF dual-hosted git repository.

jedcunningham pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


from b8aae46aec Add provider common package release when releasing subset 
of providers (#28348)
 add 552d13be56 Add support for k8s 1.26 (#28320)

No new revisions were added by this update.

Summary of changes:
 README.md  |  2 +-
 dev/breeze/src/airflow_breeze/global_constants.py  |  2 +-
 docs/apache-airflow/installation/prerequisites.rst |  2 +-
 images/breeze/output-commands-hash.txt | 24 
 images/breeze/output_k8s_configure-cluster.svg | 54 +-
 images/breeze/output_k8s_create-cluster.svg|  4 +-
 images/breeze/output_k8s_delete-cluster.svg| 28 ++
 images/breeze/output_k8s_deploy-airflow.svg| 64 --
 images/breeze/output_k8s_k9s.svg   | 26 +
 images/breeze/output_k8s_logs.svg  | 28 ++
 images/breeze/output_k8s_run-complete-tests.svg| 62 +++--
 images/breeze/output_k8s_shell.svg | 34 +++-
 images/breeze/output_k8s_status.svg| 30 +-
 images/breeze/output_k8s_tests.svg | 62 +++--
 images/breeze/output_k8s_upload-k8s-image.svg  | 52 ++
 15 files changed, 257 insertions(+), 217 deletions(-)

[GitHub] [airflow] jedcunningham merged pull request #28320: Add support for k8s 1.26



jedcunningham merged PR #28320:
URL: https://github.com/apache/airflow/pull/28320


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk opened a new pull request, #28352: Move provider issue generation to breeze



potiuk opened a new pull request, #28352:
URL: https://github.com/apache/airflow/pull/28352

   The exercise where another PMC performed a release process had proven useful 
to show the value of Breeze in release management commands - there were no 
problems with running the Breeze commands, but for scripts put elsewhere (docs 
building, dev) it was a bit problematic because each required own virtualenv.
   
   This is the first PR to simplify that - one less venv for the issue 
generation process - now it is moved to Breeze so no more setup is needed.
   
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] jedcunningham commented on a diff in pull request #28324: Add `version_renamed` and `previous_name` in config sections



jedcunningham commented on code in PR #28324:
URL: https://github.com/apache/airflow/pull/28324#discussion_r1047906563


##
docs/apache-airflow/configurations-ref.rst:
##
@@ -48,6 +48,10 @@ that you run airflow components on is synchronized (for 
example using ntpd) othe
 [{{ section["name"] }}]
 {{ "=" * (section["name"]|length + 2) }}
 
+{% if 'renamed' in section %}
+*{{"Renamed in version "+ section['renamed']['version']+", previous name 
was "+ section['renamed']['previous_name'] }}*

Review Comment:
   ```suggestion
   *Renamed in version {{ section['renamed']['version'] }}, previous name 
was {{ section['renamed']['previous_name'] }}*
   ```
   
   nit, assuming this works this seems cleaner?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] RachitSharma2001 commented on a diff in pull request #28318: Add FTPSFileTransmitOperator



RachitSharma2001 commented on code in PR #28318:
URL: https://github.com/apache/airflow/pull/28318#discussion_r1047889098


##
tests/providers/ftp/operators/test_ftp.py:
##
@@ -184,3 +184,112 @@ def test_unequal_local_remote_file_paths(self):
 local_filepath=["/tmp/test1", "/tmp/test2"],
 remote_filepath="/tmp/test1",
 )
+
+
+class TestFTPSFileTransmitOperator:
+def setup_method(self):
+self.test_local_dir = "/tmp"
+self.test_local_dir_int = "/tmp/interdir"
+self.test_remote_dir = "/ftpshome"
+self.test_remote_dir_int = "/ftpshome/interdir"
+self.test_local_filename = "test_local_file"
+self.test_remote_filename = "test_remote_file"
+self.test_local_filepath = 
f"{self.test_local_dir}/{self.test_local_filename}"
+self.test_remote_filepath = 
f"{self.test_remote_dir}/{self.test_remote_filename}"
+self.test_local_filepath_int_dir = 
f"{self.test_local_dir_int}/{self.test_local_filename}"
+self.test_remote_filepath_int_dir = 
f"{self.test_remote_dir_int}/{self.test_remote_filename}"
+
+def teardown_method(self):
+if os.path.exists(self.test_local_dir_int):
+os.rmdir(self.test_local_dir_int)
+
+@mock.patch("airflow.providers.ftp.operators.ftp.FTPSHook.store_file")
+
@mock.patch("airflow.providers.ftp.operators.ftp.FTPSHook.create_directory")
+def test_file_transfer_put(self, mock_create_dir, mock_put):
+ftps_op = FTPSFileTransmitOperator(
+task_id="test_ftps_put",
+ftp_conn_id=DEFAULT_CONN_ID,
+local_filepath=self.test_local_filepath,
+remote_filepath=self.test_remote_filepath,
+operation=FTPOperation.PUT,
+)
+ftps_op.execute(None)
+assert not mock_create_dir.called
+mock_put.assert_called_once_with(self.test_remote_filepath, 
self.test_local_filepath)
+
+@mock.patch("airflow.providers.ftp.operators.ftp.FTPSHook.store_file")
+
@mock.patch("airflow.providers.ftp.operators.ftp.FTPSHook.create_directory")
+def test_file_transfer_with_intermediate_dir_put(self, mock_create_dir, 
mock_put):
+ftps_op = FTPSFileTransmitOperator(
+task_id="test_ftps_put_imm_dirs",
+ftp_conn_id=DEFAULT_CONN_ID,
+local_filepath=self.test_local_filepath,
+remote_filepath=self.test_remote_filepath_int_dir,
+operation=FTPOperation.PUT,
+create_intermediate_dirs=True,
+)
+ftps_op.execute(None)
+mock_create_dir.assert_called_with(self.test_remote_dir_int)
+mock_put.assert_called_once_with(self.test_remote_filepath_int_dir, 
self.test_local_filepath)
+
+@mock.patch("airflow.providers.ftp.operators.ftp.FTPSHook.retrieve_file")
+def test_file_transfer_get(self, mock_get):
+ftps_op = FTPSFileTransmitOperator(
+task_id="test_ftps_get",
+ftp_conn_id=DEFAULT_CONN_ID,
+local_filepath=self.test_local_filepath,
+remote_filepath=self.test_remote_filepath,
+operation=FTPOperation.GET,
+)
+ftps_op.execute(None)
+mock_get.assert_called_once_with(self.test_remote_filepath, 
self.test_local_filepath)
+
+@mock.patch("airflow.providers.ftp.operators.ftp.FTPSHook.retrieve_file")
+def test_file_transfer_with_intermediate_dir_get(self, mock_get):
+ftps_op = FTPSFileTransmitOperator(
+task_id="test_ftps_get_imm_dirs",
+ftp_conn_id=DEFAULT_CONN_ID,
+local_filepath=self.test_local_filepath_int_dir,
+remote_filepath=self.test_remote_filepath,
+operation=FTPOperation.GET,
+create_intermediate_dirs=True,
+)
+ftps_op.execute(None)
+assert os.path.exists(self.test_local_dir_int)
+mock_get.assert_called_once_with(self.test_remote_filepath, 
self.test_local_filepath_int_dir)
+
+@mock.patch("airflow.providers.ftp.operators.ftp.FTPSHook.retrieve_file")
+def test_multiple_paths_get(self, mock_get):
+local_filepath = ["/tmp/ltest1", "/tmp/ltest2"]
+remote_filepath = ["/tmp/rtest1", "/tmp/rtest2"]
+ftps_op = FTPSFileTransmitOperator(
+task_id="test_multiple_paths_get",
+ftp_conn_id=DEFAULT_CONN_ID,
+local_filepath=local_filepath,
+remote_filepath=remote_filepath,
+operation=FTPOperation.GET,
+)
+ftps_op.execute(None)
+assert mock_get.call_count == 2
+args0, _ = mock_get.call_args_list[0]
+args1, _ = mock_get.call_args_list[1]
+assert args0 == (remote_filepath[0], local_filepath[0])
+assert args1 == (remote_filepath[1], local_filepath[1])

Review Comment:
   @ferruzzi I've added this change as well. Let me know if it looks good and 
if any other change is needed.



-- 
This is an automated message from the Apache Git

[GitHub] [airflow] github-actions[bot] commented on pull request #25714: Adding ClickHouse Provider



github-actions[bot] commented on PR #25714:
URL: https://github.com/apache/airflow/pull/25714#issuecomment-1350135127

   This pull request has been automatically marked as stale because it has not 
had recent activity. It will be closed in 5 days if no further activity occurs. 
Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] closed pull request #26027: import_error_alert_emails



github-actions[bot] closed pull request #26027: import_error_alert_emails
URL: https://github.com/apache/airflow/pull/26027


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated: Add provider common package release when releasing subset of providers (#28348)

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new b8aae46aec Add provider common package release when releasing subset 
of providers (#28348)
b8aae46aec is described below

commit b8aae46aec9f291a9cf46c6c7ded2fcc99dcf9a6
Author: Jarek Potiuk 
AuthorDate: Wed Dec 14 00:55:05 2022 +0100

Add provider common package release when releasing subset of providers 
(#28348)
---
 dev/README_RELEASE_PROVIDER_PACKAGES.md | 1 +
 1 file changed, 1 insertion(+)

diff --git a/dev/README_RELEASE_PROVIDER_PACKAGES.md 
b/dev/README_RELEASE_PROVIDER_PACKAGES.md
index 1a231078eb..920527acf0 100644
--- a/dev/README_RELEASE_PROVIDER_PACKAGES.md
+++ b/dev/README_RELEASE_PROVIDER_PACKAGES.md
@@ -325,6 +325,7 @@ If we want to just release some providers you can release 
them in this way:
 ```shell script
 cd "${AIRFLOW_REPO_ROOT}"
 breeze build-docs --clean-build --for-production \
+  --package-filter apache-airflow-providers \
   --package-filter 'apache-airflow-providers-PACKAGE1' \
   --package-filter 'apache-airflow-providers-PACKAGE2' \
   ...

[GitHub] [airflow] potiuk merged pull request #28348: Add provider common package release when releasing subset of providers



potiuk merged PR #28348:
URL: https://github.com/apache/airflow/pull/28348


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow-site] branch add-documentation-2022-12-14 updated (0f52dc18c2 -> 9b9171dcb7)

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch add-documentation-2022-12-14
in repository https://gitbox.apache.org/repos/asf/airflow-site.git


from 0f52dc18c2 Add documentation for packages - 2022-12-14
 add 9b9171dcb7 fixup! Add documentation for packages - 2022-12-14

No new revisions were added by this update.

Summary of changes:
 sphinx_airflow_theme/demo/docs.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[GitHub] [airflow] eladkal opened a new issue, #28349: Status of testing Providers that were prepared on December 14, 2022



eladkal opened a new issue, #28349:
URL: https://github.com/apache/airflow/issues/28349

   ### Body
   
   I have a kind request for all the contributors to the latest provider 
packages release.
   Could you please help us to test the RC versions of the providers?
   
   Let us know in the comment, whether the issue is addressed.
   
   Those are providers that require testing as there were some substantial 
changes introduced:
   
   
   ## Provider [apache.hive: 
5.0.0rc1](https://pypi.org/project/apache-airflow-providers-apache-hive/5.0.0rc1)
  - [ ] [Move hive_cli_params to hook parameters 
(#28101)](https://github.com/apache/airflow/pull/28101): @potiuk
  - [ ] [Improve filtering for invalid schemas in Hive hook 
(#27808)](https://github.com/apache/airflow/pull/27808): @PApostol
   ## Provider [microsoft.azure: 
5.0.1rc1](https://pypi.org/project/apache-airflow-providers-microsoft-azure/5.0.1rc1)
  - [ ] [Make arguments offset and length not required 
(#28234)](https://github.com/apache/airflow/pull/28234): @malthe
   
   The guidelines on how to test providers can be found in
   
   [Verify providers by 
contributors](https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-by-contributors)
   
   
   
   ### Committer
   
   - [X] I acknowledge that I am a maintainer/committer of the Apache Airflow 
project.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow-site] potiuk commented on pull request #711: Add documentation for packages - 2022-12-14



potiuk commented on PR #711:
URL: https://github.com/apache/airflow-site/pull/711#issuecomment-1350076486

   looks like new shellcheck release broke our docs
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow-site] potiuk commented on pull request #711: Add documentation for packages - 2022-12-14



potiuk commented on PR #711:
URL: https://github.com/apache/airflow-site/pull/711#issuecomment-1350066767

   Restarted as the docs failed (I think temporary)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] snjypl commented on pull request #28161: WIP AIP-51 - Executor Coupling in Logging



snjypl commented on PR #28161:
URL: https://github.com/apache/airflow/pull/28161#issuecomment-1350057453

   > Marking my review as request for changes regarding unit testing (see 
[here](https://github.com/apache/airflow/pull/28161/files#r1044921868))
   @o-nikolas  i have added some unittests,  please take a look when you get a 
chance.  Thanks. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow-site] branch add-documentation-2022-12-14 created (now 0f52dc18c2)

This is an automated email from the ASF dual-hosted git repository.

eladkal pushed a change to branch add-documentation-2022-12-14
in repository https://gitbox.apache.org/repos/asf/airflow-site.git


  at 0f52dc18c2 Add documentation for packages - 2022-12-14

This branch includes the following new commits:

 new 0f52dc18c2 Add documentation for packages - 2022-12-14

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

[GitHub] [airflow] potiuk opened a new pull request, #28348: Add provider common package release when releasing subset of providers



potiuk opened a new pull request, #28348:
URL: https://github.com/apache/airflow/pull/28348

   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated (b6bc318f6f -> 81271424e3)

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


from b6bc318f6f Separate ER diagram dependencies to doc_gen extra (#28346)
 add 81271424e3 Remove docs package from Publish_docs.py and switch to 
python in shebang (#28347)

No new revisions were added by this update.

Summary of changes:
 docs/publish_docs.py | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

[GitHub] [airflow] potiuk merged pull request #28347: Make publish docs relative



potiuk merged PR #28347:
URL: https://github.com/apache/airflow/pull/28347


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated (7f9727fc45 -> b6bc318f6f)

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


from 7f9727fc45 Fix tag_providers.sh (#28345)
 add b6bc318f6f Separate ER diagram dependencies to doc_gen extra (#28346)

No new revisions were added by this update.

Summary of changes:
 CONTRIBUTING.rst   | 14 +++---
 INSTALL| 14 +++---
 docs/apache-airflow/extra-packages-ref.rst |  4 +++-
 setup.py   |  6 +-
 4 files changed, 22 insertions(+), 16 deletions(-)

[GitHub] [airflow] potiuk merged pull request #28346: Separate ER diagram dependencies to doc_gen extra



potiuk merged PR #28346:
URL: https://github.com/apache/airflow/pull/28346


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #28346: Separate ER diagram dependencies to doc_gen extra



potiuk commented on PR #28346:
URL: https://github.com/apache/airflow/pull/28346#issuecomment-1349990650

   > Will need also to edit 
https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#prepare-documentation
 replace the `pip install apache-airflow[doc]`
   
   Nope. Its ok. doc_gen is now only needed to generate diagrams (it's done in 
Breeze so it should be fine, The `[doc]` extra should be enough to publish docs
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk opened a new pull request, #28347: Make publish docs relative



potiuk opened a new pull request, #28347:
URL: https://github.com/apache/airflow/pull/28347

   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] eladkal commented on pull request #28346: Separate ER diagram dependencies to doc_gen extra



eladkal commented on PR #28346:
URL: https://github.com/apache/airflow/pull/28346#issuecomment-1349963309

   Will need also to edit
   
https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#prepare-documentation
   replace the `pip install apache-airflow[doc]`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk opened a new pull request, #28346: Separate ER diagram dependencies to doc_gen extra



potiuk opened a new pull request, #28346:
URL: https://github.com/apache/airflow/pull/28346

   The eralchemy dependency requires pygraphviz which does not play well with 
MacOS/M1. For those who want to easily install `doc` extra, it is a blocker.
   
   Now the extra is separated.
   
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ferruzzi commented on a diff in pull request #28338: New AWS sensor — DynamoDBValueSensor



ferruzzi commented on code in PR #28338:
URL: https://github.com/apache/airflow/pull/28338#discussion_r1047829104


##
airflow/providers/amazon/aws/example_dags/example_dynamodb_sensor.py:
##
@@ -0,0 +1,44 @@
+# Licensed to the Apache Software Foundation (ASF) under one

Review Comment:
   I actually noticed that after I made the comment and took a peek where we 
could squeeze it into the existing test and... we don't have one.  Sorry, I got 
distracted and forgot to come back and sort that out.
   
   I'd say it can be added into the existing [Dynamo-to-S3 system 
test](https://github.com/apache/airflow/blob/main/tests/system/providers/amazon/aws/example_dynamodb_to_s3.py).
  Drop the new sensor right after the create_table task perhaps? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ferruzzi commented on a diff in pull request #28338: New AWS sensor — DynamoDBValueSensor



ferruzzi commented on code in PR #28338:
URL: https://github.com/apache/airflow/pull/28338#discussion_r1047829104


##
airflow/providers/amazon/aws/example_dags/example_dynamodb_sensor.py:
##
@@ -0,0 +1,44 @@
+# Licensed to the Apache Software Foundation (ASF) under one

Review Comment:
   I actually noticed that after I made the comment and took a peek where we 
could squeeze it into the existing test and... we don't have one.  Sorry, I got 
distracted and forgot to come back and sort that out.
   
   I'd say it can be added into the existing [Dynamo-to-S3 system 
test](https://github.com/apache/airflow/blob/main/tests/system/providers/amazon/aws/example_dynamodb_to_s3.py).
  Drop the new sensor right after the table_setup perhaps? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] o-nikolas commented on a diff in pull request #28338: New AWS sensor — DynamoDBValueSensor



o-nikolas commented on code in PR #28338:
URL: https://github.com/apache/airflow/pull/28338#discussion_r1047823827


##
airflow/providers/amazon/aws/example_dags/example_dynamodb_sensor.py:
##
@@ -0,0 +1,44 @@
+# Licensed to the Apache Software Foundation (ASF) under one

Review Comment:
   Agreed with Dennis here, it would be nice to have a system test for this, 
but also we have no operators for creating a Dynamo DB table or even writing to 
it. So it would be a lot of custom code to just test the sensor. I don't feel 
too strongly either way, WDYT @ferruzzi?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] csm10495 commented on pull request #28239: Add a new config for adapting masked secrets to make it easier to prevent secret leakage in logs



csm10495 commented on PR #28239:
URL: https://github.com/apache/airflow/pull/28239#issuecomment-1349926073

   @ashb one more round. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] o-nikolas commented on a diff in pull request #28338: New AWS sensor — DynamoDBValueSensor



o-nikolas commented on code in PR #28338:
URL: https://github.com/apache/airflow/pull/28338#discussion_r1047812347


##
airflow/providers/amazon/aws/sensors/dynamodb.py:
##
@@ -0,0 +1,85 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Optional
+
+from botocore.exceptions import ClientError
+
+from airflow.providers.amazon.aws.hooks.dynamodb import DynamoDBHook
+from airflow.sensors.base import BaseSensorOperator
+
+if TYPE_CHECKING:
+from airflow.utils.context import Context
+
+
+class DynamoDBValueSensor(BaseSensorOperator):
+"""
+Waits for an attribute value to be present for an item in a DynamoDB table.
+
+:param partition_key_name: DynamoDB partition key name
+:param partition_key_value: DynamoDB partition key value
+:param attribute_name: DynamoDB attribute name
+:param attribute_value: DynamoDB attribute value
+:param sort_key_name: (optional) DynamoDB sort key name
+:param sort_key_value: (optional) DynamoDB sort key value
+"""
+
+def __init__(
+self,
+table_name: str,
+partition_key_name: str,
+partition_key_value: str,
+attribute_name: str,
+attribute_value: str,
+sort_key_name: Optional[str] = None,
+sort_key_value: Optional[str] = None,
+aws_conn_id: str = "aws_default",
+**kwargs,
+):
+super().__init__(**kwargs)
+self.table_name = table_name
+self.partition_key_name = partition_key_name
+self.partition_key_value = partition_key_value
+self.attribute_name = attribute_name
+self.attribute_value = attribute_value
+self.sort_key_name = sort_key_name
+self.sort_key_value = sort_key_value
+self.aws_conn_id = aws_conn_id
+self.hook = DynamoDBHook(aws_conn_id=self.aws_conn_id)
+self.table = self.hook.get_conn().Table(table_name)
+
+def poke(self, context: Context) -> bool:
+"""Test DynamoDB item for matching attribute value"""
+try:
+key = {self.partition_key_name: self.partition_key_value}
+msg = f"Checking table {self.table_name} for item PK: 
{self.partition_key_name}={self.partition_key_value}"
+
+if self.sort_key_value:
+key[self.sort_key_name] = self.sort_key_value
+msg += f" SK: {self.sort_key_name}={self.sort_key_value}"
+
+msg += f" attribute: {self.attribute_name}={self.attribute_value}"
+
+self.log.info(msg)
+response = self.table.get_item(Key=key)
+self.log.info(f"Response: {response}")
+return response["Item"][self.attribute_name] == 
self.attribute_value
+except ClientError as ex:
+self.log.error('AWS request failed, check logs for more info: %s', 
ex)
+return False

Review Comment:
   Do we want to keep looping if we're getting an exception back from the api?



##
airflow/providers/amazon/aws/sensors/dynamodb.py:
##
@@ -0,0 +1,85 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Optional
+
+from botocore.exceptions import ClientError
+
+from airflow.providers.amazon.aws.hooks.dynamodb import DynamoDBHook
+from airflow.sensors.base import BaseSensorOperator
+
+if TYPE_CHECKING:
+from airflow.utils.context import Context
+
+
+class

[airflow] branch main updated (609ca9091b -> 7f9727fc45)

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


from 609ca9091b Fix provider release instructions (#28344)
 add 7f9727fc45 Fix tag_providers.sh (#28345)

No new revisions were added by this update.

Summary of changes:
 dev/provider_packages/tag_providers.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[GitHub] [airflow] potiuk merged pull request #28345: Fix tag_providers.sh



potiuk merged PR #28345:
URL: https://github.com/apache/airflow/pull/28345


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] eladkal opened a new pull request, #28345: Fix tag_providers.sh



eladkal opened a new pull request, #28345:
URL: https://github.com/apache/airflow/pull/28345

   to support both https/ssh
   
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] denimalpaca commented on issue #28343: BigQueryColumnCheckOperator doesn't actually implement use_legacy_sql kwarg



denimalpaca commented on issue #28343:
URL: https://github.com/apache/airflow/issues/28343#issuecomment-1349778209

   Not entirely sure what's going on here, but it looks like it should be fine 
with the 
[`_BigQueryDbHookMixin`](https://github.com/apache/airflow/blob/main/airflow/providers/google/cloud/operators/bigquery.py#L115)
 class. That defines `get_hook()` which is using `self.use_legacy_sql`. So 
that's getting passed into the hook.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] annotated tag providers-apache-hive/5.0.0rc1 updated (1976ff72c5 -> 93bac40034)

This is an automated email from the ASF dual-hosted git repository.

eladkal pushed a change to annotated tag providers-apache-hive/5.0.0rc1
in repository https://gitbox.apache.org/repos/asf/airflow.git


*** WARNING: tag providers-apache-hive/5.0.0rc1 was modified! ***

from 1976ff72c5 (commit)
  to 93bac40034 (tag)
 tagging 1976ff72c52c3f5bf9f3e16329b55ab3888c3406 (commit)
 replaces providers-exasol/4.1.2
  by Elad Kalif
  on Tue Dec 13 23:37:53 2022 +0200

- Log -
Release 2022-12-13 of providers
---


No new revisions were added by this update.

Summary of changes:

[airflow] annotated tag providers-microsoft-azure/5.0.1rc1 updated (1976ff72c5 -> d2e3c0c6a2)

This is an automated email from the ASF dual-hosted git repository.

eladkal pushed a change to annotated tag providers-microsoft-azure/5.0.1rc1
in repository https://gitbox.apache.org/repos/asf/airflow.git


*** WARNING: tag providers-microsoft-azure/5.0.1rc1 was modified! ***

from 1976ff72c5 (commit)
  to d2e3c0c6a2 (tag)
 tagging 1976ff72c52c3f5bf9f3e16329b55ab3888c3406 (commit)
 replaces providers-exasol/4.1.2
  by Elad Kalif
  on Tue Dec 13 23:37:53 2022 +0200

- Log -
Release 2022-12-13 of providers
---


No new revisions were added by this update.

Summary of changes:

[GitHub] [airflow] Taragolis commented on a diff in pull request #28279: Add Amazon Elastic Container Registry (ECR) Hook



Taragolis commented on code in PR #28279:
URL: https://github.com/apache/airflow/pull/28279#discussion_r1047754974


##
tests/providers/amazon/aws/hooks/test_ecr.py:
##
@@ -0,0 +1,103 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from unittest import mock
+
+import boto3
+import pytest
+from moto import mock_ecr
+from moto.core import DEFAULT_ACCOUNT_ID
+
+from airflow.providers.amazon.aws.hooks.ecr import EcrHook
+
+
+@pytest.fixture
+def patch_hook(monkeypatch):
+"""Patch hook object by dummy boto3 ECR client."""
+ecr_client = boto3.client("ecr")
+monkeypatch.setattr(EcrHook, "conn", ecr_client)
+yield
+
+
+@mock_ecr
+class TestEcrHook:
+def test_service_type(self):
+"""Test expected boto3 client type."""
+assert EcrHook().client_type == "ecr"
+
+@pytest.mark.parametrize(
+"accounts_ids",
+[
+pytest.param("", id="empty-string"),
+pytest.param(None, id="none"),
+pytest.param([], id="empty-list"),
+],
+)

Review Comment:
   Yeah... Update it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Taragolis commented on a diff in pull request #28279: Add Amazon Elastic Container Registry (ECR) Hook



Taragolis commented on code in PR #28279:
URL: https://github.com/apache/airflow/pull/28279#discussion_r1047681828


##
tests/providers/amazon/aws/hooks/test_ecr.py:
##
@@ -0,0 +1,103 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from unittest import mock
+
+import boto3
+import pytest
+from moto import mock_ecr
+from moto.core import DEFAULT_ACCOUNT_ID
+
+from airflow.providers.amazon.aws.hooks.ecr import EcrHook
+
+
+@pytest.fixture
+def patch_hook(monkeypatch):
+"""Patch hook object by dummy boto3 ECR client."""
+ecr_client = boto3.client("ecr")
+monkeypatch.setattr(EcrHook, "conn", ecr_client)
+yield
+
+
+@mock_ecr
+class TestEcrHook:
+def test_service_type(self):
+"""Test expected boto3 client type."""
+assert EcrHook().client_type == "ecr"
+
+@pytest.mark.parametrize(
+"accounts_ids",
+[
+pytest.param("", id="empty-string"),
+pytest.param(None, id="none"),
+pytest.param([], id="empty-list"),
+],
+)

Review Comment:
   It is more about how it looks in `pytest` output and in PyCharm/IDEA.
   More refers how `pytest` work with empty string and objects.
   
   ---
   
   ```python
   @pytest.mark.parametrize("accounts_ids", ["", None, []])
   def test_sample(accounts_ids):
   pass
   ```
   ```console
   = test session starts 
==
   collecting ... collected 3 items
   
   test_docker.py::test_sample[] 
   test_docker.py::test_sample[None] 
   test_docker.py::test_sample[accounts_ids2] 
   
   == 3 passed in 0.12s 
===
   ```
   
![image](https://user-images.githubusercontent.com/3998685/207433355-09909185-0d4e-4373-af20-8983181a93f1.png)
   
   ---
   
   ```python
   @pytest.mark.parametrize(
   "accounts_ids",
   [
   pytest.param("", id="empty-string"),
   pytest.param(None, id="none"),
   pytest.param([], id="empty-list"),
   ],
   )
   def test_sample(accounts_ids):
   pass
   ```
   ```console
   = test session starts 
==
   collecting ... collected 3 items
   
   test_docker.py::test_sample[] 
   test_docker.py::test_sample[None] 
   test_docker.py::test_sample[accounts_ids2] 
   
   == 3 passed in 0.12s 
===
   ```
   
![image](https://user-images.githubusercontent.com/3998685/207433529-ca64fe1c-ba52-463a-89de-9e977fab6204.png)
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ferruzzi commented on a diff in pull request #28279: Add Amazon Elastic Container Registry (ECR) Hook



ferruzzi commented on code in PR #28279:
URL: https://github.com/apache/airflow/pull/28279#discussion_r1047747492


##
tests/providers/amazon/aws/hooks/test_ecr.py:
##
@@ -0,0 +1,103 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from unittest import mock
+
+import boto3
+import pytest
+from moto import mock_ecr
+from moto.core import DEFAULT_ACCOUNT_ID
+
+from airflow.providers.amazon.aws.hooks.ecr import EcrHook
+
+
+@pytest.fixture
+def patch_hook(monkeypatch):
+"""Patch hook object by dummy boto3 ECR client."""
+ecr_client = boto3.client("ecr")
+monkeypatch.setattr(EcrHook, "conn", ecr_client)
+yield
+
+
+@mock_ecr
+class TestEcrHook:
+def test_service_type(self):
+"""Test expected boto3 client type."""
+assert EcrHook().client_type == "ecr"
+
+@pytest.mark.parametrize(
+"accounts_ids",
+[
+pytest.param("", id="empty-string"),
+pytest.param(None, id="none"),
+pytest.param([], id="empty-list"),
+],
+)

Review Comment:
   Interesting.  Pretty sure the second block of terminal output is a mistaken 
repost of the first output, but I get what you are saying.  I don't think I've 
ever seen that used.  :+1: 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ferruzzi commented on a diff in pull request #28279: Add Amazon Elastic Container Registry (ECR) Hook



ferruzzi commented on code in PR #28279:
URL: https://github.com/apache/airflow/pull/28279#discussion_r1047744952


##
tests/providers/amazon/aws/hooks/test_ecr.py:
##
@@ -0,0 +1,103 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from unittest import mock
+
+import boto3
+import pytest
+from moto import mock_ecr
+from moto.core import DEFAULT_ACCOUNT_ID
+
+from airflow.providers.amazon.aws.hooks.ecr import EcrHook
+
+
+@pytest.fixture
+def patch_hook(monkeypatch):
+"""Patch hook object by dummy boto3 ECR client."""
+ecr_client = boto3.client("ecr")
+monkeypatch.setattr(EcrHook, "conn", ecr_client)
+yield
+
+
+@mock_ecr
+class TestEcrHook:
+def test_service_type(self):
+"""Test expected boto3 client type."""
+assert EcrHook().client_type == "ecr"
+
+@pytest.mark.parametrize(
+"accounts_ids",
+[
+pytest.param("", id="empty-string"),
+pytest.param(None, id="none"),
+pytest.param([], id="empty-list"),
+],
+)
+def test_get_temporary_credentials_default_account_id(self, patch_hook, 
accounts_ids):
+"""Test different types of empty account/registry ids."""
+result = EcrHook().get_temporary_credentials(registry_ids=accounts_ids)
+assert len(result) == 1
+assert result[0].username == "AWS"
+assert result[0].registry.startswith(DEFAULT_ACCOUNT_ID)
+assert result[0].password == f"{DEFAULT_ACCOUNT_ID}-auth-token"
+
+@pytest.mark.parametrize(
+"accounts_id, expected_registry",
+[
+pytest.param(DEFAULT_ACCOUNT_ID, DEFAULT_ACCOUNT_ID, 
id="moto-default-account"),
+pytest.param("", "", 
id="custom-account-id"),
+pytest.param([""], "", id="empty-list"),
+],
+)
+def test_get_temporary_credentials_single_account_id(self, patch_hook, 
accounts_id, expected_registry):

Review Comment:
   Sure, it was just a thought.  :+1: 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated: Fix provider release instructions (#28344)

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 609ca9091b Fix provider release instructions (#28344)
609ca9091b is described below

commit 609ca9091b9080e1c41e93d8aad179f720eea6c4
Author: eladkal <45845474+elad...@users.noreply.github.com>
AuthorDate: Tue Dec 13 22:55:21 2022 +0200

Fix provider release instructions (#28344)
---
 dev/README_RELEASE_PROVIDER_PACKAGES.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/dev/README_RELEASE_PROVIDER_PACKAGES.md 
b/dev/README_RELEASE_PROVIDER_PACKAGES.md
index e484be18f8..1a231078eb 100644
--- a/dev/README_RELEASE_PROVIDER_PACKAGES.md
+++ b/dev/README_RELEASE_PROVIDER_PACKAGES.md
@@ -214,7 +214,7 @@ svn update --set-depth=infinity asf-dist/dev/airflow
 cd asf-dist/dev/airflow/providers
 
 # Remove previously released providers
-rm -rf *
+svn rm *
 
 # Move the artifacts to svn folder
 mv ${AIRFLOW_REPO_ROOT}/dist/* .

[GitHub] [airflow] potiuk merged pull request #28344: Fix provider release instructions



potiuk merged PR #28344:
URL: https://github.com/apache/airflow/pull/28344


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] eladkal opened a new pull request, #28344: Fix provider release instructions



eladkal opened a new pull request, #28344:
URL: https://github.com/apache/airflow/pull/28344

   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

svn commit: r58708 - /dev/airflow/providers/

Author: eladkal
Date: Tue Dec 13 20:49:08 2022
New Revision: 58708

Log:
delete old providers

Removed:
dev/airflow/providers/apache-airflow-providers-exasol-4.1.2.tar.gz
dev/airflow/providers/apache-airflow-providers-exasol-4.1.2.tar.gz.asc
dev/airflow/providers/apache-airflow-providers-exasol-4.1.2.tar.gz.sha512
dev/airflow/providers/apache-airflow-providers-snowflake-4.0.2.tar.gz
dev/airflow/providers/apache-airflow-providers-snowflake-4.0.2.tar.gz.asc
dev/airflow/providers/apache-airflow-providers-snowflake-4.0.2.tar.gz.sha512
dev/airflow/providers/apache-airflow-providers-zendesk-4.2.0.tar.gz
dev/airflow/providers/apache-airflow-providers-zendesk-4.2.0.tar.gz.asc
dev/airflow/providers/apache-airflow-providers-zendesk-4.2.0.tar.gz.sha512
dev/airflow/providers/apache_airflow_providers_exasol-4.1.2-py3-none-any.whl

dev/airflow/providers/apache_airflow_providers_exasol-4.1.2-py3-none-any.whl.asc

dev/airflow/providers/apache_airflow_providers_exasol-4.1.2-py3-none-any.whl.sha512

dev/airflow/providers/apache_airflow_providers_snowflake-4.0.2-py3-none-any.whl

dev/airflow/providers/apache_airflow_providers_snowflake-4.0.2-py3-none-any.whl.asc

dev/airflow/providers/apache_airflow_providers_snowflake-4.0.2-py3-none-any.whl.sha512

dev/airflow/providers/apache_airflow_providers_zendesk-4.2.0-py3-none-any.whl

dev/airflow/providers/apache_airflow_providers_zendesk-4.2.0-py3-none-any.whl.asc

dev/airflow/providers/apache_airflow_providers_zendesk-4.2.0-py3-none-any.whl.sha512

[GitHub] [airflow] boring-cyborg[bot] commented on issue #28343: BigQueryColumnCheckOperator doesn't actually implement use_legacy_sql kwarg



boring-cyborg[bot] commented on issue #28343:
URL: https://github.com/apache/airflow/issues/28343#issuecomment-1349651539

   Thanks for opening your first issue here! Be sure to follow the issue 
template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] tmccall8829 opened a new issue, #28343: BigQueryColumnCheckOperator doesn't actually implement use_legacy_sql kwarg



tmccall8829 opened a new issue, #28343:
URL: https://github.com/apache/airflow/issues/28343

   ### Apache Airflow Provider(s)
   
   google
   
   ### Versions of Apache Airflow Providers
   
   `apache-airflow-providers-google==8.6.0`
   
   ### Apache Airflow version
   
   v2.3.4
   
   ### Operating System
   
   Composer (so I think debian, probably?)
   
   ### Deployment
   
   Composer
   
   ### Deployment details
   
   We use a standard/vanilla composer deployment setup -- nothing fancy.
   
   ### What happened
   
   Using the following BigQueryColumnCheckOperator block:
   ```
   # minimum_valid_fresh_date is a pendulum datetime obj
   basic_column_quality_checks = BigQueryColumnCheckOperator(
   task_id="check_columns",
   table="my_dataset.my_table",
   use_legacy_sql=False,
   column_mapping={
   "updated_at": {"min": {"geq_to": minimum_valid_fresh_date}},
   },
   )
   ```
   
   results in the following error in airflow:
   ```
   google.api_core.exceptions.BadRequest: 400 Querying tables with INTERVAL or 
JSON type is not supported in Legacy SQL: 787345995969:my_dataset.my_table.
   ```
   
   This occurs even though `use_legacy_sql` is set to `False`.
   
   ### What you think should happen instead
   
   Not sure if the problem here is that the error message is wrong/outdated, or 
if the `use_legacy_sql` flag isn't being actually passed through to 
`BigQueryHook()` when it gets called in 
`BigQueryColumnCheckOperator._submit_job()`. Either way, we should _not_ be 
using legacy SQL in this case, since we explicitly specify 
`use_legacy_sql=False`.
   
   ### How to reproduce
   
   Run a simple DAG with just a single task (the operator I pasted above) on 
airflow v2.3.4 with `apache-airflow-providers-google==8.6.0`. The BQ table in 
question should have a column `updated_at`.
   
   ### Anything else
   
   Digging into source, I _think_ the problem is that the returned hook in 
`BigQueryColumnCheckOperator._submit_job()` does not get passed 
`self.use_legacy_sql`, so the hook obj just uses the default (which is set to 
`True`). If it's really that simple, happy to submit a PR to fix!
   
   FWIW, it actually seems like this occurs in other BigQuery operators defined 
here as well... it could definitely be that I'm misreading the source, though, 
so apologies if that's the case!
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mrichman commented on a diff in pull request #28338: New AWS sensor — DynamoDBValueSensor



mrichman commented on code in PR #28338:
URL: https://github.com/apache/airflow/pull/28338#discussion_r1047716150


##
airflow/providers/amazon/aws/example_dags/example_dynamodb_sensor.py:
##
@@ -0,0 +1,44 @@
+# Licensed to the Apache Software Foundation (ASF) under one

Review Comment:
   Removed `example_dynamodb_sensor.py` from this PR for now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

svn commit: r58707 - /dev/airflow/providers/

Author: eladkal
Date: Tue Dec 13 20:35:43 2022
New Revision: 58707

Log:
Add artifacts for Airflow Providers 2022-12-13

Added:
dev/airflow/providers/apache-airflow-providers-apache-hive-5.0.0.tar.gz   
(with props)
dev/airflow/providers/apache-airflow-providers-apache-hive-5.0.0.tar.gz.asc

dev/airflow/providers/apache-airflow-providers-apache-hive-5.0.0.tar.gz.sha512
dev/airflow/providers/apache-airflow-providers-microsoft-azure-5.0.1.tar.gz 
  (with props)

dev/airflow/providers/apache-airflow-providers-microsoft-azure-5.0.1.tar.gz.asc

dev/airflow/providers/apache-airflow-providers-microsoft-azure-5.0.1.tar.gz.sha512

dev/airflow/providers/apache_airflow_providers_apache_hive-5.0.0-py3-none-any.whl
   (with props)

dev/airflow/providers/apache_airflow_providers_apache_hive-5.0.0-py3-none-any.whl.asc

dev/airflow/providers/apache_airflow_providers_apache_hive-5.0.0-py3-none-any.whl.sha512

dev/airflow/providers/apache_airflow_providers_microsoft_azure-5.0.1-py3-none-any.whl
   (with props)

dev/airflow/providers/apache_airflow_providers_microsoft_azure-5.0.1-py3-none-any.whl.asc

dev/airflow/providers/apache_airflow_providers_microsoft_azure-5.0.1-py3-none-any.whl.sha512

Added: dev/airflow/providers/apache-airflow-providers-apache-hive-5.0.0.tar.gz
==
Binary file - no diff available.

Propchange: 
dev/airflow/providers/apache-airflow-providers-apache-hive-5.0.0.tar.gz
--
svn:mime-type = application/octet-stream

Added: 
dev/airflow/providers/apache-airflow-providers-apache-hive-5.0.0.tar.gz.asc
==
--- dev/airflow/providers/apache-airflow-providers-apache-hive-5.0.0.tar.gz.asc 
(added)
+++ dev/airflow/providers/apache-airflow-providers-apache-hive-5.0.0.tar.gz.asc 
Tue Dec 13 20:35:43 2022
@@ -0,0 +1,7 @@
+-BEGIN PGP SIGNATURE-
+
+iIkEABYKADEWIQSDQO8ECQokO9vDRUWG4IhmPszevgUCY5jgORMcZWxhZGthbEBh
+cGFjaGUub3JnAAoJEIbgiGY+zN6+/QIA/RWr2RErNjBrtaDvmGn2XWqaTMtIZBnp
+UO7LgdIMKkTeAQD/82YK72XIBiUUmz6IDglnOkwXcFbC24yovqinu0qtCQ==
+=NsOH
+-END PGP SIGNATURE-

Added: 
dev/airflow/providers/apache-airflow-providers-apache-hive-5.0.0.tar.gz.sha512
==
--- 
dev/airflow/providers/apache-airflow-providers-apache-hive-5.0.0.tar.gz.sha512 
(added)
+++ 
dev/airflow/providers/apache-airflow-providers-apache-hive-5.0.0.tar.gz.sha512 
Tue Dec 13 20:35:43 2022
@@ -0,0 +1 @@
+cb9e76cea6b862e59d306b7839597ea4157362480557a92ffb20643d5e151ddbe689d3404bcdd6f7e307dcab197760732b8b4a03e891d27b82fc812568e2e54c
  apache-airflow-providers-apache-hive-5.0.0.tar.gz

Added: 
dev/airflow/providers/apache-airflow-providers-microsoft-azure-5.0.1.tar.gz
==
Binary file - no diff available.

Propchange: 
dev/airflow/providers/apache-airflow-providers-microsoft-azure-5.0.1.tar.gz
--
svn:mime-type = application/octet-stream

Added: 
dev/airflow/providers/apache-airflow-providers-microsoft-azure-5.0.1.tar.gz.asc
==
--- 
dev/airflow/providers/apache-airflow-providers-microsoft-azure-5.0.1.tar.gz.asc 
(added)
+++ 
dev/airflow/providers/apache-airflow-providers-microsoft-azure-5.0.1.tar.gz.asc 
Tue Dec 13 20:35:43 2022
@@ -0,0 +1,7 @@
+-BEGIN PGP SIGNATURE-
+
+iIkEABYKADEWIQSDQO8ECQokO9vDRUWG4IhmPszevgUCY5jgRRMcZWxhZGthbEBh
+cGFjaGUub3JnAAoJEIbgiGY+zN6+AOoA/2yV0ETQACGdBUxWlrs6UpIaSVbgIO4r
+heZZvXRypgpkAQCsQ8q4k1Ih38r80QOrE++Yb/AVadN2Y3RkOAAypxxADg==
+=evjt
+-END PGP SIGNATURE-

Added: 
dev/airflow/providers/apache-airflow-providers-microsoft-azure-5.0.1.tar.gz.sha512
==
--- 
dev/airflow/providers/apache-airflow-providers-microsoft-azure-5.0.1.tar.gz.sha512
 (added)
+++ 
dev/airflow/providers/apache-airflow-providers-microsoft-azure-5.0.1.tar.gz.sha512
 Tue Dec 13 20:35:43 2022
@@ -0,0 +1 @@
+dce5c3efc0e783fba43299c979006a44ba68f98ad10162ffce60cef096274c8df26f903ae61f3a2edd1a72eccf732c4aba181f989865fbc61239b12b15705ab5
  apache-airflow-providers-microsoft-azure-5.0.1.tar.gz

Added: 
dev/airflow/providers/apache_airflow_providers_apache_hive-5.0.0-py3-none-any.whl
==
Binary file - no diff available.

Propchange: 
dev/airflow/providers/apache_airflow_providers_apache_hive-5.0.0-py3-none-any.whl
--
svn:mime-type = application/octet-stream

Added:

[GitHub] [airflow] Taragolis commented on a diff in pull request #28279: Add Amazon Elastic Container Registry (ECR) Hook



Taragolis commented on code in PR #28279:
URL: https://github.com/apache/airflow/pull/28279#discussion_r1047691649


##
tests/providers/amazon/aws/hooks/test_ecr.py:
##
@@ -0,0 +1,103 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from unittest import mock
+
+import boto3
+import pytest
+from moto import mock_ecr
+from moto.core import DEFAULT_ACCOUNT_ID
+
+from airflow.providers.amazon.aws.hooks.ecr import EcrHook
+
+
+@pytest.fixture
+def patch_hook(monkeypatch):
+"""Patch hook object by dummy boto3 ECR client."""
+ecr_client = boto3.client("ecr")
+monkeypatch.setattr(EcrHook, "conn", ecr_client)
+yield
+
+
+@mock_ecr
+class TestEcrHook:
+def test_service_type(self):
+"""Test expected boto3 client type."""
+assert EcrHook().client_type == "ecr"
+
+@pytest.mark.parametrize(
+"accounts_ids",
+[
+pytest.param("", id="empty-string"),
+pytest.param(None, id="none"),
+pytest.param([], id="empty-list"),
+],
+)
+def test_get_temporary_credentials_default_account_id(self, patch_hook, 
accounts_ids):
+"""Test different types of empty account/registry ids."""
+result = EcrHook().get_temporary_credentials(registry_ids=accounts_ids)
+assert len(result) == 1
+assert result[0].username == "AWS"
+assert result[0].registry.startswith(DEFAULT_ACCOUNT_ID)
+assert result[0].password == f"{DEFAULT_ACCOUNT_ID}-auth-token"
+
+@pytest.mark.parametrize(
+"accounts_id, expected_registry",
+[
+pytest.param(DEFAULT_ACCOUNT_ID, DEFAULT_ACCOUNT_ID, 
id="moto-default-account"),
+pytest.param("", "", 
id="custom-account-id"),
+pytest.param([""], "", id="empty-list"),
+],
+)
+def test_get_temporary_credentials_single_account_id(self, patch_hook, 
accounts_id, expected_registry):

Review Comment:
   Oh... just split for implicit and explicit account id / regisrty id.
   If something changed in `moto`, AWS API (unlikely) or if one day we get rid 
of this line https://github.com/apache/airflow/pull/28279/files#r1047488028 it 
might safe a bit of time to realise where actual issue happen



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Taragolis commented on a diff in pull request #28279: Add Amazon Elastic Container Registry (ECR) Hook



Taragolis commented on code in PR #28279:
URL: https://github.com/apache/airflow/pull/28279#discussion_r1047687069


##
tests/providers/amazon/aws/hooks/test_ecr.py:
##
@@ -0,0 +1,103 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from unittest import mock
+
+import boto3
+import pytest
+from moto import mock_ecr
+from moto.core import DEFAULT_ACCOUNT_ID
+
+from airflow.providers.amazon.aws.hooks.ecr import EcrHook
+
+
+@pytest.fixture
+def patch_hook(monkeypatch):
+"""Patch hook object by dummy boto3 ECR client."""
+ecr_client = boto3.client("ecr")
+monkeypatch.setattr(EcrHook, "conn", ecr_client)
+yield
+
+
+@mock_ecr
+class TestEcrHook:
+def test_service_type(self):
+"""Test expected boto3 client type."""
+assert EcrHook().client_type == "ecr"
+
+@pytest.mark.parametrize(
+"accounts_ids",
+[
+pytest.param("", id="empty-string"),
+pytest.param(None, id="none"),
+pytest.param([], id="empty-list"),
+],
+)

Review Comment:
   And predictable cli command for run test for single parameter. It's useful 
during debagging when you change something and just one parameter stop working
   
   ```console
   ❯ pytest path/to.py::TestClassName:test_sample[empty-list] 
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated: Prepare ad hoc providers release (#28327)

This is an automated email from the ASF dual-hosted git repository.

eladkal pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 443df3b9c8 Prepare ad hoc providers release (#28327)
443df3b9c8 is described below

commit 443df3b9c8ef698e0204490c535f78c6c70276f3
Author: eladkal <45845474+elad...@users.noreply.github.com>
AuthorDate: Tue Dec 13 22:12:30 2022 +0200

Prepare ad hoc providers release (#28327)

* Prepare ad hoc providers release
This includes apache.hive and microsoft.azure
---
 airflow/providers/apache/hive/CHANGELOG.rst   |  8 
 airflow/providers/microsoft/azure/CHANGELOG.rst   | 13 +
 airflow/providers/microsoft/azure/provider.yaml   |  1 +
 docs/apache-airflow-providers-apache-hive/commits.rst | 15 ++-
 docs/apache-airflow-providers-apache-hive/index.rst   |  6 +++---
 docs/apache-airflow-providers-microsoft-azure/commits.rst | 14 +-
 docs/apache-airflow-providers-microsoft-azure/index.rst   |  6 +++---
 7 files changed, 55 insertions(+), 8 deletions(-)

diff --git a/airflow/providers/apache/hive/CHANGELOG.rst 
b/airflow/providers/apache/hive/CHANGELOG.rst
index 7086ab4a06..9a74537b6c 100644
--- a/airflow/providers/apache/hive/CHANGELOG.rst
+++ b/airflow/providers/apache/hive/CHANGELOG.rst
@@ -33,6 +33,14 @@ Breaking changes
 The ``hive_cli_params`` from connection were moved to the Hook. If you have 
extra parameters defined in your
 connections as ``hive_cli_params`` extra, you should move them to the DAG 
where your HiveOperator is used.
 
+* ``Move hive_cli_params to hook parameters (#28101)``
+
+Features
+
+
+* ``Improve filtering for invalid schemas in Hive hook (#27808)``
+
+
 4.1.1
 .
 
diff --git a/airflow/providers/microsoft/azure/CHANGELOG.rst 
b/airflow/providers/microsoft/azure/CHANGELOG.rst
index 5bfd784cc8..54a71b6596 100644
--- a/airflow/providers/microsoft/azure/CHANGELOG.rst
+++ b/airflow/providers/microsoft/azure/CHANGELOG.rst
@@ -24,6 +24,19 @@
 Changelog
 -
 
+5.0.1
+.
+
+
+Bug Fixes
+~
+
+* ``Make arguments 'offset' and 'length' not required (#28234)``
+
+.. Below changes are excluded from the changelog. Move them to
+   appropriate section above if needed. Do not delete the lines(!):
+
+
 5.0.0
 .
 
diff --git a/airflow/providers/microsoft/azure/provider.yaml 
b/airflow/providers/microsoft/azure/provider.yaml
index 7d8341f77f..43929ea747 100644
--- a/airflow/providers/microsoft/azure/provider.yaml
+++ b/airflow/providers/microsoft/azure/provider.yaml
@@ -21,6 +21,7 @@ name: Microsoft Azure
 description: |
 `Microsoft Azure `__
 versions:
+  - 5.0.1
   - 5.0.0
   - 4.3.0
   - 4.2.0
diff --git a/docs/apache-airflow-providers-apache-hive/commits.rst 
b/docs/apache-airflow-providers-apache-hive/commits.rst
index 8ab8d54a4e..bf9912540e 100644
--- a/docs/apache-airflow-providers-apache-hive/commits.rst
+++ b/docs/apache-airflow-providers-apache-hive/commits.rst
@@ -28,14 +28,27 @@ For high-level changelog, see :doc:`package information 
including changelog https://github.com/apache/airflow/commit/5f8481c799ea6bd742a5ccc194b2ff8dbe01eab5>`_
  2022-12-06   ``Move hive_cli_params to hook parameters (#28101)``
+`2d45f9d6c3 
`_
  2022-11-27   ``Improve filtering for invalid schemas in Hive hook (#27808)``
+=
  ===  ===
+
 4.1.1
 .
 
-Latest change: 2022-11-24
+Latest change: 2022-11-26
 
 
=
  ===  
 Commit 
CommittedSubject
 
=
  ===  
+`25bdbc8e67 
`_
  2022-11-26   ``Updated docs for RC3 wave of providers (#27937)``
 `2e20e9f7eb 
`_
  2022-11-24   ``Prepare for follow-up relase for November providers (#27774)``
 `80c327bd3b 
`_
  2022-11-24   ``Bump common.sql provider to 1.3.1 (#27888)``
 
=
  ===  
diff --git

[GitHub] [airflow] eladkal merged pull request #28327: Prepare ad hoc providers release



eladkal merged PR #28327:
URL: https://github.com/apache/airflow/pull/28327


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Taragolis commented on a diff in pull request #28279: Add Amazon Elastic Container Registry (ECR) Hook



Taragolis commented on code in PR #28279:
URL: https://github.com/apache/airflow/pull/28279#discussion_r1047681828


##
tests/providers/amazon/aws/hooks/test_ecr.py:
##
@@ -0,0 +1,103 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from unittest import mock
+
+import boto3
+import pytest
+from moto import mock_ecr
+from moto.core import DEFAULT_ACCOUNT_ID
+
+from airflow.providers.amazon.aws.hooks.ecr import EcrHook
+
+
+@pytest.fixture
+def patch_hook(monkeypatch):
+"""Patch hook object by dummy boto3 ECR client."""
+ecr_client = boto3.client("ecr")
+monkeypatch.setattr(EcrHook, "conn", ecr_client)
+yield
+
+
+@mock_ecr
+class TestEcrHook:
+def test_service_type(self):
+"""Test expected boto3 client type."""
+assert EcrHook().client_type == "ecr"
+
+@pytest.mark.parametrize(
+"accounts_ids",
+[
+pytest.param("", id="empty-string"),
+pytest.param(None, id="none"),
+pytest.param([], id="empty-list"),
+],
+)

Review Comment:
   It is more about how it looks in `pytest` output and in PyCharm/IDEA.
   More refers how `pytest` work with empty string and objects.
   
   ---
   
   ```python
   @pytest.mark.parametrize("accounts_ids", ["", None, []])
   def test_sample(accounts_ids):
   pass
   ```
   ```console
   = test session starts 
==
   collecting ... collected 3 items
   
   test_docker.py::test_sample[] 
   test_docker.py::test_sample[None] 
   test_docker.py::test_sample[accounts_ids2] 
   
   == 3 passed in 0.12s 
===
   ```
   
![image](https://user-images.githubusercontent.com/3998685/207433355-09909185-0d4e-4373-af20-8983181a93f1.png)
   
   ---
   
   ```python
   @pytest.mark.parametrize(
   "accounts_ids",
   [
   pytest.param("", id="empty-string"),
   pytest.param(None, id="none"),
   pytest.param([], id="empty-list"),
   ],
   )
   def test_sample(accounts_ids):
   pass
   ```
   ```console
   = test session starts 
==
   collecting ... collected 3 items
   
   test_docker.py::test_sample[] 
   test_docker.py::test_sample[None] 
   test_docker.py::test_sample[accounts_ids2] 
   
   == 3 passed in 0.12s 
===
   ```
   
![image](https://user-images.githubusercontent.com/3998685/207433529-ca64fe1c-ba52-463a-89de-9e977fab6204.png)
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] csm10495 commented on pull request #28239: Add a new config for adapting masked secrets to make it easier to prevent secret leakage in logs



csm10495 commented on PR #28239:
URL: https://github.com/apache/airflow/pull/28239#issuecomment-1349620878

   I think i have to go back to repo to put together those suggestions.. will 
do that a bit later. Check again tomorrow @ashb . Thanks! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ashb commented on a diff in pull request #28239: Add a new config for adapting masked secrets to make it easier to prevent secret leakage in logs



ashb commented on code in PR #28239:
URL: https://github.com/apache/airflow/pull/28239#discussion_r1047675719


##
airflow/utils/log/secrets_masker.py:
##
@@ -240,17 +241,53 @@ def redact(self, item: Redactable, name: str | None = 
None) -> Redacted:
 """
 return self._redact(item, name, depth=0)
 
-def add_mask(self, secret: str | dict | Iterable, name: str | None = None):
-"""Add a new secret to be masked to this filter instance."""
+@cached_property
+def _mask_adapter(self) -> None | Callable:
+"""Pulls the secret mask adapter from config.
+
+This lives in a function here to be cached and only hit the config 
once.
+"""
 from airflow.configuration import conf
 
-test_mode: bool = conf.getboolean("core", "unit_test_mode")
+return conf.getimport("logging", "secret_mask_adapter", fallback=None)
+
+@cached_property
+def _test_mode(self) -> bool:
+"""Pulls the unit test mode flag from config.
+
+This lives in a function here to be cached and only hit the config 
once.
+"""
+from airflow.configuration import conf
+
+return conf.getboolean("core", "unit_test_mode")
+
+def _add_adaptations(self, secret: str | dict | Iterable, name: str | None 
= None):
+"""Adds any adaptations to the secret that should be masked."""
+if self._mask_adapter:
+# This can return an iterable of secrets to mask OR a single 
secret as a string
+secret_or_secrets = self._mask_adapter(secret)
+
+if not isinstance(secret_or_secrets, str):
+# if its not a string, it must be an iterable
+for secret in secret_or_secrets:
+self.add_mask(secret, name, add_adaptations=False)
+else:
+self.add_mask(secret_or_secrets, name, add_adaptations=False)
+
+def add_mask(self, secret: str | dict | Iterable, name: str | None = None, 
add_adaptations: bool = True):
+"""Add a new secret to be masked to this filter instance.
+
+If add_adaptations is True, the secret mask adapter will be used to 
add adaptations for the secret
+as well.
+"""
 if isinstance(secret, dict):
 for k, v in secret.items():
 self.add_mask(v, k)
 elif isinstance(secret, str):
-if not secret or (test_mode and secret in 
SECRETS_TO_SKIP_MASKING_FOR_TESTS):
+if not secret or (self._test_mode and secret in 
SECRETS_TO_SKIP_MASKING_FOR_TESTS):
 return
+if add_adaptations:
+self._add_adaptations(secret, name)
 pattern = re.escape(secret)
 if pattern not in self.patterns and (not name or 
should_hide_value_for_key(name)):
 self.patterns.add(pattern)

Review Comment:
   This suggestion won't blindly apply, but we should remove the current 
`self.replacer = re.compile("|".join(self.patterns))` line
   
   ```suggestion
   new_mask = False
   for s in self._adaptations(secret):
   pattern = re.escape(secret)
   if pattern not in self.patterns and (not name or 
should_hide_value_for_key(name)):
   self.patterns.add(pattern)
   new_mask = True
   if new_mask:
   self.replacer = re.compile("|".join(self.patterns))  
  
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ashb commented on a diff in pull request #28239: Add a new config for adapting masked secrets to make it easier to prevent secret leakage in logs



ashb commented on code in PR #28239:
URL: https://github.com/apache/airflow/pull/28239#discussion_r1047679429


##
airflow/utils/log/secrets_masker.py:
##
@@ -240,17 +241,53 @@ def redact(self, item: Redactable, name: str | None = 
None) -> Redacted:
 """
 return self._redact(item, name, depth=0)
 
-def add_mask(self, secret: str | dict | Iterable, name: str | None = None):
-"""Add a new secret to be masked to this filter instance."""
+@cached_property
+def _mask_adapter(self) -> None | Callable:
+"""Pulls the secret mask adapter from config.
+
+This lives in a function here to be cached and only hit the config 
once.
+"""
 from airflow.configuration import conf
 
-test_mode: bool = conf.getboolean("core", "unit_test_mode")
+return conf.getimport("logging", "secret_mask_adapter", fallback=None)
+
+@cached_property
+def _test_mode(self) -> bool:
+"""Pulls the unit test mode flag from config.
+
+This lives in a function here to be cached and only hit the config 
once.
+"""
+from airflow.configuration import conf
+
+return conf.getboolean("core", "unit_test_mode")
+
+def _add_adaptations(self, secret: str | dict | Iterable, name: str | None 
= None):
+"""Adds any adaptations to the secret that should be masked."""
+if self._mask_adapter:
+# This can return an iterable of secrets to mask OR a single 
secret as a string
+secret_or_secrets = self._mask_adapter(secret)
+
+if not isinstance(secret_or_secrets, str):
+# if its not a string, it must be an iterable
+for secret in secret_or_secrets:
+self.add_mask(secret, name, add_adaptations=False)
+else:
+self.add_mask(secret_or_secrets, name, add_adaptations=False)
+
+def add_mask(self, secret: str | dict | Iterable, name: str | None = None, 
add_adaptations: bool = True):
+"""Add a new secret to be masked to this filter instance.
+
+If add_adaptations is True, the secret mask adapter will be used to 
add adaptations for the secret
+as well.
+"""
 if isinstance(secret, dict):
 for k, v in secret.items():
 self.add_mask(v, k)
 elif isinstance(secret, str):
-if not secret or (test_mode and secret in 
SECRETS_TO_SKIP_MASKING_FOR_TESTS):
+if not secret or (self._test_mode and secret in 
SECRETS_TO_SKIP_MASKING_FOR_TESTS):
 return
+if add_adaptations:
+self._add_adaptations(secret, name)

Review Comment:
   ```suggestion
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ashb commented on a diff in pull request #28239: Add a new config for adapting masked secrets to make it easier to prevent secret leakage in logs



ashb commented on code in PR #28239:
URL: https://github.com/apache/airflow/pull/28239#discussion_r1047671128


##
airflow/utils/log/secrets_masker.py:
##
@@ -240,17 +241,53 @@ def redact(self, item: Redactable, name: str | None = 
None) -> Redacted:
 """
 return self._redact(item, name, depth=0)
 
-def add_mask(self, secret: str | dict | Iterable, name: str | None = None):
-"""Add a new secret to be masked to this filter instance."""
+@cached_property
+def _mask_adapter(self) -> None | Callable:
+"""Pulls the secret mask adapter from config.
+
+This lives in a function here to be cached and only hit the config 
once.
+"""
 from airflow.configuration import conf
 
-test_mode: bool = conf.getboolean("core", "unit_test_mode")
+return conf.getimport("logging", "secret_mask_adapter", fallback=None)
+
+@cached_property
+def _test_mode(self) -> bool:
+"""Pulls the unit test mode flag from config.
+
+This lives in a function here to be cached and only hit the config 
once.
+"""
+from airflow.configuration import conf
+
+return conf.getboolean("core", "unit_test_mode")
+
+def _add_adaptations(self, secret: str | dict | Iterable, name: str | None 
= None):
+"""Adds any adaptations to the secret that should be masked."""
+if self._mask_adapter:
+# This can return an iterable of secrets to mask OR a single 
secret as a string
+secret_or_secrets = self._mask_adapter(secret)
+
+if not isinstance(secret_or_secrets, str):
+# if its not a string, it must be an iterable
+for secret in secret_or_secrets:
+self.add_mask(secret, name, add_adaptations=False)
+else:
+self.add_mask(secret_or_secrets, name, add_adaptations=False)

Review Comment:
   ```suggestion
   def _adaptations(self, secret: str) -> list[str]:
   """Adds any adaptations to the secret that should be masked."""
   yield secret
   if not self._mask_adapter:
   return
   # This can return an iterable of secrets to mask OR a single secret 
as a string
   secret_or_secrets = self._mask_adapter(secret)
   
   if not isinstance(secret_or_secrets, str):
   # if its not a string, it must be an iterable
   yield from secret_or_secrets
  else:
yield secret_or_secrets
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ashb commented on a diff in pull request #28239: Add a new config for adapting masked secrets to make it easier to prevent secret leakage in logs



ashb commented on code in PR #28239:
URL: https://github.com/apache/airflow/pull/28239#discussion_r1047671128


##
airflow/utils/log/secrets_masker.py:
##
@@ -240,17 +241,53 @@ def redact(self, item: Redactable, name: str | None = 
None) -> Redacted:
 """
 return self._redact(item, name, depth=0)
 
-def add_mask(self, secret: str | dict | Iterable, name: str | None = None):
-"""Add a new secret to be masked to this filter instance."""
+@cached_property
+def _mask_adapter(self) -> None | Callable:
+"""Pulls the secret mask adapter from config.
+
+This lives in a function here to be cached and only hit the config 
once.
+"""
 from airflow.configuration import conf
 
-test_mode: bool = conf.getboolean("core", "unit_test_mode")
+return conf.getimport("logging", "secret_mask_adapter", fallback=None)
+
+@cached_property
+def _test_mode(self) -> bool:
+"""Pulls the unit test mode flag from config.
+
+This lives in a function here to be cached and only hit the config 
once.
+"""
+from airflow.configuration import conf
+
+return conf.getboolean("core", "unit_test_mode")
+
+def _add_adaptations(self, secret: str | dict | Iterable, name: str | None 
= None):
+"""Adds any adaptations to the secret that should be masked."""
+if self._mask_adapter:
+# This can return an iterable of secrets to mask OR a single 
secret as a string
+secret_or_secrets = self._mask_adapter(secret)
+
+if not isinstance(secret_or_secrets, str):
+# if its not a string, it must be an iterable
+for secret in secret_or_secrets:
+self.add_mask(secret, name, add_adaptations=False)
+else:
+self.add_mask(secret_or_secrets, name, add_adaptations=False)

Review Comment:
   ```suggestion
   def _adaptations(self, secret: str) -> list[str]:
   """Adds any adaptations to the secret that should be masked."""
   if not self._mask_adapter:
   return [secret]
   # This can return an iterable of secrets to mask OR a single secret 
as a string
   secret_or_secrets = self._mask_adapter(secret)
   
   if not isinstance(secret_or_secrets, str):
   # if its not a string, it must be an iterable
   return secret_or_secrets
  else:
return [secret]
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ashb commented on a diff in pull request #28239: Add a new config for adapting masked secrets to make it easier to prevent secret leakage in logs



ashb commented on code in PR #28239:
URL: https://github.com/apache/airflow/pull/28239#discussion_r1047671128


##
airflow/utils/log/secrets_masker.py:
##
@@ -240,17 +241,53 @@ def redact(self, item: Redactable, name: str | None = 
None) -> Redacted:
 """
 return self._redact(item, name, depth=0)
 
-def add_mask(self, secret: str | dict | Iterable, name: str | None = None):
-"""Add a new secret to be masked to this filter instance."""
+@cached_property
+def _mask_adapter(self) -> None | Callable:
+"""Pulls the secret mask adapter from config.
+
+This lives in a function here to be cached and only hit the config 
once.
+"""
 from airflow.configuration import conf
 
-test_mode: bool = conf.getboolean("core", "unit_test_mode")
+return conf.getimport("logging", "secret_mask_adapter", fallback=None)
+
+@cached_property
+def _test_mode(self) -> bool:
+"""Pulls the unit test mode flag from config.
+
+This lives in a function here to be cached and only hit the config 
once.
+"""
+from airflow.configuration import conf
+
+return conf.getboolean("core", "unit_test_mode")
+
+def _add_adaptations(self, secret: str | dict | Iterable, name: str | None 
= None):
+"""Adds any adaptations to the secret that should be masked."""
+if self._mask_adapter:
+# This can return an iterable of secrets to mask OR a single 
secret as a string
+secret_or_secrets = self._mask_adapter(secret)
+
+if not isinstance(secret_or_secrets, str):
+# if its not a string, it must be an iterable
+for secret in secret_or_secrets:
+self.add_mask(secret, name, add_adaptations=False)
+else:
+self.add_mask(secret_or_secrets, name, add_adaptations=False)

Review Comment:
   ```suggestion
   def _adaptations(self, secret: str):
   """Adds any adaptations to the secret that should be masked."""
   if not self._mask_adapter:
   return [secret]
   # This can return an iterable of secrets to mask OR a single secret 
as a string
   secret_or_secrets = self._mask_adapter(secret)
   
   if not isinstance(secret_or_secrets, str):
   # if its not a string, it must be an iterable
   return secret_or_secrets
  else:
return [secret]
   ```



##
airflow/utils/log/secrets_masker.py:
##
@@ -240,17 +241,53 @@ def redact(self, item: Redactable, name: str | None = 
None) -> Redacted:
 """
 return self._redact(item, name, depth=0)
 
-def add_mask(self, secret: str | dict | Iterable, name: str | None = None):
-"""Add a new secret to be masked to this filter instance."""
+@cached_property
+def _mask_adapter(self) -> None | Callable:
+"""Pulls the secret mask adapter from config.
+
+This lives in a function here to be cached and only hit the config 
once.
+"""
 from airflow.configuration import conf
 
-test_mode: bool = conf.getboolean("core", "unit_test_mode")
+return conf.getimport("logging", "secret_mask_adapter", fallback=None)
+
+@cached_property
+def _test_mode(self) -> bool:
+"""Pulls the unit test mode flag from config.
+
+This lives in a function here to be cached and only hit the config 
once.
+"""
+from airflow.configuration import conf
+
+return conf.getboolean("core", "unit_test_mode")
+
+def _add_adaptations(self, secret: str | dict | Iterable, name: str | None 
= None):
+"""Adds any adaptations to the secret that should be masked."""
+if self._mask_adapter:
+# This can return an iterable of secrets to mask OR a single 
secret as a string
+secret_or_secrets = self._mask_adapter(secret)
+
+if not isinstance(secret_or_secrets, str):
+# if its not a string, it must be an iterable
+for secret in secret_or_secrets:
+self.add_mask(secret, name, add_adaptations=False)
+else:
+self.add_mask(secret_or_secrets, name, add_adaptations=False)
+
+def add_mask(self, secret: str | dict | Iterable, name: str | None = None, 
add_adaptations: bool = True):
+"""Add a new secret to be masked to this filter instance.
+
+If add_adaptations is True, the secret mask adapter will be used to 
add adaptations for the secret
+as well.
+"""
 if isinstance(secret, dict):
 for k, v in secret.items():
 self.add_mask(v, k)
 elif isinstance(secret, str):
-if not secret or (test_mode and secret in 
SECRETS_TO_SKIP_MASKING_FOR_TESTS):
+if not secret or (self._test_mode and secret in 
SECRETS_TO_SKIP_MASKING_FOR_TESTS):

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #28024: Add AWS SageMaker operator to register a model's version



vandonr-amz commented on code in PR #28024:
URL: https://github.com/apache/airflow/pull/28024#discussion_r1047675077


##
airflow/providers/amazon/aws/operators/sagemaker.py:
##
@@ -750,3 +751,99 @@ def execute(self, context: Context) -> Any:
 sagemaker_hook = SageMakerHook(aws_conn_id=self.aws_conn_id)
 sagemaker_hook.delete_model(model_name=self.config["ModelName"])
 self.log.info("Model %s deleted successfully.", 
self.config["ModelName"])
+
+
+class ApprovalStatus(Enum):
+"""Approval statuses for a Sagemaker Model Package."""
+
+APPROVED = "Approved"
+REJECTED = "Rejected"
+PENDING_MANUAL_APPROVAL = "PendingManualApproval"
+
+
+class SageMakerRegisterModelVersionOperator(SageMakerBaseOperator):
+"""
+Registers an Amazon SageMaker model by creating a model version that 
specifies the model group to which it
+belongs. Will create the model group if it does not exist already.
+
+.. seealso::
+For more information on how to use this operator, take a look at the 
guide:
+:ref:`howto/operator:SageMakerRegisterModelVersionOperator`
+
+:param image_uri: The Amazon EC2 Container Registry (Amazon ECR) path 
where inference code is stored.
+:param model_url: The Amazon S3 path where the model artifacts (the 
trained weights of the model), which
+result from model training, are stored. This path must point to a 
single gzip compressed tar archive
+(.tar.gz suffix).
+:param package_group_name: The name of the model package group that the 
model is going to be registered
+to. Will be created if it doesn't already exist.
+:param package_group_desc: Description of the model package group, if it 
was to be created (optional).
+:param package_desc: Description of the model package (optional).
+:param model_approval: Approval status of the model package. Defaults to 
PendingManualApproval
+:param aws_conn_id: The AWS connection ID to use.
+:param config: Can contain extra parameters for the boto call to 
create_model_package, and/or overrides
+for any parameter defined above. For a complete list of available 
parameters, see
+
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_model_package

Review Comment:
   come to think of it, maybe I should split that, and have a config for the 
base operator on one side, and the extra params for mine in an other dict



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk closed issue #28342: Dataset - updated_at information not working as expected (or documentation is not clear, what information should it convey...)



potiuk closed issue #28342: Dataset - updated_at information not working as 
expected (or documentation is not clear, what information should it convey...) 
URL: https://github.com/apache/airflow/issues/28342


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #28342: Dataset - updated_at information not working as expected (or documentation is not clear, what information should it convey...)



potiuk commented on issue #28342:
URL: https://github.com/apache/airflow/issues/28342#issuecomment-1349610391

   Updated_at same as created_at are DB modification auto-fields. Same for all 
the otehr models they indicate record update in DB not Dataset update (and can 
be used for example to delete old data).
   
   What you missed is DatasetEvent - there all the history of all the events 
generated for a given dataset are stored.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] csm10495 commented on a diff in pull request #28239: Add a new config for adapting masked secrets to make it easier to prevent secret leakage in logs



csm10495 commented on code in PR #28239:
URL: https://github.com/apache/airflow/pull/28239#discussion_r1047661033


##
airflow/utils/log/secrets_masker.py:
##
@@ -240,17 +241,53 @@ def redact(self, item: Redactable, name: str | None = 
None) -> Redacted:
 """
 return self._redact(item, name, depth=0)
 
-def add_mask(self, secret: str | dict | Iterable, name: str | None = None):
-"""Add a new secret to be masked to this filter instance."""
+@cached_property
+def _mask_adapter(self) -> None | Callable:
+"""Pulls the secret mask adapter from config.
+
+This lives in a function here to be cached and only hit the config 
once.
+"""
 from airflow.configuration import conf
 
-test_mode: bool = conf.getboolean("core", "unit_test_mode")
+return conf.getimport("logging", "secret_mask_adapter", fallback=None)
+
+@cached_property
+def _test_mode(self) -> bool:
+"""Pulls the unit test mode flag from config.
+
+This lives in a function here to be cached and only hit the config 
once.
+"""
+from airflow.configuration import conf
+
+return conf.getboolean("core", "unit_test_mode")
+
+def _add_adaptations(self, secret: str | dict | Iterable, name: str | None 
= None):
+"""Adds any adaptations to the secret that should be masked."""
+if self._mask_adapter:
+# This can return an iterable of secrets to mask OR a single 
secret as a string
+secret_or_secrets = self._mask_adapter(secret)
+
+if not isinstance(secret_or_secrets, str):
+# if its not a string, it must be an iterable
+for secret in secret_or_secrets:
+self.add_mask(secret, name, add_adaptations=False)
+else:
+self.add_mask(secret_or_secrets, name, add_adaptations=False)
+
+def add_mask(self, secret: str | dict | Iterable, name: str | None = None, 
add_adaptations: bool = True):

Review Comment:
   It breaks the recursion problem. add_adaptations gets set to False when 
add_adaptations calls add_mask



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ashb commented on a diff in pull request #28239: Add a new config for adapting masked secrets to make it easier to prevent secret leakage in logs



ashb commented on code in PR #28239:
URL: https://github.com/apache/airflow/pull/28239#discussion_r1047660905


##
airflow/utils/log/secrets_masker.py:
##
@@ -240,17 +241,53 @@ def redact(self, item: Redactable, name: str | None = 
None) -> Redacted:
 """
 return self._redact(item, name, depth=0)
 
-def add_mask(self, secret: str | dict | Iterable, name: str | None = None):
-"""Add a new secret to be masked to this filter instance."""
+@cached_property
+def _mask_adapter(self) -> None | Callable:
+"""Pulls the secret mask adapter from config.
+
+This lives in a function here to be cached and only hit the config 
once.
+"""
 from airflow.configuration import conf
 
-test_mode: bool = conf.getboolean("core", "unit_test_mode")
+return conf.getimport("logging", "secret_mask_adapter", fallback=None)
+
+@cached_property
+def _test_mode(self) -> bool:
+"""Pulls the unit test mode flag from config.
+
+This lives in a function here to be cached and only hit the config 
once.
+"""
+from airflow.configuration import conf
+
+return conf.getboolean("core", "unit_test_mode")
+
+def _add_adaptations(self, secret: str | dict | Iterable, name: str | None 
= None):
+"""Adds any adaptations to the secret that should be masked."""
+if self._mask_adapter:
+# This can return an iterable of secrets to mask OR a single 
secret as a string
+secret_or_secrets = self._mask_adapter(secret)
+
+if not isinstance(secret_or_secrets, str):
+# if its not a string, it must be an iterable
+for secret in secret_or_secrets:
+self.add_mask(secret, name, add_adaptations=False)
+else:
+self.add_mask(secret_or_secrets, name, add_adaptations=False)
+
+def add_mask(self, secret: str | dict | Iterable, name: str | None = None, 
add_adaptations: bool = True):

Review Comment:
   Oh I see.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ashb commented on a diff in pull request #28239: Add a new config for adapting masked secrets to make it easier to prevent secret leakage in logs



ashb commented on code in PR #28239:
URL: https://github.com/apache/airflow/pull/28239#discussion_r1047659711


##
airflow/utils/log/secrets_masker.py:
##
@@ -240,17 +241,53 @@ def redact(self, item: Redactable, name: str | None = 
None) -> Redacted:
 """
 return self._redact(item, name, depth=0)
 
-def add_mask(self, secret: str | dict | Iterable, name: str | None = None):
-"""Add a new secret to be masked to this filter instance."""
+@cached_property
+def _mask_adapter(self) -> None | Callable:
+"""Pulls the secret mask adapter from config.
+
+This lives in a function here to be cached and only hit the config 
once.
+"""
 from airflow.configuration import conf
 
-test_mode: bool = conf.getboolean("core", "unit_test_mode")
+return conf.getimport("logging", "secret_mask_adapter", fallback=None)
+
+@cached_property
+def _test_mode(self) -> bool:
+"""Pulls the unit test mode flag from config.
+
+This lives in a function here to be cached and only hit the config 
once.
+"""
+from airflow.configuration import conf
+
+return conf.getboolean("core", "unit_test_mode")
+
+def _add_adaptations(self, secret: str | dict | Iterable, name: str | None 
= None):
+"""Adds any adaptations to the secret that should be masked."""
+if self._mask_adapter:
+# This can return an iterable of secrets to mask OR a single 
secret as a string
+secret_or_secrets = self._mask_adapter(secret)
+
+if not isinstance(secret_or_secrets, str):
+# if its not a string, it must be an iterable
+for secret in secret_or_secrets:
+self.add_mask(secret, name, add_adaptations=False)
+else:
+self.add_mask(secret_or_secrets, name, add_adaptations=False)
+
+def add_mask(self, secret: str | dict | Iterable, name: str | None = None, 
add_adaptations: bool = True):

Review Comment:
   Do we need the `add_adaptations` param? When (and how) would that ever be 
set to false?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ashb commented on a diff in pull request #28239: Add a new config for adapting masked secrets to make it easier to prevent secret leakage in logs



ashb commented on code in PR #28239:
URL: https://github.com/apache/airflow/pull/28239#discussion_r1047655862


##
airflow/utils/log/secrets_masker.py:
##
@@ -240,17 +241,58 @@ def redact(self, item: Redactable, name: str | None = 
None) -> Redacted:
 """
 return self._redact(item, name, depth=0)
 
-def add_mask(self, secret: str | dict | Iterable, name: str | None = None):
-"""Add a new secret to be masked to this filter instance."""
+@cached_property
+def _mask_adapter(self) -> None | Callable:
+"""Pulls the secret mask adapter from config.
+
+This lives in a function here to be cached and only hit the config 
once.
+"""
 from airflow.configuration import conf
 
-test_mode: bool = conf.getboolean("core", "unit_test_mode")
+mask_adapter: Callable | None = None
+try:
+mask_adapter = conf.getimport("logging", "secret_mask_adapter")
+except AirflowConfigException:
+pass
+return mask_adapter

Review Comment:
   I think this would work as
   
   ```suggestion
   return conf.getimport("logging", "secret_mask_adapter", 
fallback=None)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mrichman commented on pull request #28338: New AWS sensor — DynamoDBValueSensor



mrichman commented on PR #28338:
URL: https://github.com/apache/airflow/pull/28338#issuecomment-1349515026

   @ferruzzi Yes please review. @o-nikolas suggested I create as a draft to 
limit visibility. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ferruzzi commented on a diff in pull request #28338: New AWS sensor — DynamoDBValueSensor



ferruzzi commented on code in PR #28338:
URL: https://github.com/apache/airflow/pull/28338#discussion_r1047588453


##
airflow/providers/amazon/aws/example_dags/example_dynamodb_sensor.py:
##
@@ -0,0 +1,44 @@
+# Licensed to the Apache Software Foundation (ASF) under one

Review Comment:
   Example DAGs are being dropped and have all been moved into system tests 
[here](https://github.com/apache/airflow/tree/main/tests/system/providers/amazon/aws).
  Please either update the DynamoDB system test with this new sensor if 
possible or create a new system test there instead of this Example DAG.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ferruzzi commented on pull request #28338: New AWS sensor — DynamoDBValueSensor



ferruzzi commented on PR #28338:
URL: https://github.com/apache/airflow/pull/28338#issuecomment-1349500672

   This is marked as a draft, do you want a review on it or are you still 
working on it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ferruzzi commented on a diff in pull request #28279: Add Amazon Elastic Container Registry (ECR) Hook



ferruzzi commented on code in PR #28279:
URL: https://github.com/apache/airflow/pull/28279#discussion_r1047577892


##
tests/providers/amazon/aws/hooks/test_ecr.py:
##
@@ -0,0 +1,103 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from unittest import mock
+
+import boto3
+import pytest
+from moto import mock_ecr
+from moto.core import DEFAULT_ACCOUNT_ID
+
+from airflow.providers.amazon.aws.hooks.ecr import EcrHook
+
+
+@pytest.fixture
+def patch_hook(monkeypatch):
+"""Patch hook object by dummy boto3 ECR client."""
+ecr_client = boto3.client("ecr")
+monkeypatch.setattr(EcrHook, "conn", ecr_client)
+yield
+
+
+@mock_ecr
+class TestEcrHook:
+def test_service_type(self):
+"""Test expected boto3 client type."""
+assert EcrHook().client_type == "ecr"
+
+@pytest.mark.parametrize(
+"accounts_ids",
+[
+pytest.param("", id="empty-string"),
+pytest.param(None, id="none"),
+pytest.param([], id="empty-list"),
+],
+)
+def test_get_temporary_credentials_default_account_id(self, patch_hook, 
accounts_ids):
+"""Test different types of empty account/registry ids."""
+result = EcrHook().get_temporary_credentials(registry_ids=accounts_ids)
+assert len(result) == 1
+assert result[0].username == "AWS"
+assert result[0].registry.startswith(DEFAULT_ACCOUNT_ID)
+assert result[0].password == f"{DEFAULT_ACCOUNT_ID}-auth-token"
+
+@pytest.mark.parametrize(
+"accounts_id, expected_registry",
+[
+pytest.param(DEFAULT_ACCOUNT_ID, DEFAULT_ACCOUNT_ID, 
id="moto-default-account"),
+pytest.param("", "", 
id="custom-account-id"),
+pytest.param([""], "", id="empty-list"),

Review Comment:
   If you do keep the id field, then this one is wrong.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ferruzzi commented on a diff in pull request #28279: Add Amazon Elastic Container Registry (ECR) Hook



ferruzzi commented on code in PR #28279:
URL: https://github.com/apache/airflow/pull/28279#discussion_r1047577311


##
tests/providers/amazon/aws/hooks/test_ecr.py:
##
@@ -0,0 +1,103 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from unittest import mock
+
+import boto3
+import pytest
+from moto import mock_ecr
+from moto.core import DEFAULT_ACCOUNT_ID
+
+from airflow.providers.amazon.aws.hooks.ecr import EcrHook
+
+
+@pytest.fixture
+def patch_hook(monkeypatch):
+"""Patch hook object by dummy boto3 ECR client."""
+ecr_client = boto3.client("ecr")
+monkeypatch.setattr(EcrHook, "conn", ecr_client)
+yield
+
+
+@mock_ecr
+class TestEcrHook:
+def test_service_type(self):
+"""Test expected boto3 client type."""
+assert EcrHook().client_type == "ecr"
+
+@pytest.mark.parametrize(
+"accounts_ids",
+[
+pytest.param("", id="empty-string"),
+pytest.param(None, id="none"),
+pytest.param([], id="empty-list"),
+],
+)
+def test_get_temporary_credentials_default_account_id(self, patch_hook, 
accounts_ids):
+"""Test different types of empty account/registry ids."""
+result = EcrHook().get_temporary_credentials(registry_ids=accounts_ids)
+assert len(result) == 1
+assert result[0].username == "AWS"
+assert result[0].registry.startswith(DEFAULT_ACCOUNT_ID)
+assert result[0].password == f"{DEFAULT_ACCOUNT_ID}-auth-token"
+
+@pytest.mark.parametrize(
+"accounts_id, expected_registry",
+[
+pytest.param(DEFAULT_ACCOUNT_ID, DEFAULT_ACCOUNT_ID, 
id="moto-default-account"),
+pytest.param("", "", 
id="custom-account-id"),
+pytest.param([""], "", id="empty-list"),
+],
+)
+def test_get_temporary_credentials_single_account_id(self, patch_hook, 
accounts_id, expected_registry):

Review Comment:
   I'd also consider combining this test and the one above, but that's up to 
you.
   
   ```
   @pytest.mark.parametrize(
   "accounts_id, expected_registry",
   [
   ("", DEFAULT_ACCOUNT_ID),
   (None, DEFAULT_ACCOUNT_ID),
   ([], DEFAULT_ACCOUNT_ID),
   (DEFAULT_ACCOUNT_ID, DEFAULT_ACCOUNT_ID),
   ("", ""),
   ([""], "")
   ]
   )
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ferruzzi commented on a diff in pull request #28279: Add Amazon Elastic Container Registry (ECR) Hook



ferruzzi commented on code in PR #28279:
URL: https://github.com/apache/airflow/pull/28279#discussion_r1047572324


##
tests/providers/amazon/aws/hooks/test_ecr.py:
##
@@ -0,0 +1,103 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from unittest import mock
+
+import boto3
+import pytest
+from moto import mock_ecr
+from moto.core import DEFAULT_ACCOUNT_ID
+
+from airflow.providers.amazon.aws.hooks.ecr import EcrHook
+
+
+@pytest.fixture
+def patch_hook(monkeypatch):
+"""Patch hook object by dummy boto3 ECR client."""
+ecr_client = boto3.client("ecr")
+monkeypatch.setattr(EcrHook, "conn", ecr_client)
+yield
+
+
+@mock_ecr
+class TestEcrHook:
+def test_service_type(self):
+"""Test expected boto3 client type."""
+assert EcrHook().client_type == "ecr"
+
+@pytest.mark.parametrize(
+"accounts_ids",
+[
+pytest.param("", id="empty-string"),
+pytest.param(None, id="none"),
+pytest.param([], id="empty-list"),
+],
+)

Review Comment:
   This might be my own ignorance, but why bother with the pytest.param and 
setting an id?  I've always done this as 
   
   `@pytest.mark.parametrize("accounts_ids", ["", None, []])`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #28321: Add support in AWS Batch Operator for multinode jobs



vandonr-amz commented on code in PR #28321:
URL: https://github.com/apache/airflow/pull/28321#discussion_r1047552663


##
airflow/providers/amazon/aws/operators/batch.py:
##
@@ -151,8 +155,9 @@ def __init__(
 self.job_name = job_name
 self.job_definition = job_definition
 self.job_queue = job_queue
-self.overrides = overrides or {}
-self.array_properties = array_properties or {}
+self.container_overrides = overrides
+self.node_overrides = node_overrides

Review Comment:
   yes they are, specifying both returns an input validation error.
   BTW @Taragolis , could you approve running the workflow to help get this PR 
ready-to-review ?  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ChhaSahil commented on issue #28289: Add AWS Neptune hook and operators



ChhaSahil commented on issue #28289:
URL: https://github.com/apache/airflow/issues/28289#issuecomment-1349349932

   Hello @eladkal 
   I would also love to take this one!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil commented on a diff in pull request #28246: Better table name validation for db clean



kaxil commented on code in PR #28246:
URL: https://github.com/apache/airflow/pull/28246#discussion_r1047526850


##
airflow/utils/db_cleanup.py:
##
@@ -337,8 +337,14 @@ def run_cleanup(
 :param session: Session representing connection to the metadata database.
 """
 clean_before_timestamp = timezone.coerce_datetime(clean_before_timestamp)
-effective_table_names = table_names if table_names else 
list(config_dict.keys())
-effective_config_dict = {k: v for k, v in config_dict.items() if k in 
effective_table_names}
+desired_table_names = set(table_names if table_names else 
config_dict.keys())
+effective_config_dict = {k: v for k, v in config_dict.items() if k in 
desired_table_names}
+effective_table_names = set(effective_config_dict.keys())
+if desired_table_names != effective_table_names:
+outliers = desired_table_names - effective_table_names
+logger.warning("The following table(s) are not valid choices and will 
be skipped: %s", outliers)

Review Comment:
   We can probably add a unit test case for this



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] rodrigocollavo commented on issue #19251: Add a way to skip the secret_backend



rodrigocollavo commented on issue #19251:
URL: https://github.com/apache/airflow/issues/19251#issuecomment-1349299012

   Thanks for the full details.
   In my case the problem was more related to the amount of error logs due to 
the variable or connection not found in the secret manager, but I wanted to 
ensure Airflow always check for values in the secret manager first and then 
fallback to the airflow DB, without enforcing any name convention.
   So, in case someone else find it useful, my solution is just to "mute" the 
error log but it will still check if the value exists in the secret manager.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Taragolis commented on a diff in pull request #28279: Add Amazon Elastic Container Registry (ECR) Hook



Taragolis commented on code in PR #28279:
URL: https://github.com/apache/airflow/pull/28279#discussion_r1047525571


##
airflow/providers/amazon/aws/hooks/ecr.py:
##
@@ -0,0 +1,101 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import base64
+import logging
+from dataclasses import dataclass
+from datetime import datetime
+
+from airflow.providers.amazon.aws.hooks.base_aws import AwsBaseHook
+from airflow.utils.log.secrets_masker import mask_secret
+
+logger = logging.getLogger(__name__)
+
+
+@dataclass(frozen=True)
+class EcrCredentials:
+"""Helper (frozen dataclass) for storing temporary ECR credentials."""
+
+username: str
+password: str
+proxy_endpoint: str
+expires_at: datetime
+
+def __post_init__(self):
+mask_secret(self.password)
+logger.debug("Credentials to Amazon ECR %r expires at %s.", 
self.proxy_endpoint, self.expires_at)
+
+@property
+def registry(self) -> str:
+"""Return registry in appropriate `docker login` format."""
+# 
https://github.com/docker/docker-py/issues/2256#issuecomment-824940506
+return self.proxy_endpoint.replace("https://;, "")
+
+
+class EcrHook(AwsBaseHook):
+"""
+Interact with Amazon Elastic Container Registry (ECR)
+
+Additional arguments (such as ``aws_conn_id``) may be specified and
+are passed down to the underlying AwsBaseHook.
+
+.. seealso::
+:class:`~airflow.providers.amazon.aws.hooks.base_aws.AwsBaseHook`
+"""
+
+def __init__(self, **kwargs):
+kwargs.pop("client_type", None)
+super().__init__(client_type="ecr", **kwargs)
+
+def get_temporary_credentials(self, registry_ids: list[str] | str | None = 
None) -> list[EcrCredentials]:
+"""Get temporary credentials for Amazon ECR.
+
+Return list of 
:class:`~airflow.providers.amazon.aws.hooks.ecr.EcrCredentials`,
+obtained credentials valid for 12 hours.
+
+:param registry_ids: Either AWS Account ID or list of AWS Account IDs 
that are associated
+with the registries for which obtain credentials. If you do not 
specify a registry,
+the default registry is assumed.
+
+.. seealso::
+- `boto3 ECR client get_authorization_token method 
`_.
+"""
+registry_ids = registry_ids or None

Review Comment:
   So value in some cases might be `""` just for avoid this situation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Taragolis commented on a diff in pull request #28279: Add Amazon Elastic Container Registry (ECR) Hook