[GitHub] [airflow] bhirsz commented on pull request #26858: Migrate Dataproc Metastore system tests according to AIP-47
bhirsz commented on PR #26858: URL: https://github.com/apache/airflow/pull/26858#issuecomment-1293041952 @potiuk all green, migrated tests and minor change to typing for one of the arguments in operator -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] bhirsz commented on pull request #26911: Migrate Bigtable operators system tests according to AIP-47
bhirsz commented on PR #26911: URL: https://github.com/apache/airflow/pull/26911#issuecomment-1293041383 @potiuk all green, only tests migrated -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] Thrameos commented on pull request #27264: Attempt to add Python 3.11 support
Thrameos commented on PR #27264: URL: https://github.com/apache/airflow/pull/27264#issuecomment-1292961292 JPype for 3.11 should be out on PyPi shortly. I need to have all the azure images up to date before I can pull the release script. So that portion of your issue should be resolved. jpype-project/jpype#1099 If you need to test prior to the drop on PyPi, you can pull the wheels from https://github.com/jpype-project/jpype/releases/tag/v1.4.1 If there is a problem with the JPype release, please post an issue so that I can resolve them this weekend. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (9ab1a6a3e7 -> 8f99c793ec)
This is an automated email from the ASF dual-hosted git repository. uranusjr pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 9ab1a6a3e7 Update old style typing (#26872) add 8f99c793ec Fix IntegrityError during webserver startup (#27297) No new revisions were added by this update. Summary of changes: airflow/cli/commands/webserver_command.py| 6 ++ tests/cli/commands/test_webserver_command.py | 1 + 2 files changed, 7 insertions(+)
[GitHub] [airflow] uranusjr merged pull request #27297: Fix IntegrityError during webserver startup
uranusjr merged PR #27297: URL: https://github.com/apache/airflow/pull/27297 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] uranusjr closed issue #23512: Random "duplicate key value violates unique constraint" errors when initializing the postgres database
uranusjr closed issue #23512: Random "duplicate key value violates unique constraint" errors when initializing the postgres database URL: https://github.com/apache/airflow/issues/23512 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #27307: Task exited with return code Negsignal.SIGSEGV
boring-cyborg[bot] commented on issue #27307: URL: https://github.com/apache/airflow/issues/27307#issuecomment-1292950377 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] samsudinde opened a new issue, #27307: Task exited with return code Negsignal.SIGSEGV
samsudinde opened a new issue, #27307:
URL: https://github.com/apache/airflow/issues/27307
### Apache Airflow version
2.4.2
### What happened
I tried to load data .csv into bigquery. but I got this errors.
[2022-10-27T11:09:58.907+0700] {taskinstance.py:1165} INFO - Dependencies
all met for
[2022-10-27T11:09:58.910+0700] {taskinstance.py:1165} INFO - Dependencies
all met for
[2022-10-27T11:09:58.911+0700] {taskinstance.py:1362} INFO -
[2022-10-27T11:09:58.911+0700] {taskinstance.py:1363} INFO - Starting
attempt 2 of 7
[2022-10-27T11:09:58.911+0700] {taskinstance.py:1364} INFO -
[2022-10-27T11:09:58.915+0700] {taskinstance.py:1383} INFO - Executing
on 2022-10-27
00:00:00+00:00
[2022-10-27T11:09:58.919+0700] {standard_task_runner.py:55} INFO - Started
process 28260 to run task
[2022-10-27T11:09:58.925+0700] {standard_task_runner.py:82} INFO - Running:
['airflow', 'tasks', 'run', 'replication_invoice_item_new',
'upload_test_and_debug_area_invoice_item',
'scheduled__2022-10-27T00:00:00+00:00', '--job-id', '64', '--raw', '--subdir',
'DAGS_FOLDER/db_replication/invoice_item.py', '--cfg-path',
'/var/folders/fk/vjp80qzx7ms45v1jv4vz5698gn/T/tmpiwnk5v73']
[2022-10-27T11:09:58.927+0700] {standard_task_runner.py:83} INFO - Job 64:
Subtask upload_test_and_debug_area_invoice_item
[2022-10-27T11:09:58.955+0700] {task_command.py:376} INFO - Running
on host
Samsudins-MacBook-Pro.local
[2022-10-27T11:09:58.974+0700] {taskinstance.py:1590} INFO - Exporting the
following env vars:
AIRFLOW_CTX_DAG_ID=replication_invoice_item_new
AIRFLOW_CTX_TASK_ID=upload_test_and_debug_area_invoice_item
AIRFLOW_CTX_EXECUTION_DATE=2022-10-27T00:00:00+00:00
AIRFLOW_CTX_TRY_NUMBER=2
AIRFLOW_CTX_DAG_RUN_ID=scheduled__2022-10-27T00:00:00+00:00
[2022-10-27T11:09:59.201+0700] {local_task_job.py:164} INFO - Task exited
with return code Negsignal.SIGSEGV
[2022-10-27T11:09:59.210+0700] {local_task_job.py:273} INFO - 0 downstream
tasks scheduled from follow-on schedule check
### What you think should happen instead
I was migrate my own laptop from os linux to mac m1. the code production
should be fine on my previouse laptop, but I tried to test on my new laptop its
doesnt work well to process load data.
do have any idea about this?
### How to reproduce
_No response_
### Operating System
mac m1
### Versions of Apache Airflow Providers
_No response_
### Deployment
Other
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #27306: Helm Chart fails on EKS
boring-cyborg[bot] commented on issue #27306: URL: https://github.com/apache/airflow/issues/27306#issuecomment-1292942581 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] KishinNext opened a new issue, #27306: Helm Chart fails on EKS
KishinNext opened a new issue, #27306: URL: https://github.com/apache/airflow/issues/27306 ### Official Helm Chart version 1.7.0 (latest released) ### Apache Airflow version 2.4.1 ### Kubernetes Version v4.5.7 ### Helm Chart configuration The last helm char for Airflow ### Docker Image customisations None ### What happened I'm using the last helm chart of Airflow, and I used this configuration for the cluster on EKS... but I get this error ### Error ``` nstall.go:192: [debug] Original chart version: "" install.go:209: [debug] CHART PATH: /home/ec2-user/.cache/helm/repository/airflow-1.7.0.tgz client.go:310: [debug] Starting delete for "airflow-broker-url" Secret client.go:339: [debug] secrets "airflow-broker-url" not found client.go:128: [debug] creating 1 resource(s) client.go:310: [debug] Starting delete for "airflow-fernet-key" Secret client.go:339: [debug] secrets "airflow-fernet-key" not found client.go:128: [debug] creating 1 resource(s) client.go:310: [debug] Starting delete for "airflow-redis-password" Secret client.go:339: [debug] secrets "airflow-redis-password" not found client.go:128: [debug] creating 1 resource(s) client.go:128: [debug] creating 30 resource(s) client.go:310: [debug] Starting delete for "airflow-run-airflow-migrations" Job client.go:339: [debug] jobs.batch "airflow-run-airflow-migrations" not found client.go:128: [debug] creating 1 resource(s) client.go:540: [debug] Watching for changes to Job airflow-run-airflow-migrations with timeout of 10m0s client.go:568: [debug] Add/Modify event for airflow-run-airflow-migrations: ADDED client.go:607: [debug] airflow-run-airflow-migrations: Jobs active: 0, jobs failed: 0, jobs succeeded: 0 client.go:568: [debug] Add/Modify event for airflow-run-airflow-migrations: MODIFIED client.go:607: [debug] airflow-run-airflow-migrations: Jobs active: 1, jobs failed: 0, jobs succeeded: 0 Error: INSTALLATION FAILED: failed post-install: timed out waiting for the condition helm.go:84: [debug] failed post-install: timed out waiting for the condition INSTALLATION FAILED main.newInstallCmd.func2 helm.sh/helm/v3/cmd/helm/install.go:141 github.com/spf13/cobra.(*Command).execute github.com/spf13/cobra@v1.5.0/command.go:872 github.com/spf13/cobra.(*Command).ExecuteC github.com/spf13/cobra@v1.5.0/command.go:990 github.com/spf13/cobra.(*Command).Execute github.com/spf13/cobra@v1.5.0/command.go:918 main.main helm.sh/helm/v3/cmd/helm/helm.go:83 runtime.main runtime/proc.go:250 runtime.goexit runtime/asm_amd64.s:1571 ``` ### PostgreSQL describe pod ``` Name: airflow-postgresql-0 Namespace:airflow Priority: 0 Service Account: default Node: Labels: app.kubernetes.io/component=primary app.kubernetes.io/instance=airflow app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=postgresql controller-revision-hash=airflow-postgresql-d9b49657b helm.sh/chart=postgresql-10.5.3 role=primary statefulset.kubernetes.io/pod-name=airflow-postgresql-0 Annotations: kubernetes.io/psp: eks.privileged Status: Pending IP: IPs: Controlled By:StatefulSet/airflow-postgresql Containers: airflow-postgresql: Image: docker.io/bitnami/postgresql:11.12.0-debian-10-r44 Port: 5432/TCP Host Port: 0/TCP Requests: cpu: 250m memory: 256Mi Liveness: exec [/bin/sh -c exec pg_isready -U "postgres" -h 127.0.0.1 -p 5432] delay=30s timeout=5s period=10s #success=1 #failure=6 Readiness: exec [/bin/sh -c -e exec pg_isready -U "postgres" -h 127.0.0.1 -p 5432 [ -f /opt/bitnami/postgresql/tmp/.initialized ] || [ -f /bitnami/postgresql/.initialized ] ] delay=5s timeout=5s period=10s #success=1 #failure=6 Environment: BITNAMI_DEBUG:false POSTGRESQL_PORT_NUMBER: 5432 POSTGRESQL_VOLUME_DIR:/bitnami/postgresql PGDATA: /bitnami/postgresql/data POSTGRES_USER:postgres POSTGRES_PASSWORD: Optional: false POSTGRESQL_ENABLE_LDAP: no POSTGRESQL_ENABLE_TLS:no POSTGRESQL_LOG_HOSTNAME: false POSTGRESQL_LOG_CONNECTIONS: false POSTGRESQL_LOG_DISCONNECTIONS:false POSTGRESQL_PGAUDIT_LOG_CATALOG: off POSTGRESQL_CLIENT_MIN_MESSAGES: error POSTGRESQL_SHARE
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #27305: Add readOnly flag to gitSync section of Helm Chart
boring-cyborg[bot] commented on PR #27305: URL: https://github.com/apache/airflow/pull/27305#issuecomment-1292931946 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst) Here are some useful points: - Pay attention to the quality of your code (flake8, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that. - In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it. - Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations. - Be patient and persistent. It might take some time to get a review or get the final approval from Committers. - Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack. - Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices). Apache Airflow is a community-driven project and together we are making it better 🚀. In case of doubts contact the developers at: Mailing List: d...@airflow.apache.org Slack: https://s.apache.org/airflow-slack -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ostapkonst opened a new pull request, #27305: Add readOnly flag to gitSync section of Helm Chart
ostapkonst opened a new pull request, #27305:
URL: https://github.com/apache/airflow/pull/27305
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] KishinNext commented on issue #16176: Quickstart Helm Chart fails post-install
KishinNext commented on issue #16176:
URL: https://github.com/apache/airflow/issues/16176#issuecomment-1292930998
Any configuration to get the correct cluster to run Airflow? I'm using the
last helm chart of Airflow, and I used this configuration for the cluster...
but I get the same error :(
```
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: airflow
region: us-east-1
version: "1.23"
managedNodeGroups:
- name: workers
instanceType: t3.medium
privateNetworking: true
minSize: 1
maxSize: 3
desiredCapacity: 3
volumeSize: 20
ssh:
allow: true
publicKeyName: airflow-workstation
labels: { role: worker }
tags:
nodegroup-role: worker
iam:
withAddonPolicies:
ebs: true
imageBuilder: true
efs: true
albIngress: true
autoScaler: true
cloudWatch: true
externalDNS: true
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[airflow] branch main updated (b757bfa44c -> 9ab1a6a3e7)
This is an automated email from the ASF dual-hosted git repository. uranusjr pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from b757bfa44c Ignore azure-datalake-store deprecated mod use (#27287) add 9ab1a6a3e7 Update old style typing (#26872) No new revisions were added by this update. Summary of changes: airflow/api/client/__init__.py | 3 +- airflow/configuration.py | 3 - airflow/dag_processing/manager.py | 31 ++ airflow/dag_processing/processor.py| 19 +--- airflow/decorators/__init__.pyi| 118 ++--- .../example_branch_python_dop_operator_3.py| 3 +- airflow/example_dags/subdags/subdag.py | 3 +- airflow/executors/base_executor.py | 1 - airflow/executors/celery_executor.py | 2 - airflow/executors/celery_kubernetes_executor.py| 2 - airflow/executors/local_kubernetes_executor.py | 2 - airflow/jobs/backfill_job.py | 6 +- airflow/jobs/base_job.py | 2 - airflow/jobs/scheduler_job.py | 5 - airflow/kubernetes/kubernetes_helper_functions.py | 2 +- airflow/kubernetes/pod_launcher_deprecated.py | 2 - airflow/macros/__init__.py | 1 - airflow/macros/hive.py | 12 ++- airflow/migrations/utils.py| 3 +- ...0_5_change_datetime_to_datetime2_6_on_mssql_.py | 6 +- .../0060_2_0_0_remove_id_column_from_xcom.py | 3 +- .../0088_2_2_0_improve_mssql_compatibility.py | 3 +- airflow/models/crypto.py | 3 +- airflow/models/dag.py | 7 +- airflow/models/dagbag.py | 3 +- airflow/models/dagrun.py | 4 +- airflow/models/serialized_dag.py | 4 +- airflow/models/taskinstance.py | 2 - airflow/models/xcom.py | 1 - airflow/operators/python.py| 3 +- airflow/plugins_manager.py | 4 +- airflow/providers/alibaba/cloud/hooks/oss.py | 3 - airflow/providers/amazon/aws/hooks/base_aws.py | 2 - airflow/providers/amazon/aws/hooks/batch_client.py | 12 --- .../providers/amazon/aws/hooks/batch_waiters.py| 8 +- airflow/providers/amazon/aws/hooks/datasync.py | 8 -- airflow/providers/amazon/aws/hooks/dms.py | 3 +- airflow/providers/amazon/aws/hooks/ec2.py | 3 - airflow/providers/amazon/aws/hooks/eks.py | 16 --- .../aws/hooks/elasticache_replication_group.py | 12 +-- airflow/providers/amazon/aws/hooks/glue_catalog.py | 4 - airflow/providers/amazon/aws/hooks/logs.py | 1 - airflow/providers/amazon/aws/hooks/quicksight.py | 6 +- airflow/providers/amazon/aws/hooks/redshift_sql.py | 1 - airflow/providers/amazon/aws/hooks/s3.py | 31 +- airflow/providers/amazon/aws/hooks/sagemaker.py| 2 - .../providers/amazon/aws/hooks/secrets_manager.py | 2 - airflow/providers/amazon/aws/hooks/sqs.py | 2 - .../providers/amazon/aws/hooks/step_function.py| 2 - airflow/providers/amazon/aws/operators/athena.py | 4 +- airflow/providers/amazon/aws/operators/batch.py| 4 +- airflow/providers/amazon/aws/operators/dms.py | 3 +- airflow/providers/amazon/aws/operators/sqs.py | 3 +- airflow/providers/amazon/aws/sensors/emr.py| 11 -- airflow/providers/amazon/aws/sensors/quicksight.py | 3 +- .../amazon/aws/transfers/redshift_to_s3.py | 4 +- airflow/providers/apache/beam/hooks/beam.py| 1 - airflow/providers/apache/hdfs/hooks/webhdfs.py | 2 - airflow/providers/apache/hdfs/sensors/hdfs.py | 1 - airflow/providers/apache/hive/hooks/hive.py| 6 -- .../providers/apache/hive/operators/hive_stats.py | 4 +- airflow/providers/apache/livy/hooks/livy.py| 32 ++ airflow/providers/apache/livy/operators/livy.py| 1 - airflow/providers/apache/livy/sensors/livy.py | 1 - airflow/providers/celery/sensors/celery_queue.py | 1 - airflow/providers/cloudant/hooks/cloudant.py | 1 - .../providers/cncf/kubernetes/backcompat/pod.py| 6 +- .../providers/cncf/kubernetes/utils/pod_manager.py | 3 +- airflow/providers/common/sql/hooks/sql.py | 17 ++- airflow/providers/common/sql/operators/sql.py | 1 - airflow/providers/databricks/hooks/databricks.py | 4 +- .../providers/databricks/hooks/databricks_base.py | 4 +- airflow/providers/datadog/sensors/datadog.py | 2 +- airflow/providers/docker/operators/docker.py | 1 - .../providers/elasticsearch/log/es_task_handler.py | 1 - airflow/providers/exasol/hooks/exasol.py
[GitHub] [airflow] uranusjr merged pull request #26872: Update old style typing
uranusjr merged PR #26872: URL: https://github.com/apache/airflow/pull/26872 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow-site] ashshuota commented on issue #677: Possible incomplete page published.
ashshuota commented on issue #677: URL: https://github.com/apache/airflow-site/issues/677#issuecomment-1292869879 This is the first link that appears on duckduckgo search results when searched for "apache airflow install". Fortunately, does not seem to be the case for google. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch python-3-11 updated (cf340d758d -> bdd765d1de)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a change to branch python-3-11
in repository https://gitbox.apache.org/repos/asf/airflow.git
omit cf340d758d Attempt to add Python 3.11 support
add 13b6b3b078 Added docs regarding templated field (#27301)
add 49e618054b Make grid view widths adjustable (#27273)
add e6c8c076f4 Assign public IP to ECS task network interface to allow the
task to pull container image (#27303)
add 87eb46bbc6 Common sql bugfixes and improvements (#26761)
add 734995ff26 Add deprecation warning re unset namespace in k8s hook
(#27202)
add f9b02db3e4 Remove short form of --upgrade-on-failure (#27270)
add 6b2dc6f2cc System test for SQL to S3 Transfer (AIP-47) (#27097)
add b757bfa44c Ignore azure-datalake-store deprecated mod use (#27287)
add bdd765d1de Attempt to add Python 3.11 support
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (cf340d758d)
\
N -- N -- N refs/heads/python-3-11 (bdd765d1de)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../amazon/aws/example_dags/example_sql_to_s3.py | 43 ---
.../providers/amazon/aws/transfers/sql_to_s3.py| 1 +
airflow/providers/cncf/kubernetes/CHANGELOG.rst| 7 +
.../providers/cncf/kubernetes/hooks/kubernetes.py | 39 ++-
airflow/providers/common/sql/operators/sql.py | 233 +++--
.../providers/google/cloud/operators/bigquery.py | 107 +++---
airflow/providers/ssh/operators/ssh.py | 2 +-
airflow/www/static/js/dag/Main.tsx | 85 -
airflow/www/static/js/dag/details/index.tsx| 4 +-
airflow/www/static/js/dag/grid/index.tsx | 11 +-
.../src/airflow_breeze/utils/common_options.py | 1 -
.../operators/transfer/sql_to_s3.rst | 2 +-
.../operators.rst | 31 +-
images/breeze/output-commands-hash.txt | 8 +-
images/breeze/output_ci-image_build.svg| 292
images/breeze/output_prod-image_build.svg | 376 ++---
scripts/in_container/verify_providers.py | 10 +-
setup.py | 11 +-
.../cncf/kubernetes/hooks/test_kubernetes.py | 11 +
tests/providers/common/sql/operators/test_sql.py | 225 ++--
.../providers/amazon/aws/example_ecs_fargate.py| 1 +
...dshift_s3_transfers.py => example_sql_to_s3.py} | 103 ++
.../cloud/bigquery/example_bigquery_queries.py | 2 +-
23 files changed, 925 insertions(+), 680 deletions(-)
delete mode 100644
airflow/providers/amazon/aws/example_dags/example_sql_to_s3.py
copy tests/system/providers/amazon/aws/{example_redshift_s3_transfers.py =>
example_sql_to_s3.py} (76%)
[GitHub] [airflow-site] potiuk commented on issue #677: Possible incomplete page published.
potiuk commented on issue #677: URL: https://github.com/apache/airflow-site/issues/677#issuecomment-1292856423 fixed version should be up in ~10 minutes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow-site] branch main updated: Remove old and unused install page (#678)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow-site.git
The following commit(s) were added to refs/heads/main by this push:
new 71ac7c8bcf Remove old and unused install page (#678)
71ac7c8bcf is described below
commit 71ac7c8bcfd0c1c55d14f89699a0fbe39fbd1923
Author: Jarek Potiuk
AuthorDate: Thu Oct 27 03:42:25 2022 +0200
Remove old and unused install page (#678)
Fixes: #677
---
landing-pages/site/content/en/install/_index.html | 124 --
1 file changed, 124 deletions(-)
diff --git a/landing-pages/site/content/en/install/_index.html
b/landing-pages/site/content/en/install/_index.html
deleted file mode 100644
index 86c8f0ac09..00
--- a/landing-pages/site/content/en/install/_index.html
+++ /dev/null
@@ -1,124 +0,0 @@
-title: Install
-menu:
-main:
-weight: 30
-
-
-{{< accordion title="Install Apache Airflow locally" description="Working
on an Open Source project such as Apache Airflow is very demanding but also
equally rewarding when you realize how many businesses use it every day."
open="true">}}
-When installing locally is
a good option for you?
-
-
-If you don’t want to do it by yourself
-If you don’t want to do it by yourself
-If you don’t want to do it by yourself
-If you don’t want to do it by yourself
-
-
-Available methods
-
-
-
-
-
-PIP
installation
-https://github.com";>
-Quick
start
-
-
-
-{{< /accordion >}}
-{{< accordion title="Install Apache Airflow on server"
description="Working on an Open Source project such as Apache Airflow is very
demanding but also equally rewarding when you realize how many businesses use
it every day." >}}
-When installing locally is
a good option for you?
-
-
-If you don’t want to do it by yourself
-If you don’t want to do it by yourself
-If you don’t want to do it by yourself
-If you don’t want to do it by yourself
-
-
-Available methods
-
-
-
-
-
-PIP
installation
-https://github.com";>
-Quick
start
-
-
-
-{{< /accordion >}}
-{{< accordion title="Install Apache Airflow on cluster"
description="Working on an Open Source project such as Apache Airflow is very
demanding but also equally rewarding when you realize how many businesses use
it every day." >}}
-When installing locally is
a good option for you?
-
-
-If you don’t want to do it by yourself
-If you don’t want to do it by yourself
-If you don’t want to do it by yourself
-If you don’t want to do it by yourself
-
-
-Available methods
-
-
-
-
-
-PIP
installation
-https://github.com";>
-Quick
start
-
-
-
-{{< /accordion >}}
-{{< accordion title="Install Apache Airflow to contribute"
description="Working on an Open Source project such as Apache Airflow is very
demanding but also equally rewarding when you realize how many businesses use
it every day." >}}
-When installing locally is
a good option for you?
-
-
-If you don’t want to do it by yourself
-If you don’t want to do it by yourself
-If you don’t want to do it by yourself
-If you don’t want to do it by yourself
-
-
-Available methods
-
-
-
-
-
-PIP
installation
-https://github.com";>
-Quick
start
-
-
-
-{{< /accordion >}}
-{{< accordion title="Install Apache Airflow by Cloud" description="Working
on an Open Source project such as Apache Airflow is very demanding but also
equally rewarding when you realize how many businesses use it every day." >}}
-When installing locally is
a good option for you?
-
-
-If you don’t want to do it by yourself
-If you don’t want to do it by yourself
-If you don’t want to do it by yourself
-If you don’t want to do it by yourself
-
-
-Available methods
-
-
-
-
-
-PIP
installation
-https://github.com";>
-Quick
start
-
-
-
-{{< /accordion >}}
-
[GitHub] [airflow-site] potiuk merged pull request #678: Remove old and unused install page
potiuk merged PR #678: URL: https://github.com/apache/airflow-site/pull/678 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow-site] potiuk closed issue #677: Possible incomplete page published.
potiuk closed issue #677: Possible incomplete page published. URL: https://github.com/apache/airflow-site/issues/677 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow-site] potiuk commented on issue #677: Possible incomplete page published.
potiuk commented on issue #677: URL: https://github.com/apache/airflow-site/issues/677#issuecomment-1292849139 Thanks!. Not sure how you got there - because this (apparently old and unlinked) page does not seem to be linked anywhere. Added #678 to remove it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow-site] potiuk opened a new pull request, #678: Remove old and unused install page
potiuk opened a new pull request, #678: URL: https://github.com/apache/airflow-site/pull/678 Fixes: #677 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow-site] branch remove-install-page created (now dbe514f905)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch remove-install-page in repository https://gitbox.apache.org/repos/asf/airflow-site.git at dbe514f905 Remove old and unused install page This branch includes the following new commits: new dbe514f905 Remove old and unused install page The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[airflow-site] 01/01: Remove old and unused install page
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch remove-install-page
in repository https://gitbox.apache.org/repos/asf/airflow-site.git
commit dbe514f90592fef621c3e207e0491ee6c92f4567
Author: Jarek Potiuk
AuthorDate: Thu Oct 27 03:27:16 2022 +0200
Remove old and unused install page
Fixes: #677
---
landing-pages/site/content/en/install/_index.html | 124 --
1 file changed, 124 deletions(-)
diff --git a/landing-pages/site/content/en/install/_index.html
b/landing-pages/site/content/en/install/_index.html
deleted file mode 100644
index 86c8f0ac09..00
--- a/landing-pages/site/content/en/install/_index.html
+++ /dev/null
@@ -1,124 +0,0 @@
-title: Install
-menu:
-main:
-weight: 30
-
-
-{{< accordion title="Install Apache Airflow locally" description="Working
on an Open Source project such as Apache Airflow is very demanding but also
equally rewarding when you realize how many businesses use it every day."
open="true">}}
-When installing locally is
a good option for you?
-
-
-If you don’t want to do it by yourself
-If you don’t want to do it by yourself
-If you don’t want to do it by yourself
-If you don’t want to do it by yourself
-
-
-Available methods
-
-
-
-
-
-PIP
installation
-https://github.com";>
-Quick
start
-
-
-
-{{< /accordion >}}
-{{< accordion title="Install Apache Airflow on server"
description="Working on an Open Source project such as Apache Airflow is very
demanding but also equally rewarding when you realize how many businesses use
it every day." >}}
-When installing locally is
a good option for you?
-
-
-If you don’t want to do it by yourself
-If you don’t want to do it by yourself
-If you don’t want to do it by yourself
-If you don’t want to do it by yourself
-
-
-Available methods
-
-
-
-
-
-PIP
installation
-https://github.com";>
-Quick
start
-
-
-
-{{< /accordion >}}
-{{< accordion title="Install Apache Airflow on cluster"
description="Working on an Open Source project such as Apache Airflow is very
demanding but also equally rewarding when you realize how many businesses use
it every day." >}}
-When installing locally is
a good option for you?
-
-
-If you don’t want to do it by yourself
-If you don’t want to do it by yourself
-If you don’t want to do it by yourself
-If you don’t want to do it by yourself
-
-
-Available methods
-
-
-
-
-
-PIP
installation
-https://github.com";>
-Quick
start
-
-
-
-{{< /accordion >}}
-{{< accordion title="Install Apache Airflow to contribute"
description="Working on an Open Source project such as Apache Airflow is very
demanding but also equally rewarding when you realize how many businesses use
it every day." >}}
-When installing locally is
a good option for you?
-
-
-If you don’t want to do it by yourself
-If you don’t want to do it by yourself
-If you don’t want to do it by yourself
-If you don’t want to do it by yourself
-
-
-Available methods
-
-
-
-
-
-PIP
installation
-https://github.com";>
-Quick
start
-
-
-
-{{< /accordion >}}
-{{< accordion title="Install Apache Airflow by Cloud" description="Working
on an Open Source project such as Apache Airflow is very demanding but also
equally rewarding when you realize how many businesses use it every day." >}}
-When installing locally is
a good option for you?
-
-
-If you don’t want to do it by yourself
-If you don’t want to do it by yourself
-If you don’t want to do it by yourself
-If you don’t want to do it by yourself
-
-
-Available methods
-
-
-
-
-
-PIP
installation
-https://github.com";>
-Quick
start
-
-
-
-{{< /accordion >}}
-
[GitHub] [airflow-site] ashshuota opened a new issue, #677: Possible incomplete page published.
ashshuota opened a new issue, #677: URL: https://github.com/apache/airflow-site/issues/677 The [installation page](https://airflow.apache.org/install/) contains a lot of boilerplate content and missing links and by no means is ready for production. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #26035: add emr steps sensor
github-actions[bot] commented on PR #26035: URL: https://github.com/apache/airflow/pull/26035#issuecomment-1292800080 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (6b2dc6f2cc -> b757bfa44c)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 6b2dc6f2cc System test for SQL to S3 Transfer (AIP-47) (#27097) add b757bfa44c Ignore azure-datalake-store deprecated mod use (#27287) No new revisions were added by this update. Summary of changes: scripts/in_container/verify_providers.py | 10 +++--- 1 file changed, 7 insertions(+), 3 deletions(-)
[GitHub] [airflow] potiuk merged pull request #27287: Ignore azure-datalake-store deprecated mod use
potiuk merged PR #27287: URL: https://github.com/apache/airflow/pull/27287 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (f9b02db3e4 -> 6b2dc6f2cc)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
from f9b02db3e4 Remove short form of --upgrade-on-failure (#27270)
add 6b2dc6f2cc System test for SQL to S3 Transfer (AIP-47) (#27097)
No new revisions were added by this update.
Summary of changes:
.../amazon/aws/example_dags/example_sql_to_s3.py | 43 -
.../providers/amazon/aws/transfers/sql_to_s3.py| 1 +
.../operators/transfer/sql_to_s3.rst | 2 +-
...dshift_s3_transfers.py => example_sql_to_s3.py} | 103 ++---
4 files changed, 31 insertions(+), 118 deletions(-)
delete mode 100644
airflow/providers/amazon/aws/example_dags/example_sql_to_s3.py
copy tests/system/providers/amazon/aws/{example_redshift_s3_transfers.py =>
example_sql_to_s3.py} (76%)
[GitHub] [airflow] potiuk merged pull request #27097: System test for SQL to S3 Transfer (AIP-47)
potiuk merged PR #27097: URL: https://github.com/apache/airflow/pull/27097 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27285: Using .output on non-templated fields
potiuk commented on issue #27285:
URL: https://github.com/apache/airflow/issues/27285#issuecomment-1292795423
Following the comment - I have a bold proposal ... It's not exaclty what the
original proposal is, but in a way it provides a possibility to do what was
originally requested here.
Why don't we add an option (disabled by default) to make ALL ELIGIBLE fields
- "templeted_fields" (and automatically .output -capable).
That bothered me for a while but I think there is very little impact of
making all fields templated and often people complained that templated fields.
Performance overhead should be negligible (just walking through parameters and
jinjafying them which in most cases will be no-op).
The only drawback it might have is that the if a string contains " {{}}"
acidentally - this will be replaced with "" - which is backwards-incompatible.
We could also provide a mechanism that would eclude a field from being
templated just in case.
I think that has a number of benefits - for example our users will not have
extend operators that miss some fields in "templated_fields".
I am not too worried anout "outlets" and executor_config not being
available for .output and user's education. As long as we simply error out in
this case that should be good.
In a way it woudl be similar to `render_template_as_native_obj` DAG
paraemeter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] rossturk commented on pull request #27226: Fix Google Analytics tag version
rossturk commented on PR #27226: URL: https://github.com/apache/airflow/pull/27226#issuecomment-1292784509 Correct - Matomo is gathering analytics for the Airflow website & docs, and they can be accessed unauthenticated via API or web @ analytics.apache.org. I think we can remove the GA tags completely đź‘Ť -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] vchiapaikeo commented on issue #27296: Task completes work but ends up failing due to a lock wait timeout exceeded error and does not honor retries
vchiapaikeo commented on issue #27296: URL: https://github.com/apache/airflow/issues/27296#issuecomment-1292782722 Adding a bit more analysis here. I'm noticing the only place where a query like this (SQL: UPDATE dag_run SET last_scheduling_decision=%s WHERE dag_run.id = %s) would be run is here in `DagRun.update_state`: https://github.com/apache/airflow/blob/2.4.2/airflow/models/dagrun.py#L516-L518 Specifically, `last_scheduling_decision` gets set here: https://github.com/apache/airflow/blob/2.4.2/airflow/models/dagrun.py#L552 And I think the most likely place that `DagRun.update_state` is being called from is here in `SchedulerJob._schedule_dag_run`: https://github.com/apache/airflow/blob/2.4.2/airflow/jobs/scheduler_job.py#L1242-L1246 https://github.com/apache/airflow/blob/2.4.2/airflow/jobs/scheduler_job.py#L1301 What I don't quite understand is if this is a call from SchedulerJob to update the dagrun, why are we seeing these logs on the worker pod? Is that because we're using KubernetesExecutor and the airflow worker pod itself is actually run as LocalExecutor? I also wonder what could be holding a lock on this same record for >50s... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] lihan commented on issue #27300: Scheduler encounters database update error, then gets stuck in endless loop, yet still shows as healthy
lihan commented on issue #27300: URL: https://github.com/apache/airflow/issues/27300#issuecomment-1292776762 Same error message was observed in https://github.com/apache/airflow/issues/27259 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #27226: Fix Google Analytics tag version
potiuk commented on PR #27226: URL: https://github.com/apache/airflow/pull/27226#issuecomment-1292774646 Hmm. I thought we wanted to get rid of Analytics in favour of Matomo (which is far more privacy concerned and ASF supported). https://github.com/apache/airflow-site/pull/581 and https://github.com/apache/airflow-site/pull/576 @rossturk - I think you were looking at it - is matomo working for our website ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #26944: Use DbApiHook.run for DbApiHook.get_records and DbApiHook.get_first
potiuk commented on code in PR #26944: URL: https://github.com/apache/airflow/pull/26944#discussion_r1006269450 ## airflow/providers/common/sql/hooks/sql.py: ## @@ -175,41 +207,26 @@ def get_pandas_df_by_chunks(self, sql, parameters=None, *, chunksize, **kwargs): yield from psql.read_sql(sql, con=conn, params=parameters, chunksize=chunksize, **kwargs) def get_records( -self, -sql: str | list[str], -parameters: Iterable | Mapping | None = None, -**kwargs: dict, -): +self, sql: str | list[str], parameters: Iterable | Mapping | None = None +) -> Any | list[Any]: """ Executes the sql and returns a set of records. -:param sql: the sql statement to be executed (str) or a list of -sql statements to execute +:param sql: the sql statement to be executed (str) or a list of sql statements to execute :param parameters: The parameters to render the SQL query with. """ -with closing(self.get_conn()) as conn: -with closing(conn.cursor()) as cur: -if parameters is not None: -cur.execute(sql, parameters) -else: -cur.execute(sql) -return cur.fetchall() +return self.run(sql=sql, parameters=parameters, handler=fetch_all_handler) -def get_first(self, sql: str | list[str], parameters=None): +def get_first( +self, sql: str | list[str], parameters: Iterable | Mapping | None = None +) -> Any | list[Any]: Review Comment: Same here I think `Any` should do. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #26944: Use DbApiHook.run for DbApiHook.get_records and DbApiHook.get_first
potiuk commented on code in PR #26944: URL: https://github.com/apache/airflow/pull/26944#discussion_r1006269110 ## airflow/providers/common/sql/hooks/sql.py: ## @@ -175,41 +207,26 @@ def get_pandas_df_by_chunks(self, sql, parameters=None, *, chunksize, **kwargs): yield from psql.read_sql(sql, con=conn, params=parameters, chunksize=chunksize, **kwargs) def get_records( -self, -sql: str | list[str], -parameters: Iterable | Mapping | None = None, -**kwargs: dict, -): +self, sql: str | list[str], parameters: Iterable | Mapping | None = None +) -> Any | list[Any]: Review Comment: I think that should also remove the # type: ignore[override] in subcleases? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #26944: Use DbApiHook.run for DbApiHook.get_records and DbApiHook.get_first
potiuk commented on code in PR #26944: URL: https://github.com/apache/airflow/pull/26944#discussion_r1006268609 ## airflow/providers/common/sql/hooks/sql.py: ## @@ -175,41 +207,26 @@ def get_pandas_df_by_chunks(self, sql, parameters=None, *, chunksize, **kwargs): yield from psql.read_sql(sql, con=conn, params=parameters, chunksize=chunksize, **kwargs) def get_records( -self, -sql: str | list[str], -parameters: Iterable | Mapping | None = None, -**kwargs: dict, -): +self, sql: str | list[str], parameters: Iterable | Mapping | None = None +) -> Any | list[Any]: Review Comment: Isn't `Any | list[Any] == Any` ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #26944: Use DbApiHook.run for DbApiHook.get_records and DbApiHook.get_first
potiuk commented on code in PR #26944: URL: https://github.com/apache/airflow/pull/26944#discussion_r1006268023 ## airflow/providers/common/sql/hooks/sql.py: ## @@ -38,6 +41,35 @@ def fetch_all_handler(cursor) -> list[tuple] | None: return None +def fetch_one_handler(cursor) -> list[tuple] | None: +"""Handler for DbApiHook.run() to return results""" +if cursor.description is not None: +return cursor.fetchone() +else: +return None + + +def _backported_get_hook(connection, *, hook_params=None): +"""Return hook based on conn_type +For supporting Airflow versions < 2.3, we backport "get_hook()" method. This should be removed +when "apache-airflow-providers-slack" will depend on Airflow >= 2.3. Review Comment: Agree we cam remove it (and add apache-airflow >= 2.3 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #26944: Use DbApiHook.run for DbApiHook.get_records and DbApiHook.get_first
potiuk commented on code in PR #26944: URL: https://github.com/apache/airflow/pull/26944#discussion_r1006267713 ## airflow/providers/common/sql/hooks/sql.py: ## @@ -38,6 +41,35 @@ def fetch_all_handler(cursor) -> list[tuple] | None: return None +def fetch_one_handler(cursor) -> list[tuple] | None: +"""Handler for DbApiHook.run() to return results""" +if cursor.description is not None: +return cursor.fetchone() +else: +return None + + +def _backported_get_hook(connection, *, hook_params=None): +"""Return hook based on conn_type +For supporting Airflow versions < 2.3, we backport "get_hook()" method. This should be removed +when "apache-airflow-providers-slack" will depend on Airflow >= 2.3. Review Comment: No particular reason. Just a mistake :) . I think it would be great to add it :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #27250: Add note about pushing the lazy XCom proxy to XCom
uranusjr commented on code in PR #27250: URL: https://github.com/apache/airflow/pull/27250#discussion_r1006263748 ## docs/apache-airflow/concepts/dynamic-task-mapping.rst: ## @@ -72,6 +72,17 @@ The grid view also provides visibility into your mapped tasks in the details pan You can use normal sequence syntax on this object (e.g. ``values[0]``), or iterate through it normally with a ``for`` loop. ``list(values)`` will give you a "real" ``list``, but please be aware of the potential performance implications if the list is large. +Note that the same also applies to when you push this proxy object into XCom. This, for example, would not +work with the default XCom backend: + +.. code-block:: python + +@task +def forward_values(values): +return values # This is a lazy proxy and can't be pushed! + +You need to explicitly call ``list(values)`` instead, and accept the performance implications. Review Comment: It entirely depends on what the user pushes into the XCom and how the data is stored so there’s no way to really describe fully. All the normal XCom caveats apply, times how many upstream tasks were mapped into. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #27270: Remove short form of --upgrade-on-failure
potiuk commented on PR #27270: URL: https://github.com/apache/airflow/pull/27270#issuecomment-1292728805 Now-> I resolved conflicts in https://github.com/apache/airflow/pull/27191 after merging it ... Good revires there are welcome so that I can merge it :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (734995ff26 -> f9b02db3e4)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 734995ff26 Add deprecation warning re unset namespace in k8s hook (#27202) add f9b02db3e4 Remove short form of --upgrade-on-failure (#27270) No new revisions were added by this update. Summary of changes: .../src/airflow_breeze/utils/common_options.py | 1 - images/breeze/output-commands-hash.txt | 8 +- images/breeze/output_ci-image_build.svg| 292 images/breeze/output_prod-image_build.svg | 376 ++--- 4 files changed, 338 insertions(+), 339 deletions(-)
[GitHub] [airflow] potiuk merged pull request #27270: Remove short form of --upgrade-on-failure
potiuk merged PR #27270: URL: https://github.com/apache/airflow/pull/27270 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #27304: Fix coverage upload step
potiuk commented on PR #27304: URL: https://github.com/apache/airflow/pull/27304#issuecomment-1292722400 test failing :( -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] dstandish commented on a diff in pull request #27197: Drop support for providing ``resource`` as dict in ``KubernetesPodOperator``
dstandish commented on code in PR #27197:
URL: https://github.com/apache/airflow/pull/27197#discussion_r1006233579
##
airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py:
##
@@ -213,21 +212,18 @@ def __init__(
pod_runtime_info_envs: list[k8s.V1EnvVar] | None = None,
termination_grace_period: int | None = None,
configmaps: list[str] | None = None,
-resources: dict[str, Any] | None = None,
**kwargs,
) -> None:
-
-if isinstance(resources, k8s.V1ResourceRequirements):
-warnings.warn(
+# TODO: remove in future provider release. This is a mitigate step to
advise users to switch to the
+# container_resources parameter.
+resources = kwargs.get("resources")
+if resources is not None and isinstance(resources,
k8s.V1ResourceRequirements):
Review Comment:
```suggestion
if isinstance(resources, k8s.V1ResourceRequirements):
```
same thing i think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #27304: Fix coverage upload step
potiuk commented on PR #27304: URL: https://github.com/apache/airflow/pull/27304#issuecomment-1292721920 Nice. Does it mean that somene ACTUALLY looks at those reports :D ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] jedcunningham opened a new pull request, #27304: Fix coverage upload step
jedcunningham opened a new pull request, #27304: URL: https://github.com/apache/airflow/pull/27304 A simple typo was preventing coverage uploads from happening, and this fixes it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (87eb46bbc6 -> 734995ff26)
This is an automated email from the ASF dual-hosted git repository. dstandish pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 87eb46bbc6 Common sql bugfixes and improvements (#26761) add 734995ff26 Add deprecation warning re unset namespace in k8s hook (#27202) No new revisions were added by this update. Summary of changes: airflow/providers/cncf/kubernetes/CHANGELOG.rst| 7 .../providers/cncf/kubernetes/hooks/kubernetes.py | 39 +- .../cncf/kubernetes/hooks/test_kubernetes.py | 11 ++ 3 files changed, 49 insertions(+), 8 deletions(-)
[GitHub] [airflow] dstandish merged pull request #27202: Add deprecation warning re unset namespace in k8s hook
dstandish merged PR #27202: URL: https://github.com/apache/airflow/pull/27202 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #26761: Common sql bugfixes and improvements
potiuk commented on PR #26761: URL: https://github.com/apache/airflow/pull/26761#issuecomment-1292700310 ❤️ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (e6c8c076f4 -> 87eb46bbc6)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from e6c8c076f4 Assign public IP to ECS task network interface to allow the task to pull container image (#27303) add 87eb46bbc6 Common sql bugfixes and improvements (#26761) No new revisions were added by this update. Summary of changes: airflow/providers/common/sql/operators/sql.py | 233 - .../providers/google/cloud/operators/bigquery.py | 107 +- .../operators.rst | 31 ++- tests/providers/common/sql/operators/test_sql.py | 225 ++-- .../cloud/bigquery/example_bigquery_queries.py | 2 +- 5 files changed, 413 insertions(+), 185 deletions(-)
[GitHub] [airflow] potiuk merged pull request #26761: Common sql bugfixes and improvements
potiuk merged PR #26761: URL: https://github.com/apache/airflow/pull/26761 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (49e618054b -> e6c8c076f4)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 49e618054b Make grid view widths adjustable (#27273) add e6c8c076f4 Assign public IP to ECS task network interface to allow the task to pull container image (#27303) No new revisions were added by this update. Summary of changes: tests/system/providers/amazon/aws/example_ecs_fargate.py | 1 + 1 file changed, 1 insertion(+)
[GitHub] [airflow] potiuk merged pull request #27303: Fix ECS with Fargate system test
potiuk merged PR #27303: URL: https://github.com/apache/airflow/pull/27303 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] syedahsn opened a new pull request, #27303: Fix ECS with Fargate system test
syedahsn opened a new pull request, #27303:
URL: https://github.com/apache/airflow/pull/27303
The ECS with Fargate system test was failing due to a network configuration
issue, resulting in the task not being able to pull the container image. The
solution is to assign a public IP to ECS task network interface to allow the
task to pull container image.
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[airflow] branch main updated (13b6b3b078 -> 49e618054b)
This is an automated email from the ASF dual-hosted git repository. bbovenzi pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 13b6b3b078 Added docs regarding templated field (#27301) add 49e618054b Make grid view widths adjustable (#27273) No new revisions were added by this update. Summary of changes: airflow/www/static/js/dag/Main.tsx | 85 + airflow/www/static/js/dag/details/index.tsx | 4 +- airflow/www/static/js/dag/grid/index.tsx| 11 ++-- 3 files changed, 84 insertions(+), 16 deletions(-)
[GitHub] [airflow] bbovenzi merged pull request #27273: Make grid view widths adjustable
bbovenzi merged PR #27273: URL: https://github.com/apache/airflow/pull/27273 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] dstandish commented on a diff in pull request #26739: Added Istio support to KubernetesPodOperator
dstandish commented on code in PR #26739: URL: https://github.com/apache/airflow/pull/26739#discussion_r1006191640 ## airflow/providers/cncf/kubernetes/utils/pod_manager.py: ## @@ -272,6 +304,10 @@ def await_pod_completion(self, pod: V1Pod) -> V1Pod: remote_pod = self.read_pod(pod) if remote_pod.status.phase in PodPhase.terminal_states: break +if self.istio_enabled and remote_pod.status.phase == PodPhase.RUNNING and self.container_is_completed( +remote_pod, 'base' Review Comment: `'base'` is defined in a constant somewhere -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] dstandish commented on a diff in pull request #26739: Added Istio support to KubernetesPodOperator
dstandish commented on code in PR #26739: URL: https://github.com/apache/airflow/pull/26739#discussion_r1006190619 ## airflow/providers/cncf/kubernetes/utils/pod_manager.py: ## @@ -84,6 +84,36 @@ def container_is_running(pod: V1Pod, container_name: str) -> bool: return container_status.state.running is not None +def container_is_completed(pod: V1Pod, container_name: str) -> bool: +""" +Examines V1Pod ``pod`` to determine whether ``container_name`` is completed. +If that container is present and completed, returns True. Returns False otherwise. +""" +container_statuses = pod.status.container_statuses if pod and pod.status else None +if not container_statuses: +return False +container_status = next(iter([x for x in container_statuses if x.name == container_name]), None) +if not container_status: +return False +return container_status.state.terminated is not None + + +def container_is_succeeded(pod: V1Pod, container_name: str) -> bool: +""" +Examines V1Pod ``pod`` to determine whether ``container_name`` is completed and succeeded. +If that container is present and completed and succeeded, returns True. Returns False otherwise. +""" +if not container_is_completed(pod, container_name): +return False +container_statuses = pod.status.container_statuses if pod and pod.status else None +if not container_statuses: +return False +container_status = next(iter([x for x in container_statuses if x.name == container_name]), None) +if not container_status: +return False +return container_status.state.terminated.exit_code == 0 Review Comment: seems you are duplicating a lot of logic here. looks like you could pull out a `get_container_status` function -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] dstandish commented on a diff in pull request #26739: Added Istio support to KubernetesPodOperator
dstandish commented on code in PR #26739: URL: https://github.com/apache/airflow/pull/26739#discussion_r1006188972 ## airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py: ## @@ -453,7 +455,7 @@ def cleanup(self, pod: k8s.V1Pod, remote_pod: k8s.V1Pod): if not self.is_delete_operator_pod: with _suppress(Exception): self.patch_already_checked(remote_pod) -if pod_phase != PodPhase.SUCCEEDED: +if (not self.istio_enabled and pod_phase != PodPhase.SUCCEEDED) or (self.istio_enabled and pod_phase != PodPhase.SUCCEEDED and not self.pod_manager.container_is_succeeded(pod, 'base')): Review Comment: this expression can probably be simplified. i would try to do so. it's too hard to understand as it is. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] dstandish commented on a diff in pull request #26739: Added Istio support to KubernetesPodOperator
dstandish commented on code in PR #26739: URL: https://github.com/apache/airflow/pull/26739#discussion_r1006188972 ## airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py: ## @@ -453,7 +455,7 @@ def cleanup(self, pod: k8s.V1Pod, remote_pod: k8s.V1Pod): if not self.is_delete_operator_pod: with _suppress(Exception): self.patch_already_checked(remote_pod) -if pod_phase != PodPhase.SUCCEEDED: +if (not self.istio_enabled and pod_phase != PodPhase.SUCCEEDED) or (self.istio_enabled and pod_phase != PodPhase.SUCCEEDED and not self.pod_manager.container_is_succeeded(pod, 'base')): Review Comment: this expression can probably be simplified. i would try to do so. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] dstandish commented on a diff in pull request #26739: Added Istio support to KubernetesPodOperator
dstandish commented on code in PR #26739: URL: https://github.com/apache/airflow/pull/26739#discussion_r1006188366 ## airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py: ## @@ -453,7 +455,7 @@ def cleanup(self, pod: k8s.V1Pod, remote_pod: k8s.V1Pod): if not self.is_delete_operator_pod: with _suppress(Exception): self.patch_already_checked(remote_pod) -if pod_phase != PodPhase.SUCCEEDED: +if (not self.istio_enabled and pod_phase != PodPhase.SUCCEEDED) or (self.istio_enabled and pod_phase != PodPhase.SUCCEEDED and not self.pod_manager.container_is_succeeded(pod, 'base')): Review Comment: you will need to enable pre-commit locally see https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] dstandish commented on a diff in pull request #27202: Add deprecation warning re unset namespace in k8s hook
dstandish commented on code in PR #27202:
URL: https://github.com/apache/airflow/pull/27202#discussion_r1006178099
##
airflow/providers/cncf/kubernetes/hooks/kubernetes.py:
##
@@ -319,9 +319,32 @@ def get_custom_object(
raise AirflowException(f"Exception when calling ->
get_custom_object: {e}\n")
def get_namespace(self) -> str | None:
-"""Returns the namespace that defined in the connection"""
+"""
+Returns the namespace defined in the connection or 'default'.
+
+TODO: in provider version 6.0, return None when namespace not defined
in connection
+"""
+namespace = self._get_namespace()
+if self.conn_id and not namespace:
+warnings.warn(
+"Airflow connection defined but namespace is not set;
'default'. In "
+"cncf.kubernetes provider version 6.0 we will return None when
namespace is "
+"not defined in the connection so that it's clear whether user
intends 'default' or "
+"whether namespace is unset (which is required in order to
apply precedence logic in "
+"KubernetesPodOperator.",
Review Comment:
i _always_ forget to close parentheses
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a diff in pull request #27202: Add deprecation warning re unset namespace in k8s hook
jedcunningham commented on code in PR #27202:
URL: https://github.com/apache/airflow/pull/27202#discussion_r1004949553
##
airflow/providers/cncf/kubernetes/hooks/kubernetes.py:
##
@@ -319,9 +319,32 @@ def get_custom_object(
raise AirflowException(f"Exception when calling ->
get_custom_object: {e}\n")
def get_namespace(self) -> str | None:
-"""Returns the namespace that defined in the connection"""
+"""
+Returns the namespace defined in the connection or 'default'.
+
+TODO: in provider version 6.0, return None when namespace not defined
in connection
+"""
+namespace = self._get_namespace()
+if self.conn_id and not namespace:
+warnings.warn(
+"Airflow connection defined but namespace is not set;
'default'. In "
Review Comment:
```suggestion
"Airflow connection defined but namespace is not set;
returning 'default'. In "
```
##
airflow/providers/cncf/kubernetes/CHANGELOG.rst:
##
@@ -39,6 +39,13 @@ Features
* KubernetsPodOperator argument ``namespace`` is now optional. If not
supplied via KPO param or pod template file or full pod spec, then we'll check
the airflow conn,
then if in a k8s pod, try to infer the namespace from the container, then
finally will use the ``default`` namespace.
+Deprecations
+
+
+* In KubernetesHook if connection defined but namespace unset, we currently
return 'default'; this behavior is deprecated. In next release, we'll return
``None``.
Review Comment:
```suggestion
* In ``KubernetesHook.get_namespace``, if a connection is defined but a
namespace isn't set, we currently return 'default'; this behavior is
deprecated. In the next release, we'll return ``None``.
```
##
airflow/providers/cncf/kubernetes/hooks/kubernetes.py:
##
@@ -319,9 +319,32 @@ def get_custom_object(
raise AirflowException(f"Exception when calling ->
get_custom_object: {e}\n")
def get_namespace(self) -> str | None:
-"""Returns the namespace that defined in the connection"""
+"""
+Returns the namespace defined in the connection or 'default'.
+
+TODO: in provider version 6.0, return None when namespace not defined
in connection
+"""
+namespace = self._get_namespace()
+if self.conn_id and not namespace:
+warnings.warn(
+"Airflow connection defined but namespace is not set;
'default'. In "
+"cncf.kubernetes provider version 6.0 we will return None when
namespace is "
+"not defined in the connection so that it's clear whether user
intends 'default' or "
+"whether namespace is unset (which is required in order to
apply precedence logic in "
+"KubernetesPodOperator.",
Review Comment:
```suggestion
"KubernetesPodOperator).",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[airflow] branch main updated (1af25809b1 -> 13b6b3b078)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 1af25809b1 Fix a typo in Breeze doc (#27293) add 13b6b3b078 Added docs regarding templated field (#27301) No new revisions were added by this update. Summary of changes: airflow/providers/ssh/operators/ssh.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[GitHub] [airflow] potiuk merged pull request #27301: Added docs regarding templated field
potiuk merged PR #27301: URL: https://github.com/apache/airflow/pull/27301 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #27273: Make grid view widths adjustable
potiuk commented on PR #27273: URL: https://github.com/apache/airflow/pull/27273#issuecomment-1292607645 Making Airflow UI better one piece at-a-time :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] bbovenzi commented on pull request #27273: Make grid view widths adjustable
bbovenzi commented on PR #27273: URL: https://github.com/apache/airflow/pull/27273#issuecomment-1292578439 I cleaned up the magic numbers. I don't think I understand the "jump" you're seeing. A gif would be helpful. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] RachitSharma2001 commented on pull request #26974: Add FTP Operator
RachitSharma2001 commented on PR #26974: URL: https://github.com/apache/airflow/pull/26974#issuecomment-1292538043 @potiuk @eladkal @kaxil I have fixed the static check test failure from the previously ran workflow. When you have the time, could you approve the workflow for my latest commit? Thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #27214: Refactor amazon providers tests which use `moto`
Taragolis commented on code in PR #27214:
URL: https://github.com/apache/airflow/pull/27214#discussion_r1006081685
##
tests/providers/amazon/conftest.py:
##
@@ -0,0 +1,59 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import warnings
+
+try:
+import importlib_metadata
+except ImportError:
+from importlib import metadata as importlib_metadata # type:
ignore[no-redef]
+
+import pytest
+
+
+@pytest.fixture(scope="session")
+def botocore_version():
+try:
+version = importlib_metadata.version("botocore")
+except importlib_metadata.PackageNotFoundError:
+warnings.warn("'botocore' package not found'", UserWarning)
+return None
+
+try:
+return tuple(map(int, version.split(".")[:3]))
+except Exception:
+warnings.warn(f"Unable to parse botocore {version!r}", UserWarning)
+return None
+
+
+@pytest.fixture(autouse=True)
+def filter_botocore_warnings(botocore_version):
+"""Filter known botocore future warnings."""
+
+with warnings.catch_warnings():
+if botocore_version and botocore_version < (1, 28):
+# Deprecate Usage of sslComonName in Endpoint Creation.
+# See: https://github.com/boto/botocore/issues/2705
+warnings.filterwarnings(
+"ignore",
+category=FutureWarning,
+module="botocore.client",
+message="The .* client is currently using a deprecated
endpoint.*",
+)
+yield
Review Comment:
Got an answer that default endpoints would change in botocore 1.29
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #27214: Refactor amazon providers tests which use `moto`
Taragolis commented on code in PR #27214:
URL: https://github.com/apache/airflow/pull/27214#discussion_r1006081685
##
tests/providers/amazon/conftest.py:
##
@@ -0,0 +1,59 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import warnings
+
+try:
+import importlib_metadata
+except ImportError:
+from importlib import metadata as importlib_metadata # type:
ignore[no-redef]
+
+import pytest
+
+
+@pytest.fixture(scope="session")
+def botocore_version():
+try:
+version = importlib_metadata.version("botocore")
+except importlib_metadata.PackageNotFoundError:
+warnings.warn("'botocore' package not found'", UserWarning)
+return None
+
+try:
+return tuple(map(int, version.split(".")[:3]))
+except Exception:
+warnings.warn(f"Unable to parse botocore {version!r}", UserWarning)
+return None
+
+
+@pytest.fixture(autouse=True)
+def filter_botocore_warnings(botocore_version):
+"""Filter known botocore future warnings."""
+
+with warnings.catch_warnings():
+if botocore_version and botocore_version < (1, 28):
+# Deprecate Usage of sslComonName in Endpoint Creation.
+# See: https://github.com/boto/botocore/issues/2705
+warnings.filterwarnings(
+"ignore",
+category=FutureWarning,
+module="botocore.client",
+message="The .* client is currently using a deprecated
endpoint.*",
+)
+yield
Review Comment:
Got an answer that default endpoints will changed in botocore 1.29
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] pierrejeambrun commented on pull request #27273: Make grid view widths adjustable
pierrejeambrun commented on PR #27273: URL: https://github.com/apache/airflow/pull/27273#issuecomment-1292477728 Tested locally and working great. I just have one issue when displaying details for a task group with long 'tooltip'. There is a weird 'jump' when trying to resize at first. Something like this:  And then on clicking on the separator for a rezise, it flickers, and the separator jumps to the left right away:  This seems ike something that could be just on my side and cause by my local setup: - Version 103.0.5060.134 (Official Build) (64-bit) - Ubuntu 20.04.4 LTS -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] jayceslesar commented on issue #19477: 'NoneType' object has no attribute 'refresh_from_task' Error when manually running task instance
jayceslesar commented on issue #19477: URL: https://github.com/apache/airflow/issues/19477#issuecomment-1292475921 This was fixed in the UI but not the cli -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] bdsoha opened a new pull request, #27301: Added docs regarding templated field
bdsoha opened a new pull request, #27301: URL: https://github.com/apache/airflow/pull/27301 Added missing *docs* regarding the `environment` templated field. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] hugowangler commented on issue #27299: `TriggerDagRunOperator` task fails with index out of range exception while trying to reset dag run
hugowangler commented on issue #27299: URL: https://github.com/apache/airflow/issues/27299#issuecomment-1292445658 I actually noticed now that the `example` DAG is actually correctly triggered and rerun on with the specified `execution_date`. So it's only the `trigger` task in `trigger_example` that "fails". Would still consider it a bug but the actual trigger of the dag run still happens. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] denimalpaca commented on pull request #26761: Common sql bugfixes and improvements
denimalpaca commented on PR #26761: URL: https://github.com/apache/airflow/pull/26761#issuecomment-1292438001 @potiuk fixed at time of commenting :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ejstembler opened a new issue, #27300: Scheduler encounters database update error, then gets stuck in endless loop, yet still shows as healthy
ejstembler opened a new issue, #27300: URL: https://github.com/apache/airflow/issues/27300 ### Apache Airflow version Other Airflow 2 version (please specify below) ### What happened Airflow version: `v2.3.3+astro.2`. We've encounter this issue twice this year. Something causes the Scheduler to get stuck in an endless loop, yet it shows as healthy even though nothing is being processed. The last time we encounter this issue, this week. The Scheduler encountered a database update error: ``` sqlalchemy.orm.exc.StaleDataError: UPDATE statement on table 'dag' expected to update 1 row(s); 0 were matched. ``` As a result, the Schedule logs should it's stuck in an endless loop, the same messages are repeating over-and-over.  Because of this, nothing runs, and the entire Airflow instance is considered down. In this particular case, the issue was resolved by manually deleting the duplicate row in the `dag` table. When we encounter a similar case earlier in the year, the root cause was different and required a different solution. (Upsizing workers). ### What you think should happen instead The Scheduler should not crash or get stuck in an endless loop. It should handle exceptional cases gracefully. It should not be reported as healthy if it is crashing continuously or stuck in an endless loop. Some strategies for handling this, off the top of my head: * The Scheduler should have stricter error handling and when an error is encountered, it should log the error, and continue on to the next scheduled DAG. * The Scheduler itself should not be allowed to get into an endless loop. * Check the logs for repeating message patterns? * Keep a count to make sure DAGs are being run? * Use logarithmic or exponential backoff when retrying? ### How to reproduce Enter a duplicate row in the `dags` table. There are probably other ways. Earlier in the year we encounter this same issues when Workers were not properly upsized. ### Operating System Debian GNU/Linux 11 (bullseye) ### Versions of Apache Airflow Providers [apache-airflow-providers-http](https://pypi.python.org/pypi/apache-airflow-providers-http)==2.0.1 [apache-airflow-providers-jdbc](https://pypi.python.org/pypi/apache-airflow-providers-jdbc)==2.0.1 [simple-salesforce](https://pypi.python.org/pypi/simple-salesforce)==1.1.0 [csvvalidator](https://pypi.python.org/pypi/csvvalidator)==1.2 [pandas](https://pypi.python.org/pypi/pandas)==1.3.5 [pre-commit](https://pypi.python.org/pypi/pre-commit) [pylint](https://pypi.python.org/pypi/pylint)==2.15 [pytest](https://pypi.python.org/pypi/pytest)==6.2.5 [pyspark](https://pypi.python.org/pypi/pyspark)==3.3.0 [apache-airflow-providers-google](https://pypi.python.org/pypi/apache-airflow-providers-google)==6.4.0 ### Deployment Astronomer ### Deployment details Astronomer ### Anything else _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #27299: `TriggerDagRunOperator` task fails with index out of range exception while trying to reset dag run
boring-cyborg[bot] commented on issue #27299: URL: https://github.com/apache/airflow/issues/27299#issuecomment-1292400048 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] hugowangler opened a new issue, #27299: `TriggerDagRunOperator` task fails with index out of range exception while trying to reset dag run
hugowangler opened a new issue, #27299:
URL: https://github.com/apache/airflow/issues/27299
### Apache Airflow version
2.4.2
### What happened
List index out of range exception is raised when trying to trigger a DAG run
of another DAG using the `TriggerDagRunOperator` with `reset_dag_run=True`.
```
[2022-10-26, 17:13:38 UTC] {taskinstance.py:1165} INFO - Dependencies all
met for
[2022-10-26, 17:13:38 UTC] {taskinstance.py:1165} INFO - Dependencies all
met for
[2022-10-26, 17:13:38 UTC] {taskinstance.py:1362} INFO -
[2022-10-26, 17:13:38 UTC] {taskinstance.py:1363} INFO - Starting attempt 1
of 1
[2022-10-26, 17:13:38 UTC] {taskinstance.py:1364} INFO -
[2022-10-26, 17:13:38 UTC] {taskinstance.py:1383} INFO - Executing
on 2022-10-26 17:13:33+00:00
[2022-10-26, 17:13:38 UTC] {standard_task_runner.py:55} INFO - Started
process 2181 to run task
[2022-10-26, 17:13:38 UTC] {standard_task_runner.py:82} INFO - Running:
['airflow', 'tasks', 'run', 'trigger_example', 'trigger',
'manual__2022-10-26T17:13:33+00:00', '--job-id', '920', '--raw', '--subdir',
'DAGS_FOLDER/dags/trigger-example-dag.py', '--cfg-path', '/tmp/tmpmg9ay0du']
[2022-10-26, 17:13:38 UTC] {standard_task_runner.py:83} INFO - Job 920:
Subtask trigger
[2022-10-26, 17:13:38 UTC] {task_command.py:376} INFO - Running
on host airflow-worker-0.airflow-worker.airflow.svc.cluster.local
[2022-10-26, 17:13:38 UTC] {taskinstance.py:1590} INFO - Exporting the
following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=trigger_example
AIRFLOW_CTX_TASK_ID=trigger
AIRFLOW_CTX_EXECUTION_DATE=2022-10-26T17:13:33+00:00
AIRFLOW_CTX_TRY_NUMBER=1
AIRFLOW_CTX_DAG_RUN_ID=manual__2022-10-26T17:13:33+00:00
[2022-10-26, 17:13:38 UTC] {trigger_dagrun.py:146} INFO - Clearing example
on 2022-10-24T00:00:00+00:00
[2022-10-26, 17:13:38 UTC] {taskinstance.py:1851} ERROR - Task failed with
exception
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/trigger_dagrun.py",
line 136, in execute
dag_run = trigger_dag(
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/api/common/trigger_dag.py",
line 124, in trigger_dag
triggers = _trigger_dag(
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/api/common/trigger_dag.py",
line 78, in _trigger_dag
raise DagRunAlreadyExists(
airflow.exceptions.DagRunAlreadyExists: A Dag Run already exists for dag id
example at 2022-10-24T00:00:00+00:00 with run id
manual__2022-10-24T00:00:00+00:00
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/trigger_dagrun.py",
line 157, in execute
dag_run = DagRun.find(dag_id=dag.dag_id, run_id=run_id)[0]
IndexError: list index out of range
[2022-10-26, 17:13:38 UTC] {taskinstance.py:1401} INFO - Marking task as
FAILED. dag_id=trigger_example, task_id=trigger,
execution_date=20221026T171333, start_date=20221026T171338,
end_date=20221026T171338
[2022-10-26, 17:13:38 UTC] {standard_task_runner.py:100} ERROR - Failed to
execute job 920 for task trigger (list index out of range; 2181)
[2022-10-26, 17:13:38 UTC] {local_task_job.py:164} INFO - Task exited with
return code 1
[2022-10-26, 17:13:38 UTC] {local_task_job.py:273} INFO - 0 downstream tasks
scheduled from follow-on schedule check
```
### What you think should happen instead
The DAG run should be cleared since a run at the specified `execution_date`
exists, or if something else actually is wrong this should probably be logged
better so the user understands what's wrong their DAG.
### How to reproduce
To reproduce I used the following two DAGs
# example-dag.py
```
import pendulum
from airflow.decorators import task, dag
from airflow.operators.bash import BashOperator
@dag(
dag_id="example",
schedule="@daily",
start_date=pendulum.datetime(2022, 10, 24, tz="UTC"),
catchup=True,
)
def example():
hello = BashOperator(task_id="hello", bash_command="echo hello")
@task(task_id="airflow")
def airflow():
print("airflow")
hello >> airflow()
dag = example()
```
# trigger-example-dag.py
```
import pendulum
from airflow.decorators import dag, task
from airflow.operators.trigger_dagrun import TriggerDagRunOperator
@dag(
dag_id="trigger_example",
schedule="@daily",
start_date=pendulum.datetime(2022, 10, 25, tz="UTC"),
catchup=False,
)
def trigger_example_dag():
@task(task_id=
[GitHub] [airflow] syedahsn commented on a diff in pull request #27276: Redshift Pause and Resume Operator Refactor
syedahsn commented on code in PR #27276: URL: https://github.com/apache/airflow/pull/27276#discussion_r1005996857 ## airflow/providers/amazon/aws/hooks/redshift_cluster.py: ## @@ -166,7 +166,6 @@ def get_cluster_snapshot_status(self, snapshot_identifier: str, cluster_identifi """ try: response = self.get_conn().describe_cluster_snapshots( Review Comment: See above comment. Basically, the `describe_cluster_snapshots` function requires only one of the 2 parameters. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] syedahsn commented on a diff in pull request #27276: Redshift Pause and Resume Operator Refactor
syedahsn commented on code in PR #27276: URL: https://github.com/apache/airflow/pull/27276#discussion_r1005996122 ## airflow/providers/amazon/aws/hooks/redshift_cluster.py: ## @@ -157,7 +157,7 @@ def create_cluster_snapshot( ) return response["Snapshot"] if response["Snapshot"] else None -def get_cluster_snapshot_status(self, snapshot_identifier: str, cluster_identifier: str): +def get_cluster_snapshot_status(self, snapshot_identifier: str): Review Comment: Good point. The reason I removed the `cluster_identifier` is because the `describe_cluster_snapshots` function throws an `InvalidParameterCombination` when both the `cluster_identifier` and `snapshot_identifier` are provided. ``` botocore.exceptions.ClientError: An error occurred (InvalidParameterCombination) when calling the DescribeClusterSnapshots operation: Cannot specify both a Cluster identifier and a snapshot identifier. ``` I'll add the `cluster_identifier` back to the function definition, and add a deprecation warning to it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] pankajastro commented on a diff in pull request #27276: Redshift Pause and Resume Operator Refactor
pankajastro commented on code in PR #27276: URL: https://github.com/apache/airflow/pull/27276#discussion_r1005941984 ## airflow/providers/amazon/aws/hooks/redshift_cluster.py: ## @@ -166,7 +166,6 @@ def get_cluster_snapshot_status(self, snapshot_identifier: str, cluster_identifi """ try: response = self.get_conn().describe_cluster_snapshots( Review Comment: why this change, I think the purpose of this hook method is to get the snapshot of a cluster WDYT? ## airflow/providers/amazon/aws/hooks/redshift_cluster.py: ## @@ -157,7 +157,7 @@ def create_cluster_snapshot( ) return response["Snapshot"] if response["Snapshot"] else None -def get_cluster_snapshot_status(self, snapshot_identifier: str, cluster_identifier: str): +def get_cluster_snapshot_status(self, snapshot_identifier: str): Review Comment: I feel changing definition of a hook method is a breaking change -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] raphaelauv commented on pull request #27264: Attempt to add Python 3.11 support
raphaelauv commented on PR #27264: URL: https://github.com/apache/airflow/pull/27264#issuecomment-1292308899 ``` I think (let's see the tests) there are no problems with any of the dependencies that Airlfow Core has. Most of the problems come from our providers, ``` yes it's the case , I just made a test of airflow 2.4.2 with python 3.11 on a distributed containerized deployment , it work well -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #27264: Attempt to add Python 3.11 support
potiuk commented on PR #27264: URL: https://github.com/apache/airflow/pull/27264#issuecomment-1292300112 > i'm not sure via which route exactly `plyvel` ends up in your dependency chain, but i've published py311 wheels (linux only) just now; see [wbolster/plyvel#148 (comment)](https://github.com/wbolster/plyvel/issues/148#issuecomment-1292285729) It comes from Google Provider. It's a bit special (and we want to split it off eventually) but if it works then it's one less problem to worry about. Thanks for that! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #27264: Attempt to add Python 3.11 support
potiuk commented on PR #27264: URL: https://github.com/apache/airflow/pull/27264#issuecomment-1292298781 For those who are watching I made the CI images build. This is the first step that is needed. PyArrow was (of course) tip of the iceberg. But it FAR better than it was with Python 3.10 the day Python 3.10 was released (there almost all dependencies - including some core depednencies had problems). I think (let's see the tests) there are no problems with any of the dependencies that Airlfow Core has. Most of the problems come from our providers, which makes it viable to skip some providers from workng on 3.11- and we could release airflow without those providers working on 3.11. I had to disable quite a number of providers (8) - and big ones (Google, Snowflake, Azure among them). I also implemented a complete mechanism that will allow us in the future to decide whether we want to release Airflow with 3.11 even if some providers will not support it (basically excluding the providers in all the places/extras they were using. I opened (or commmented in existing ones) quite a number of issues in relevant dependencies that are blocking - some of them (like PyArrow) are quite close (they were working on their PR the whole day today). Some other might be tricky and we might not get it at all (for example cloudera "owns" but does not really support any changes in "sasl" library which is required for Hive Provider. They were in a process of handing the sasl library to some individual who voluntered to maintain it but I think it stallled so we might want to drop Hive (and possibly some other providers from 3.11 support). For Google Provider I belive we are the furthest. The only option to have 3.11 support will be to bump the old, pre-2.0.0 google-cloud libraries the Google Provider uses. This is lng overdue and it will hold us from releaseing Google Provider for 3.11 if not done. And we will be able to release 3.11 without Google Provider support for 3.11, but that woudl be a huge bummer IMHO. Now - the next will be tests failing in this PR, Apart of those tests that are relevant to the excluded providers, I am not going to look at them yet. And there might be other issues. And mybe fixing some dependencies will reveal some other issues. I am subscribed to the issues and will update and rebase the PR as the dependencies get updated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] wbolster commented on pull request #27264: Attempt to add Python 3.11 support
wbolster commented on PR #27264: URL: https://github.com/apache/airflow/pull/27264#issuecomment-1292297378 i'm not sure via which route exactly `plyvel` ends up in your dependency chain, but i've published py311 wheels (linux only) just now; see https://github.com/wbolster/plyvel/issues/148#issuecomment-1292285729 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch python-3-11 updated (d2d831713d -> cf340d758d)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch python-3-11 in repository https://gitbox.apache.org/repos/asf/airflow.git omit d2d831713d Attempt to add Python 3.11 support add 662a8aaac3 SFTP Provider: Fix default folder permissions (#26593) add 571ffeff7c Remove UI selective check output (#27254) add aa31110bd8 Add Andrey as member of the triage team (#27278) add 833a9ee748 Use all-python-versions-list-as-string in build-images.yml (#27271) add 3938b28ef2 Mark azure sdist warning as knowing when installing (#27258) add 7ad82e3384 Chart: Default airflow version to 2.4.2 (#27230) add 21cc5316be Update airflow release note to export AIRFLOW_RELEASE_SVN before use (#27253) add 0dd65b9cf6 pRint important information outside of the folded output in CI (#27260) add 104cd1ed76 Add logging back to Sagemaker system test (#27277) add d3691fdb63 Fix emr_eks system test (#27279) add c8b2737ab1 Enable string normalization for docs (#27269) add 7653c61df8 Make GSheetsHook return an empty list when there are no values (#27261) add d83a1aaa68 Add Blackify commits to blame ignore file (#27224) add b9c30b2e51 Sorting plugins custom menu links by category before name (#27152) add a2dd000bad Add Openverse to INTHEWILD.md (#27101) add e361be74cd Fix system test for Cloud Build Trigger (#26990) add 09c045f081 Bump Trino version to fix non-working DML queries (#27168) add c3940b40f8 timetable text clarifications (#27283) add ced2925fc3 Add dynamic attr check for ast.unparse method in pre-commit (#27280) add 6a1a6f7bef Adds s3_key_prefix to template fields (#27207) add a8a67a3f40 Added environment to templated SSHOperator fields (#26824) add d544e8fbeb Remove deprecated Tableau classes (#27288) add 777b57f0c6 Adding `preserve_file_name` param to `S3Hook.download_file` method (#26886) add 1af25809b1 Fix a typo in Breeze doc (#27293) add cf340d758d Attempt to add Python 3.11 support This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (d2d831713d) \ N -- N -- N refs/heads/python-3-11 (cf340d758d) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .asf.yaml | 1 + .git-blame-ignore-revs | 6 +- .github/workflows/build-images.yml | 2 +- .github/workflows/ci.yml | 1 - .pre-commit-config.yaml| 1 - BREEZE.rst | 2 +- INTHEWILD.md | 1 + RELEASE_NOTES.rst | 16 ++-- airflow/providers/amazon/aws/hooks/s3.py | 48 +-- .../amazon/aws/transfers/dynamodb_to_s3.py | 1 + airflow/providers/google/suite/hooks/sheets.py | 2 +- airflow/providers/sftp/hooks/sftp.py | 12 +-- airflow/providers/ssh/operators/ssh.py | 7 +- airflow/providers/tableau/CHANGELOG.rst| 8 ++ .../tableau/operators/tableau_refresh_workbook.py | 93 -- airflow/providers/tableau/provider.yaml| 2 - .../tableau/sensors/tableau_job_status.py | 28 --- airflow/providers/trino/provider.yaml | 2 +- airflow/utils/db.py| 4 +- airflow/www/extensions/init_views.py | 4 +- chart/Chart.yaml | 24 +++--- chart/newsfragments/27230.significant.rst | 3 + chart/values.schema.json | 4 +- chart/values.yaml | 4 +- dev/README_RELEASE_AIRFLOW.md | 3 +- dev/breeze/SELECTIVE_CHECKS.md | 1 - .../commands/release_management_commands.py| 9 ++- .../airflow_breeze/commands/testing_commands.py| 3 + dev/breeze/src/airflow_breeze/utils/parallel.py| 32 +++- .../src/airflow_breeze/utils/selective_checks.py | 10 --- .../operators/submit_run.rst | 16 ++-- docs/apache-airflow/best-practices.rst | 4 +- docs/apache-airflow/concepts/datasets.rst | 12 +-- docs/apache-airflow/concepts/timetable.rst | 9 ++- generated/provider_dependencies.json | 2 +-
[airflow] branch python-3-11 updated (c2dd81c063 -> d2d831713d)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch python-3-11 in repository https://gitbox.apache.org/repos/asf/airflow.git omit c2dd81c063 Attempt to add Python 3.11 support add d2d831713d Attempt to add Python 3.11 support This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (c2dd81c063) \ N -- N -- N refs/heads/python-3-11 (d2d831713d) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: CI.rst | 2 +- Dockerfile | 2 +- Dockerfile.ci | 2 +- airflow/providers/apache/beam/provider.yaml| 5 + airflow/providers/apache/hive/provider.yaml| 5 + airflow/providers/databricks/provider.yaml | 6 + airflow/providers/google/provider.yaml | 6 + airflow/providers/jdbc/provider.yaml | 5 + airflow/providers/microsoft/azure/provider.yaml| 8 + airflow/providers/papermill/provider.yaml | 7 + airflow/providers/snowflake/provider.yaml | 7 + generated/provider_dependencies.json | 230 ++--- .../pre_commit_build_providers_dependencies.py | 17 +- setup.py | 53 - 14 files changed, 264 insertions(+), 91 deletions(-)
[GitHub] [airflow] eitanme commented on pull request #27190: External task sensor fail fix
eitanme commented on PR #27190: URL: https://github.com/apache/airflow/pull/27190#issuecomment-1292237510 @o-nikolas I merged in main enabled pre-commit and updated my quoting style so hopefully that's the last of the linting though we'll see for sure after PR checks run. Thanks for the help and heads up that it had changed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] mobuchowski commented on a diff in pull request #27113: notification: add dag run state notification system
mobuchowski commented on code in PR #27113: URL: https://github.com/apache/airflow/pull/27113#discussion_r1005817540 ## airflow/jobs/scheduler_job.py: ## @@ -745,6 +757,9 @@ def _execute(self) -> None: self._run_scheduler_loop() +if self.enabled_dagrun_listener: Review Comment: We can just check for that in `notify_dagrun_state_changed`, and I changed it to just do that ## airflow/jobs/scheduler_job.py: ## @@ -1298,7 +1324,11 @@ def _schedule_dag_run( self._verify_integrity_if_dag_changed(dag_run=dag_run, session=session) # TODO[HA]: Rename update_state -> schedule_dag_run, ?? something else? -schedulable_tis, callback_to_run = dag_run.update_state(session=session, execute_callbacks=False) +schedulable_tis, callback_to_run = dag_run.update_state( +session=session, +execute_callbacks=False, +notification=self.notify_dagrun_state_changed if self.enabled_dagrun_listener else None, Review Comment: Done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #11085: Airflow Elasticsearch configuration log output does not contain required elements
potiuk commented on issue #11085: URL: https://github.com/apache/airflow/issues/11085#issuecomment-1292166867 A bit of context: Upgrading to latest versions is the first step to do, to and simplest way of diagnosing stuff quickly especially in case of old versions that had mutliple releases. Any way of fixing this problem (if it appears) will be only inplemented as latest version patchlevel (if found) so you will have to upgrade regardless. It's likely it's been already solved and by trying it out you can help yourself and others in solving it in "no time" without lengthy investitation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #11085: Airflow Elasticsearch configuration log output does not contain required elements
potiuk commented on issue #11085: URL: https://github.com/apache/airflow/issues/11085#issuecomment-1292163431 Did any try latest elasticsearch provider if it has the amse issue or upgrade airlfow to latest version to see if it resolves the problem? If not, can everyone who say "I have the same problem" try it ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] pankajastro commented on pull request #27236: Typecast biquery job response col value
pankajastro commented on PR #27236: URL: https://github.com/apache/airflow/pull/27236#issuecomment-1292162781 > Can you add unit test to avoid regression? > > BTW you can open PRs directly. No need to submit issue first Added tests, please let me know how it's looking now. While creating the issue by mistake I selected the area `core` can you please update the correct label remove `core` and add `provider` and `google`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #27297: Fix IntegrityError during webserver startup
potiuk commented on PR #27297: URL: https://github.com/apache/airflow/pull/27297#issuecomment-1292159846 Cool! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] aladinoss commented on issue #21867: Retry for TaskGroup
aladinoss commented on issue #21867: URL: https://github.com/apache/airflow/issues/21867#issuecomment-1292156893 Hello, This will be very interesting, retrie on taskgroup can make another dimension for workflow management. If we think on only data oriented workflows that may have no a lot of interest, today Airflow have a wider use cases that needs this kind of mechanism (IT-workflow, Infrastructure workflow, system and application auto-remediation ...) adding a retry and metadata (kind of xcom on the task group itself ?) can make a lot of developers happy. Regards, -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #27297: Fix IntegrityError during webserver startup
ephraimbuddy commented on PR #27297: URL: https://github.com/apache/airflow/pull/27297#issuecomment-1292153798 > No issues with plugins any more ? Yes. What is loaded at that point is web UI plugin. The `ensure_plugins_loaded` which calls the main airflow plugin is not loaded with the app. The problem before was whether it could be a problem with `lazy_load_plugin` config but it's not as only the web UI plugins are loaded during loading of the app: https://github.com/apache/airflow/blob/126b7b8a073f75096d24378ffd749ce166267826/airflow/www/extensions/init_views.py#L123-L147 The `lazy_load_plugin` config is called during startup https://github.com/apache/airflow/blob/126b7b8a073f75096d24378ffd749ce166267826/airflow/__init__.py#L86-L89 which loads the main airflow plugin(not just UI) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch constraints-main updated: Updating constraints. Build id:
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 95824cde9c Updating constraints. Build id: 95824cde9c is described below commit 95824cde9cd898c50baa03603a05968509cf0a38 Author: Automated GitHub Actions commit AuthorDate: Wed Oct 26 14:36:38 2022 + Updating constraints. Build id: This update in constraints is automatically committed by the CI 'constraints-push' step based on HEAD of '' in '' with commit sha . All tests passed in this build so we determined we can push the updated constraints. See https://github.com/apache/airflow/blob/main/README.md#installing-from-pypi for details. --- constraints-3.10.txt | 4 ++-- constraints-3.7.txt | 4 ++-- constraints-3.8.txt | 4 ++-- constraints-3.9.txt | 4 ++-- constraints-no-providers-3.10.txt | 4 ++-- constraints-no-providers-3.7.txt | 4 ++-- constraints-no-providers-3.8.txt | 4 ++-- constraints-no-providers-3.9.txt | 4 ++-- constraints-source-providers-3.10.txt | 2 +- constraints-source-providers-3.7.txt | 2 +- constraints-source-providers-3.8.txt | 2 +- constraints-source-providers-3.9.txt | 2 +- 12 files changed, 20 insertions(+), 20 deletions(-) diff --git a/constraints-3.10.txt b/constraints-3.10.txt index b703f28a3f..2c083a6a40 100644 --- a/constraints-3.10.txt +++ b/constraints-3.10.txt @@ -1,5 +1,5 @@ # -# This constraints file was automatically generated on 2022-10-26T12:12:48Z +# This constraints file was automatically generated on 2022-10-26T14:35:45Z # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow. # This variant of constraints install uses the HEAD of the branch version for 'apache-airflow' but installs # the providers from PIP-released packages at the moment of the constraint generation. @@ -358,7 +358,7 @@ kombu==5.2.4 krb5==0.4.1 kubernetes==23.6.0 kylinpy==2.8.4 -lazy-object-proxy==1.7.1 +lazy-object-proxy==1.8.0 ldap3==2.9.1 linkify-it-py==2.0.0 locket==1.0.0 diff --git a/constraints-3.7.txt b/constraints-3.7.txt index 2336cbe550..7e37d95ca4 100644 --- a/constraints-3.7.txt +++ b/constraints-3.7.txt @@ -1,5 +1,5 @@ # -# This constraints file was automatically generated on 2022-10-26T12:13:12Z +# This constraints file was automatically generated on 2022-10-26T14:36:35Z # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow. # This variant of constraints install uses the HEAD of the branch version for 'apache-airflow' but installs # the providers from PIP-released packages at the moment of the constraint generation. @@ -358,7 +358,7 @@ kombu==5.2.4 krb5==0.4.1 kubernetes==23.6.0 kylinpy==2.8.4 -lazy-object-proxy==1.7.1 +lazy-object-proxy==1.8.0 ldap3==2.9.1 linkify-it-py==2.0.0 locket==1.0.0 diff --git a/constraints-3.8.txt b/constraints-3.8.txt index 7e26b05b8e..430c1279b1 100644 --- a/constraints-3.8.txt +++ b/constraints-3.8.txt @@ -1,5 +1,5 @@ # -# This constraints file was automatically generated on 2022-10-26T12:13:05Z +# This constraints file was automatically generated on 2022-10-26T14:36:21Z # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow. # This variant of constraints install uses the HEAD of the branch version for 'apache-airflow' but installs # the providers from PIP-released packages at the moment of the constraint generation. @@ -360,7 +360,7 @@ kombu==5.2.4 krb5==0.4.1 kubernetes==23.6.0 kylinpy==2.8.4 -lazy-object-proxy==1.7.1 +lazy-object-proxy==1.8.0 ldap3==2.9.1 linkify-it-py==2.0.0 locket==1.0.0 diff --git a/constraints-3.9.txt b/constraints-3.9.txt index 909715a3af..2165be6ef7 100644 --- a/constraints-3.9.txt +++ b/constraints-3.9.txt @@ -1,5 +1,5 @@ # -# This constraints file was automatically generated on 2022-10-26T12:13:02Z +# This constraints file was automatically generated on 2022-10-26T14:36:18Z # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow. # This variant of constraints install uses the HEAD of the branch version for 'apache-airflow' but installs # the providers from PIP-released packages at the moment of the constraint generation. @@ -358,7 +358,7 @@ kombu==5.2.4 krb5==0.4.1 kubernetes==23.6.0 kylinpy==2.8.4 -lazy-object-proxy==1.7.1 +lazy-object-proxy==1.8.0 ldap3==2.9.1 linkify-it-py==2.0.0 locket==1.0.0 diff --git a/constraints-no-providers-3.10.txt b/constraints-no-providers-3.10.txt index d8c13fdc15..7a799efb53 100644 --- a/constraints-no-providers-3.10.txt +++ b/constraints-no-providers-3.10.txt @@ -1,5 +1,5 @@ # -# This constraints file was automatically generated on 2022-10-26T12:10:51Z +# This constraints file was automatically generated on 2022-10-26T14:32:35Z # via "eager-upgrade" mechan
[GitHub] [airflow] sfc-gh-atatara commented on pull request #26739: Added Istio support to KubernetesPodOperator
sfc-gh-atatara commented on PR #26739: URL: https://github.com/apache/airflow/pull/26739#issuecomment-1292148085 it is actioned as single operator now -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
