Re: [PR] Add support for service account impersonation with computeEngineSSHHook (google provider) and IAP tunnel [airflow]

2023-11-23 Thread via GitHub
ginolegigot commented on PR #35136: URL: https://github.com/apache/airflow/pull/35136#issuecomment-1823990626 Hello @potiuk and @Taragolis! May i have a review ? Or call someone who can do it ? -- This is an automated message from the Apache Git Service. To respond to the message, pleas

Re: [PR] Consolidate environment variable calculation in ShellParams in Breeze [airflow]

2023-11-23 Thread via GitHub
potiuk commented on PR #35801: URL: https://github.com/apache/airflow/pull/35801#issuecomment-1824049973 Actually - I made it even better (see the latest version). I also figured how to do it so that we do not have to keep the separate lists of variables to pass in neither docker-compose ba

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2023-11-23 Thread via GitHub
hamedhsn commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1824067974 @bolkedebruin build is green now -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

[I] Add path to airflow.cfg in docker-compose.yml [airflow]

2023-11-23 Thread via GitHub
svvema opened a new issue, #35812: URL: https://github.com/apache/airflow/issues/35812 ### Description Adding a commented line in compose file like `- ${AIRFLOW_PROJ_DIR:-.}/airflow.cfg:/opt/airflow/airflow.cfg ` would save new users tons of time when customizing the configuration fi

Re: [I] Add path to airflow.cfg in docker-compose.yml [airflow]

2023-11-23 Thread via GitHub
boring-cyborg[bot] commented on issue #35812: URL: https://github.com/apache/airflow/issues/35812#issuecomment-1824107782 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for ap

Re: [PR] Make core compatible with `pendulum` 3 [airflow]

2023-11-23 Thread via GitHub
Taragolis commented on PR #34744: URL: https://github.com/apache/airflow/pull/34744#issuecomment-1824170057 closed in favor of #35798 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Make core compatible with `pendulum` 3 [airflow]

2023-11-23 Thread via GitHub
Taragolis closed pull request #34744: Make core compatible with `pendulum` 3 URL: https://github.com/apache/airflow/pull/34744 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2023-11-23 Thread via GitHub
bolkedebruin commented on code in PR #22253: URL: https://github.com/apache/airflow/pull/22253#discussion_r1403221124 ## airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py: ## @@ -17,173 +17,271 @@ # under the License. from __future__ import annotations -import

Re: [I] Add path to airflow.cfg in docker-compose.yml [airflow]

2023-11-23 Thread via GitHub
Taragolis commented on issue #35812: URL: https://github.com/apache/airflow/issues/35812#issuecomment-1824228946 docker-compose sample is not a production level example. It just exists for simple purpose "Quick run and say 'Whoa! it actually works!'" however configuration for production mig

Re: [I] Add path to airflow.cfg in docker-compose.yml [airflow]

2023-11-23 Thread via GitHub
Taragolis commented on issue #35812: URL: https://github.com/apache/airflow/issues/35812#issuecomment-1824234548 > Also the current default bind - ${AIRFLOW_PROJ_DIR:-.}/config:/opt/airflow/config is misleading where file airflow.cfg should be stored in the container. It might be sto

[PR] add guideline for missing image when running breeze [airflow]

2023-11-23 Thread via GitHub
Lee-W opened a new pull request, #35813: URL: https://github.com/apache/airflow/pull/35813 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**

Re: [PR] add guideline for missing image when running breeze [airflow]

2023-11-23 Thread via GitHub
pankajkoti commented on code in PR #35813: URL: https://github.com/apache/airflow/pull/35813#discussion_r1403251382 ## BREEZE.rst: ## @@ -949,6 +949,23 @@ In this case, disabling IPv6 in the host machine and using IPv4 instead resolved The similar issue could happen if you are

Re: [PR] add guideline for missing image when running breeze [airflow]

2023-11-23 Thread via GitHub
phanikumv commented on code in PR #35813: URL: https://github.com/apache/airflow/pull/35813#discussion_r1403253983 ## BREEZE.rst: ## @@ -949,6 +949,23 @@ In this case, disabling IPv6 in the host machine and using IPv4 instead resolved The similar issue could happen if you are

Re: [PR] add guideline for missing image when running breeze [airflow]

2023-11-23 Thread via GitHub
Lee-W commented on code in PR #35813: URL: https://github.com/apache/airflow/pull/35813#discussion_r1403255497 ## BREEZE.rst: ## @@ -949,6 +949,23 @@ In this case, disabling IPv6 in the host machine and using IPv4 instead resolved The similar issue could happen if you are behi

Re: [PR] add guideline for missing image when running breeze [airflow]

2023-11-23 Thread via GitHub
Lee-W commented on code in PR #35813: URL: https://github.com/apache/airflow/pull/35813#discussion_r1403255497 ## BREEZE.rst: ## @@ -949,6 +949,23 @@ In this case, disabling IPv6 in the host machine and using IPv4 instead resolved The similar issue could happen if you are behi

Re: [PR] add guideline for missing image when running breeze [airflow]

2023-11-23 Thread via GitHub
Lee-W commented on code in PR #35813: URL: https://github.com/apache/airflow/pull/35813#discussion_r1403256063 ## BREEZE.rst: ## @@ -949,6 +949,23 @@ In this case, disabling IPv6 in the host machine and using IPv4 instead resolved The similar issue could happen if you are behi

Re: [PR] add guideline for missing image when running breeze [airflow]

2023-11-23 Thread via GitHub
Taragolis commented on PR #35813: URL: https://github.com/apache/airflow/pull/35813#issuecomment-1824276280 `No such image` Error also could be resolve by run one of this command, both of them also support `--python` option ```console breeze ci-image pull ``` or ```co

Re: [PR] add guideline for missing image when running breeze [airflow]

2023-11-23 Thread via GitHub
potiuk commented on PR #35813: URL: https://github.com/apache/airflow/pull/35813#issuecomment-1824305316 Actually this is not the right solution. The problem is that image should be automatically build if it is missing, there should never be the need to pull it manually. Likely this

Re: [PR] Fix race condition in KubernetesExecutor with concurrently running schedulers [airflow]

2023-11-23 Thread via GitHub
droppoint commented on PR #35800: URL: https://github.com/apache/airflow/pull/35800#issuecomment-1824311383 The completed pods are not a problem because they consume no resources, and they will be deleted during the airflow cleanup-pods cronjob execution. However, a TaskInstance can get stu

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2023-11-23 Thread via GitHub
hamedhsn commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1824328879 @bolkedebruin I will fix your above comment. What is the best place to put some usage examples, can you point me to the location? -- This is an automated message from the Apache Git

Re: [I] apache-airflow-providers-amazon [airflow]

2023-11-23 Thread via GitHub
Taragolis commented on issue #35805: URL: https://github.com/apache/airflow/issues/35805#issuecomment-1824329980 @scnerd would you change issue title to less broad and which could describe your problem in short --- In addition I have a look `boto3` documentation for [`Redshift

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2023-11-23 Thread via GitHub
hamedhsn commented on code in PR #22253: URL: https://github.com/apache/airflow/pull/22253#discussion_r1403314801 ## airflow/providers/cncf/kubernetes/operators/custom_object_launcher.py: ## @@ -0,0 +1,366 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or mo

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2023-11-23 Thread via GitHub
hamedhsn commented on code in PR #22253: URL: https://github.com/apache/airflow/pull/22253#discussion_r1403317000 ## airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py: ## @@ -17,173 +17,271 @@ # under the License. from __future__ import annotations -import date

[I] git-sync-init resources is too indented [airflow]

2023-11-23 Thread via GitHub
slmg opened a new issue, #35815: URL: https://github.com/apache/airflow/issues/35815 ### Official Helm Chart version 1.10.0 ### Apache Airflow version 2.6.3 ### Kubernetes Version 1.27.7 ### Helm Chart configuration values.yaml ```yaml

Re: [PR] fix(helm): add recreate strategy support [airflow]

2023-11-23 Thread via GitHub
hakuno commented on code in PR #35362: URL: https://github.com/apache/airflow/pull/35362#discussion_r1403329260 ## chart/templates/_helpers.yaml: ## @@ -995,3 +995,11 @@ capabilities: app: keda-operator {{- end }} {{- end }} + +{{- define "deployment_strategy" }

Re: [PR] fix(helm): add recreate strategy support [airflow]

2023-11-23 Thread via GitHub
hakuno commented on code in PR #35362: URL: https://github.com/apache/airflow/pull/35362#discussion_r1403329260 ## chart/templates/_helpers.yaml: ## @@ -995,3 +995,11 @@ capabilities: app: keda-operator {{- end }} {{- end }} + +{{- define "deployment_strategy" }

[I] Airflow StatsD Configuration Error: XML 'ResourceNotFound' Message for statsd_host Environment Variable [airflow]

2023-11-23 Thread via GitHub
jordi-crespo opened a new issue, #35817: URL: https://github.com/apache/airflow/issues/35817 ### Apache Airflow version Other Airflow 2 version (please specify below) ### What happened airflow_version: 2.6.2 Deployed in: Azure Kubernetes Service I am encounterin

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2023-11-23 Thread via GitHub
bolkedebruin commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1824404326 > @bolkedebruin I will fix your above comment. What is the best place to put some usage examples, can you point me to the location? I'd like to see documentation here as a se

[PR] Create directories based on `AIRFLOW_CONFIG` path [airflow]

2023-11-23 Thread via GitHub
Taragolis opened a new pull request, #35818: URL: https://github.com/apache/airflow/pull/35818 Currently `write_default_airflow_configuration_if_needed` create Airflow Home directory, which should work in case if user doesn't redefine `AIRFLOW_CONFIG` environment variable.

[PR] Add OpenLineage support to S3FileTransformOperator [airflow]

2023-11-23 Thread via GitHub
kacpermuda opened a new pull request, #35819: URL: https://github.com/apache/airflow/pull/35819 Adding OpenLineage support for the following operators: - S3FileTransformOperator --- **^ Add meaningful description above** Read the **[Pull Request Guidelines

[PR] Allow storage options to be passed [airflow]

2023-11-23 Thread via GitHub
bolkedebruin opened a new pull request, #35820: URL: https://github.com/apache/airflow/pull/35820 This allows storage options to be passed on to the underlying implementation. It does require a release of the providers as the signature of `get_fs` has changed. @kaxil @uranusjr @Tarag

(airflow) branch main updated: Consolidate environment variable calculation in ShellParams in Breeze (#35801)

2023-11-23 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 910c95ea64 Consolidate environment variable calcula

Re: [PR] Consolidate environment variable calculation in ShellParams in Breeze [airflow]

2023-11-23 Thread via GitHub
potiuk merged PR #35801: URL: https://github.com/apache/airflow/pull/35801 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[PR] Use `OpenSearch` instead of `Open Search` and `Opensearch` [airflow]

2023-11-23 Thread via GitHub
Taragolis opened a new pull request, #35821: URL: https://github.com/apache/airflow/pull/35821 Use same name of product as it mentioned in the official page: https://opensearch.org/ --- **^ Add meaningful description above** Read the **[Pull Request Guidelin

Re: [PR] add guideline for missing image when running breeze [airflow]

2023-11-23 Thread via GitHub
Lee-W commented on PR #35813: URL: https://github.com/apache/airflow/pull/35813#issuecomment-1824629892 > Was there any special you've done (cleaning docker etc.? @Lee-W - can you come up withe a scenario where this could happen ? I might try to reproduce it, but if you more ore less know h

Re: [PR] Allow storage options to be passed [airflow]

2023-11-23 Thread via GitHub
Taragolis commented on code in PR #35820: URL: https://github.com/apache/airflow/pull/35820#discussion_r1403531115 ## airflow/providers/amazon/aws/fs/s3.py: ## @@ -45,7 +45,7 @@ class SignError(Exception): """Raises when unable to sign a S3 request.""" -def get_fs(conn_

[PR] Revert Remove PodLoggingStatus object #35422 [airflow]

2023-11-23 Thread via GitHub
dstandish opened a new pull request, #35822: URL: https://github.com/apache/airflow/pull/35822 This object was completely unused in OSS but others may have depended on it and it is kinder to remove it in a major release. -- This is an automated message from the Apache Git Service. To

Re: [PR] Fix race condition in KubernetesExecutor with concurrently running schedulers [airflow]

2023-11-23 Thread via GitHub
dstandish commented on PR #35800: URL: https://github.com/apache/airflow/pull/35800#issuecomment-1824655540 In case it is important to adopt pods, but we need a fix to do it safely, just throwing out some ideas before i disappear for holiday. The scheduler can know which other schedul

Re: [PR] add guideline for missing image when running breeze [airflow]

2023-11-23 Thread via GitHub
potiuk commented on PR #35813: URL: https://github.com/apache/airflow/pull/35813#issuecomment-1824658121 Ok. I think (maybe you can check) just adding check_and_rebuild_image_if_needed() when the start command run should solve the problem. It I lis for sure done in shell command and default

Re: [PR] Fix race condition in KubernetesExecutor with concurrently running schedulers [airflow]

2023-11-23 Thread via GitHub
JCoder01 commented on code in PR #35800: URL: https://github.com/apache/airflow/pull/35800#discussion_r1403249411 ## airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py: ## @@ -642,39 +641,6 @@ def adopt_launched_task( del tis_to_flush_by_key[ti_key]

(airflow) branch main updated: Add OpenLineage support to `S3FileTransformOperator` (#35819)

2023-11-23 Thread vincbeck
This is an automated email from the ASF dual-hosted git repository. vincbeck pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new ef2ad070c2 Add OpenLineage support to `S3FileTran

Re: [PR] Add OpenLineage support to S3FileTransformOperator [airflow]

2023-11-23 Thread via GitHub
vincbeck merged PR #35819: URL: https://github.com/apache/airflow/pull/35819 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

Re: [I] execution_timeout not enforced, task hangs up [airflow]

2023-11-23 Thread via GitHub
xmariachi commented on issue #35474: URL: https://github.com/apache/airflow/issues/35474#issuecomment-1824683187 > @xmariachi Hi, I also use mwaa and have come across this behaviour. What is the status of the task when it gets stuck? We had the task in a queued state, and noticed the sqs ag

[I] TypedDict for Context should mark Required/NotRequired attributes correctly [airflow]

2023-11-23 Thread via GitHub
hterik opened a new issue, #35823: URL: https://github.com/apache/airflow/issues/35823 ### Description Right now the definition for `Context` is a `TypedDict` with `total=False`. https://github.com/apache/airflow/blob/910c95ea64b6e14bff842a031e019c95f2ed/airflow/utils/context.p

(airflow) branch constraints-main updated: Updating constraints. Github run id:6972155622

2023-11-23 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new f8b9239e57 Updating con

Re: [PR] Revert Remove PodLoggingStatus object #35422 [airflow]

2023-11-23 Thread via GitHub
pankajkoti merged PR #35822: URL: https://github.com/apache/airflow/pull/35822 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airfl

(airflow) branch main updated (ef2ad070c2 -> ca97feed18)

2023-11-23 Thread pankajkoti
This is an automated email from the ASF dual-hosted git repository. pankajkoti pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from ef2ad070c2 Add OpenLineage support to `S3FileTransformOperator` (#35819) add ca97feed18 Revert Remove PodLogg

[PR] change indent to 4 [airflow]

2023-11-23 Thread via GitHub
romsharon98 opened a new pull request, #35824: URL: https://github.com/apache/airflow/pull/35824 Fixes: https://github.com/apache/airflow/issues/35815 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/a

Re: [I] TypedDict for Context should mark Required/NotRequired attributes correctly [airflow]

2023-11-23 Thread via GitHub
Taragolis commented on issue #35823: URL: https://github.com/apache/airflow/issues/35823#issuecomment-1824768315 > The addition of total=False was done in following PR and seems very unrelated to the change at hand there. Maybe it was a mistake? I think this was added due to deprecate

Re: [PR] Add EC2HibernateInstanceOperator and EC2RebootInstanceOperator [airflow]

2023-11-23 Thread via GitHub
vincbeck merged PR #35790: URL: https://github.com/apache/airflow/pull/35790 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

Re: [I] Add EC2RebootInstanceOperator and EC2HibernateInstanceOperator to Amazon Provider [airflow]

2023-11-23 Thread via GitHub
vincbeck closed issue #35703: Add EC2RebootInstanceOperator and EC2HibernateInstanceOperator to Amazon Provider URL: https://github.com/apache/airflow/issues/35703 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Use `OpenSearch` instead of `Open Search` and `Opensearch` [airflow]

2023-11-23 Thread via GitHub
vincbeck merged PR #35821: URL: https://github.com/apache/airflow/pull/35821 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

(airflow) branch main updated: Add `EC2HibernateInstanceOperator` and `EC2RebootInstanceOperator` (#35790)

2023-11-23 Thread vincbeck
This is an automated email from the ASF dual-hosted git repository. vincbeck pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new ca1202fd31 Add `EC2HibernateInstanceOperator` and

(airflow) branch main updated: Use `OpenSearch` instead of `Open Search` and `Opensearch` (#35821)

2023-11-23 Thread vincbeck
This is an automated email from the ASF dual-hosted git repository. vincbeck pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new eb691fc013 Use `OpenSearch` instead of `Open Sear

Re: [PR] Allow storage options to be passed [airflow]

2023-11-23 Thread via GitHub
bolkedebruin commented on code in PR #35820: URL: https://github.com/apache/airflow/pull/35820#discussion_r1403643831 ## airflow/providers/google/cloud/fs/gcs.py: ## @@ -39,7 +39,7 @@ schemes = ["gs", "gcs"] -def get_fs(conn_id: str | None) -> AbstractFileSystem: +def get_f

Re: [I] TypedDict for Context should mark Required/NotRequired attributes correctly [airflow]

2023-11-23 Thread via GitHub
ShivamPatel-24 commented on issue #35823: URL: https://github.com/apache/airflow/issues/35823#issuecomment-1824789835 Hi @Taragolis, I am looking to contribute, but am new to it. Can you guide me through what needs to be done to resolve this issue? -- This is an automated message f

Re: [PR] Fix K8S executor override config using pod_override_object [airflow]

2023-11-23 Thread via GitHub
hussein-awala commented on code in PR #35185: URL: https://github.com/apache/airflow/pull/35185#discussion_r1403682530 ## dev/breeze/src/airflow_breeze/commands/kubernetes_commands.py: ## @@ -1036,8 +1047,47 @@ def _deploy_airflow( executor=executor, use_standa

Re: [PR] Implement `is_authorized_variable` in AWS auth manager [airflow]

2023-11-23 Thread via GitHub
o-nikolas commented on code in PR #35804: URL: https://github.com/apache/airflow/pull/35804#discussion_r1403683906 ## airflow/providers/amazon/aws/auth_manager/avp/facade.py: ## @@ -0,0 +1,125 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributo

[PR] Enhance docs for zombie tasks [airflow]

2023-11-23 Thread via GitHub
pankajkoti opened a new pull request, #35825: URL: https://github.com/apache/airflow/pull/35825 closes: #35698 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rs

Re: [PR] Enhance docs for zombie tasks [airflow]

2023-11-23 Thread via GitHub
pankajkoti commented on PR #35825: URL: https://github.com/apache/airflow/pull/35825#issuecomment-1824864322 Preview of the locally built docs: ![screencapture-localhost-63342-airflow-docs-build-docs-apache-airflow-stable-core-concepts-tasks-html-2023-11-24-01_23_13 (1)](https://git

Re: [PR] Enhance docs for zombie tasks [airflow]

2023-11-23 Thread via GitHub
pankajkoti commented on PR #35825: URL: https://github.com/apache/airflow/pull/35825#issuecomment-1824867590 cc: @vatsrahul1001 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] Enhance docs for zombie tasks [airflow]

2023-11-23 Thread via GitHub
pankajkoti closed pull request #35825: Enhance docs for zombie tasks URL: https://github.com/apache/airflow/pull/35825 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

Re: [PR] Implements JSON-string connection representation generator [airflow]

2023-11-23 Thread via GitHub
Taragolis merged PR #35723: URL: https://github.com/apache/airflow/pull/35723 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflo

(airflow) branch main updated: Implements JSON-string connection representation generator (#35723)

2023-11-23 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new b07d79908c Implements JSON-string connection rep

Re: [PR] Fix K8S executor override config using pod_override_object [airflow]

2023-11-23 Thread via GitHub
potiuk commented on PR #35185: URL: https://github.com/apache/airflow/pull/35185#issuecomment-1824872772 I guess later we can add multi-namespace mode to run in CI. We likely do not want to add another dimention for k8s tests but we can do what we do in databases with some kind of rotating

[PR] breeze - docs added pipx python flag [airflow]

2023-11-23 Thread via GitHub
pateash opened a new pull request, #35827: URL: https://github.com/apache/airflow/pull/35827 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

Re: [PR] breeze - docs adding pipx python flag [airflow]

2023-11-23 Thread via GitHub
potiuk merged PR #35827: URL: https://github.com/apache/airflow/pull/35827 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

(airflow) branch main updated: breeze - docs added pipx python flag (#35827)

2023-11-23 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new afe14ceb62 breeze - docs added pipx python flag (#3

Re: [I] Airflow commands should not show sensitive information by default [airflow]

2023-11-23 Thread via GitHub
pateash commented on issue #12696: URL: https://github.com/apache/airflow/issues/12696#issuecomment-1824877991 this is already done using --anonymize so should be closed @turbaszek https://github.com/apache/airflow/assets/16856802/02a2a8a7-130c-487e-bec7-226acdb7baa3";> vs

Re: [PR] Fix K8S executor override config using pod_override_object [airflow]

2023-11-23 Thread via GitHub
hussein-awala commented on PR #35185: URL: https://github.com/apache/airflow/pull/35185#issuecomment-182483 > I guess later we can add multi-namespace mode to run in CI. We likely do not want to add another dimention for k8s tests but we can do what we do in databases with some kind of

(airflow) branch main updated: Fix K8S executor override config using pod_override_object (#35185)

2023-11-23 Thread husseinawala
This is an automated email from the ASF dual-hosted git repository. husseinawala pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 0e157b38a3 Fix K8S executor override config u

Re: [PR] Fix K8S executor override config using pod_override_object [airflow]

2023-11-23 Thread via GitHub
hussein-awala commented on PR #35185: URL: https://github.com/apache/airflow/pull/35185#issuecomment-1824891188 Finally 🎉 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [PR] Fix K8S executor override config using pod_override_object [airflow]

2023-11-23 Thread via GitHub
hussein-awala merged PR #35185: URL: https://github.com/apache/airflow/pull/35185 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@ai

[PR] Increase `run-complete-tests` parallelism to 4 which is the combinations count [airflow]

2023-11-23 Thread via GitHub
hussein-awala opened a new pull request, #35828: URL: https://github.com/apache/airflow/pull/35828 Currently, the test runs with 3 parallel processes, but there are 4 tests. The last test starts when the 3 other tests finish, which leads to resources and time wasting. Running the 4 tests in

Re: [PR] Increase `run-complete-tests` parallelism to 4 which is the combinations count [airflow]

2023-11-23 Thread via GitHub
potiuk commented on PR #35828: URL: https://github.com/apache/airflow/pull/35828#issuecomment-1824908372 Few comments: Let's see. Tried that before but it was introducing a lot of random failures (likely 4 k8s simply use too much memory). Likely we could think about switching a

Re: [I] Wrong execution date in SLA alerts [airflow]

2023-11-23 Thread via GitHub
pateash commented on issue #35415: URL: https://github.com/apache/airflow/issues/35415#issuecomment-1824923428 hi @hliu-VoleonGroup this logicalDate represents(logical date of next dagrun) or the Data interval end time ( when SLA was fired ). https://github.com/apache/airflow/ass

Re: [PR] Increase `run-complete-tests` parallelism to 4 which is the combinations count [airflow]

2023-11-23 Thread via GitHub
hussein-awala commented on PR #35828: URL: https://github.com/apache/airflow/pull/35828#issuecomment-1824939251 > doing it this way is good for experimenting, but not good for merging Yes, it's just to test, then we can discuss the implementation if we find some benefits. -- This

Re: [PR] Implement `is_authorized_variable` in AWS auth manager [airflow]

2023-11-23 Thread via GitHub
vincbeck commented on code in PR #35804: URL: https://github.com/apache/airflow/pull/35804#discussion_r1403761025 ## airflow/providers/amazon/aws/auth_manager/avp/facade.py: ## @@ -0,0 +1,125 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor

Re: [I] Airflow StatsD Configuration Error: XML 'ResourceNotFound' Message for statsd_host Environment Variable [airflow]

2023-11-23 Thread via GitHub
jscheffl commented on issue #35817: URL: https://github.com/apache/airflow/issues/35817#issuecomment-1824963562 Reading this I can confirm that statsd is working in our setup in K8s with the standard helm chart very stable sine ~1 year at least. Am I correct to assume that your statsd con

Re: [I] triggerer block_watchdog is crying "wolf!" [airflow]

2023-11-23 Thread via GitHub
jscheffl commented on issue #35806: URL: https://github.com/apache/airflow/issues/35806#issuecomment-1824972361 Thanks for the analysis and the details shared! This helps understanding your concerns directly. Regarding (1): I underatand Regarding (2) I believe the standard in a

Re: [PR] Increase `run-complete-tests` parallelism to 4 which is the combinations count [airflow]

2023-11-23 Thread via GitHub
potiuk commented on PR #35828: URL: https://github.com/apache/airflow/pull/35828#issuecomment-1824976608 > > doing it this way is good for experimenting, but not good for merging > > Yes, it's just to test, then we can discuss the implementation if we find some benefits. And ye

Re: [PR] Increase `run-complete-tests` parallelism to 4 which is the combinations count [airflow]

2023-11-23 Thread via GitHub
potiuk commented on PR #35828: URL: https://github.com/apache/airflow/pull/35828#issuecomment-1824976961 Yeah... same case - a lot of "connection reset by peer" - which likely means killed by lack of memory. -- This is an automated message from the Apache Git Service. To respond to the me

Re: [PR] Add EC2HibernateInstanceOperator and EC2RebootInstanceOperator [airflow]

2023-11-23 Thread via GitHub
ferruzzi commented on PR #35790: URL: https://github.com/apache/airflow/pull/35790#issuecomment-1825004349 This merge broke the system tests; t4g.micro instances can not be hibernated. Did you test this out? If so, I don't suppose you remember which instance size worked for you? `

Re: [PR] Add EC2HibernateInstanceOperator and EC2RebootInstanceOperator [airflow]

2023-11-23 Thread via GitHub
ferruzzi commented on PR #35790: URL: https://github.com/apache/airflow/pull/35790#issuecomment-1825009160 Looking through the code, it looks like you are checking for a hibernation option, but the exception you are raising `raise AirflowException(f"Instance {instance['InstanceId']} is not

Re: [I] Marking task as "up_for_retry" when it is still running [airflow]

2023-11-23 Thread via GitHub
github-actions[bot] commented on issue #35195: URL: https://github.com/apache/airflow/issues/35195#issuecomment-1825012227 This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further

Re: [I] Google Vertex AI ListEndpointsOperator fails with `TypeError: is not an instance of Endpoint` when listing Endpoints [airflow]

2023-11-23 Thread via GitHub
github-actions[bot] commented on issue #34451: URL: https://github.com/apache/airflow/issues/34451#issuecomment-1825012274 This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further

Re: [PR] Pass parameters to HttpHook's auth_type from Connection [airflow]

2023-11-23 Thread via GitHub
Joffreybvn commented on PR #35591: URL: https://github.com/apache/airflow/pull/35591#issuecomment-1825016420 Thanks for the detailed answer ! I won't go for proxy settings and extra parameters in this PR. Just for info, adding a tool like [forwarder](https://github.com/saucelabs/forwa

Re: [PR] Pass parameters to HttpHook's auth_type from Connection [airflow]

2023-11-23 Thread via GitHub
Joffreybvn commented on code in PR #35591: URL: https://github.com/apache/airflow/pull/35591#discussion_r1403798873 ## airflow/providers/http/hooks/http.py: ## @@ -71,20 +71,13 @@ def __init__( self.method = method.upper() self.base_url: str = "" self.

Re: [PR] Pass parameters to HttpHook's auth_type from Connection [airflow]

2023-11-23 Thread via GitHub
Joffreybvn commented on code in PR #35591: URL: https://github.com/apache/airflow/pull/35591#discussion_r1403798873 ## airflow/providers/http/hooks/http.py: ## @@ -71,20 +71,13 @@ def __init__( self.method = method.upper() self.base_url: str = "" self.

Re: [PR] Add EC2HibernateInstanceOperator and EC2RebootInstanceOperator [airflow]

2023-11-23 Thread via GitHub
dashton90 commented on PR #35790: URL: https://github.com/apache/airflow/pull/35790#issuecomment-1825021497 I tested on an m4.large instance And didn't come across as harsh at all. Please let me know how I can assist with the system tests. -- This is an automated message from the A

Re: [PR] Add EC2HibernateInstanceOperator and EC2RebootInstanceOperator [airflow]

2023-11-23 Thread via GitHub
dashton90 commented on PR #35790: URL: https://github.com/apache/airflow/pull/35790#issuecomment-1825022256 As a separate note, are these tests available publicly? Would be happy to follow any additional steps before opening a PR in the future -- This is an automated message from the Apac

Re: [PR] Add EC2HibernateInstanceOperator and EC2RebootInstanceOperator [airflow]

2023-11-23 Thread via GitHub
ferruzzi commented on PR #35790: URL: https://github.com/apache/airflow/pull/35790#issuecomment-1825024215 What do you think about the unit tests? Do they need to be tweaked? > I tested on an m4.large instance Cool. I'll see if I can find documentation around which sizes DO w

Re: [PR] Add EC2HibernateInstanceOperator and EC2RebootInstanceOperator [airflow]

2023-11-23 Thread via GitHub
dashton90 commented on PR #35790: URL: https://github.com/apache/airflow/pull/35790#issuecomment-1825027155 > Cool. I'll see if I can find documentation around which sizes DO work, but at least we have a place to start now. The supported instances are listed [here](https://docs.aws.a

Re: [PR] Add EC2HibernateInstanceOperator and EC2RebootInstanceOperator [airflow]

2023-11-23 Thread via GitHub
ferruzzi commented on PR #35790: URL: https://github.com/apache/airflow/pull/35790#issuecomment-1825031115 > The supported instances are listed [here](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/hibernating-prerequisites.html#hibernation-prereqs-supported-instance-families) Yo

Re: [PR] Add EC2HibernateInstanceOperator and EC2RebootInstanceOperator [airflow]

2023-11-23 Thread via GitHub
dashton90 commented on PR #35790: URL: https://github.com/apache/airflow/pull/35790#issuecomment-1825031778 Just re-reading through your comments: > Looking through the code, it looks like you are checking for a hibernation option, but the exception you are raising raise AirflowExcept

Re: [PR] Add EC2HibernateInstanceOperator and EC2RebootInstanceOperator [airflow]

2023-11-23 Thread via GitHub
ferruzzi commented on PR #35790: URL: https://github.com/apache/airflow/pull/35790#issuecomment-1825035261 > If you'd like, I can update the example to use an m4 or t3 instance. I'm running it against my live AWS account. It is going need a few other changes, too. The AMI architect

Re: [PR] Add EC2HibernateInstanceOperator and EC2RebootInstanceOperator [airflow]

2023-11-23 Thread via GitHub
ferruzzi commented on PR #35790: URL: https://github.com/apache/airflow/pull/35790#issuecomment-1825035554 > The example_ec2 DAG would never get to this operator since it is failing during instance creation. Yup, good catch. My mistake. -- This is an automated message from the Apa

Re: [PR] Add EC2HibernateInstanceOperator and EC2RebootInstanceOperator [airflow]

2023-11-23 Thread via GitHub
dashton90 commented on PR #35790: URL: https://github.com/apache/airflow/pull/35790#issuecomment-1825038027 > Do you want to poke at it and get it sorted out, or do you want me to? If you're already on it then I'll leave it to you. But happy to take over if you need another set of eye

Re: [PR] Add EC2HibernateInstanceOperator and EC2RebootInstanceOperator [airflow]

2023-11-23 Thread via GitHub
ferruzzi commented on PR #35790: URL: https://github.com/apache/airflow/pull/35790#issuecomment-1825044746 It looks like in order to get this working we're going to need to set up a KMS key and manually define the device mappings to configure an EBS volume. I won't be able to get this done

[PR] Refactor and simplify breeze-based pre-commits [airflow]

2023-11-23 Thread via GitHub
potiuk opened a new pull request, #35830: URL: https://github.com/apache/airflow/pull/35830 Some of our pre-commits were using code from Breeze in rather complex way - by inserting PYTHONPATH and importing code from there. That was complex and brittle and with recent changes of ShelParam #3

  1   2   >