[GitHub] [airflow] matteosdocsity commented on issue #16001: Can't view running tasks logs using KubernetesExecutor.

2023-04-18 Thread via GitHub
matteosdocsity commented on issue #16001: URL: https://github.com/apache/airflow/issues/16001#issuecomment-1514210233 is there anyone that is using stackdriver as logging? Even if this logs are pulled in streaming I cannot see anything else rather than Airflow built-in logs like: `[2

[GitHub] [airflow] MattiaGallegati commented on issue #16163: Confusing log for long running tasks: "dependency 'Task Instance Not Running' FAILED: Task is in the running state"

2023-04-18 Thread via GitHub
MattiaGallegati commented on issue #16163: URL: https://github.com/apache/airflow/issues/16163#issuecomment-1514174852 Hello, are there any new evidence on this bug? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [airflow] eladkal commented on a diff in pull request #30718: Add a new arg for KPO to only delete the pod when it doesn't fail

2023-04-18 Thread via GitHub
eladkal commented on code in PR #30718: URL: https://github.com/apache/airflow/pull/30718#discussion_r1170797073 ## airflow/providers/cncf/kubernetes/operators/pod.py: ## @@ -191,6 +191,8 @@ class KubernetesPodOperator(BaseOperator): :param is_delete_operator_pod: What to d

[GitHub] [airflow] utkarsharma2 commented on pull request #30644: Make --executor flag work for breeze start-airflow command

2023-04-18 Thread via GitHub
utkarsharma2 commented on PR #30644: URL: https://github.com/apache/airflow/pull/30644#issuecomment-1514115309 > I think all of the executors could be gradually added but we should start small. > > BTW.. This is just a beginning and draft of the change. This PR needs far more to be e

[GitHub] [airflow] Bowrna opened a new pull request, #30730: adding docker port expose capability

2023-04-18 Thread via GitHub
Bowrna opened a new pull request, #30730: URL: https://github.com/apache/airflow/pull/30730 closes: #27290 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines

[GitHub] [airflow] Bowrna commented on pull request #30729: description added for pagination in get_log api

2023-04-18 Thread via GitHub
Bowrna commented on PR #30729: URL: https://github.com/apache/airflow/pull/30729#issuecomment-1514111091 @ephraimbuddy @bbovenzi I have added changes in the description for `get_log` api about pagination part -- This is an automated message from the Apache Git Service. To respond to the m

[GitHub] [airflow] Bowrna opened a new pull request, #30729: description added for pagination in get_log api

2023-04-18 Thread via GitHub
Bowrna opened a new pull request, #30729: URL: https://github.com/apache/airflow/pull/30729 closes: #29405 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines

[GitHub] [airflow] Bowrna closed pull request #30728: Docs changes for get log api pagination part

2023-04-18 Thread via GitHub
Bowrna closed pull request #30728: Docs changes for get log api pagination part URL: https://github.com/apache/airflow/pull/30728 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [airflow] Bowrna opened a new pull request, #30728: Docs changes for get log api pagination part

2023-04-18 Thread via GitHub
Bowrna opened a new pull request, #30728: URL: https://github.com/apache/airflow/pull/30728 closes: #29405 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines

[GitHub] [airflow] dylanbstorey commented on pull request #30175: Add provider for Apache Kafka

2023-04-18 Thread via GitHub
dylanbstorey commented on PR #30175: URL: https://github.com/apache/airflow/pull/30175#issuecomment-151405 @hussein-awala - already did it at your request. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

[GitHub] [airflow] hussein-awala commented on pull request #30175: Add provider for Apache Kafka

2023-04-18 Thread via GitHub
hussein-awala commented on PR #30175: URL: https://github.com/apache/airflow/pull/30175#issuecomment-1514031935 Need to merge main and resolve the conflicts -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [airflow] dylanbstorey commented on pull request #30175: Add provider for Apache Kafka

2023-04-18 Thread via GitHub
dylanbstorey commented on PR #30175: URL: https://github.com/apache/airflow/pull/30175#issuecomment-1514015868 @raphaelauv , what changes are you looking for. Based on your comment - i don't understand. > Producer default setup "could/should" be at least once and preserve order with

[GitHub] [airflow] wookiist commented on pull request #30623: Attaching a logGroomerSidecar to dag processor

2023-04-18 Thread via GitHub
wookiist commented on PR #30623: URL: https://github.com/apache/airflow/pull/30623#issuecomment-1514010166 I accidentally rebased the commit, sorry. I'm trying to continue the story in a new PR. Thank you for your understanding. https://github.com/apache/airflow/pull/30726 -- Th

[GitHub] [airflow] wookiist commented on pull request #30726: Adding loggroomer sidecar to dag processor

2023-04-18 Thread via GitHub
wookiist commented on PR #30726: URL: https://github.com/apache/airflow/pull/30726#issuecomment-1513987215 https://github.com/apache/airflow/pull/30623 was the previous PR, but it was corrupted by my work :( So I created new PR, can we continue our discussion here? @potiuk 🙇🏼 -- Thi

[GitHub] [airflow] o-nikolas opened a new pull request, #30727: Kubernetes Executor Load Time Optimizations

2023-04-18 Thread via GitHub
o-nikolas opened a new pull request, #30727: URL: https://github.com/apache/airflow/pull/30727 ### Overview This PR aims to improve the time it takes to load/import the kubernetes executor module. ### Motivation Executors are imported in more places now that various compatibilit

[GitHub] [airflow] wookiist opened a new pull request, #30726: Adding loggroomer sidecar to dag processor

2023-04-18 Thread via GitHub
wookiist opened a new pull request, #30726: URL: https://github.com/apache/airflow/pull/30726 > I'm going to continue our discussion of PRs here, as [previous PR](https://github.com/apache/airflow/pull/30623)s have been corrupted by my bad work 😭 This PR attaches the logGroomer side

[GitHub] [airflow] potiuk commented on issue #29643: Add google-cloud-run python client to GCP requirements

2023-04-18 Thread via GitHub
potiuk commented on issue #29643: URL: https://github.com/apache/airflow/issues/29643#issuecomment-1513978389 This should be done after (or even during #30067). This should unblock depdendencies. -- This is an automated message from the Apache Git Service. To respond to the message, plea

[GitHub] [airflow] potiuk commented on pull request #29223: Split installation of sdist providers into parallel chunks

2023-04-18 Thread via GitHub
potiuk commented on PR #29223: URL: https://github.com/apache/airflow/pull/29223#issuecomment-1513976872 Conflict after merging #30705 resolved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

[GitHub] [airflow] potiuk commented on pull request #30705: Optimize parallel test execution for unit tests

2023-04-18 Thread via GitHub
potiuk commented on PR #30705: URL: https://github.com/apache/airflow/pull/30705#issuecomment-1513970913 Now we only need https://github.com/apache/airflow/pull/30682 to get the full speedup/feedback (without it, sdist test takes 27 minutes on main). -- This is an automated message from t

[airflow] branch main updated: Optimize parallel test execution for unit tests (#30705)

2023-04-18 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 310eaf9a1f Optimize parallel test execution for uni

[GitHub] [airflow] potiuk merged pull request #30705: Optimize parallel test execution for unit tests

2023-04-18 Thread via GitHub
potiuk merged PR #30705: URL: https://github.com/apache/airflow/pull/30705 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[GitHub] [airflow] potiuk commented on pull request #30352: support requirepass redis sentinel

2023-04-18 Thread via GitHub
potiuk commented on PR #30352: URL: https://github.com/apache/airflow/pull/30352#issuecomment-1513966195 @jonathanjuursema - did you manage it to work? The build is green, but I am hesitating to merge that one, as I am not sure if this is general enough. Possibly also adding documentation d

[GitHub] [airflow] potiuk closed issue #29737: Metric dag_processing.total_parse_time is exposed incorrectly

2023-04-18 Thread via GitHub
potiuk closed issue #29737: Metric dag_processing.total_parse_time is exposed incorrectly URL: https://github.com/apache/airflow/issues/29737 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

[GitHub] [airflow] potiuk commented on issue #29737: Metric dag_processing.total_parse_time is exposed incorrectly

2023-04-18 Thread via GitHub
potiuk commented on issue #29737: URL: https://github.com/apache/airflow/issues/29737#issuecomment-1513963179 I think this is something to be addressed with the Open-Telemetry implementation @feruzzi @howardyoo - possibly dropping or replacing this metric with some aggregated one. I

[GitHub] [airflow] wookiist commented on pull request #30623: Attaching a logGroomerSidecar to dag processor

2023-04-18 Thread via GitHub
wookiist commented on PR #30623: URL: https://github.com/apache/airflow/pull/30623#issuecomment-1513961309 oh sorry for noise. my mistake :( I wrongly rebased and pushed it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [airflow] potiuk commented on a diff in pull request #30705: Optimize parallel test execution for unit tests

2023-04-18 Thread via GitHub
potiuk commented on code in PR #30705: URL: https://github.com/apache/airflow/pull/30705#discussion_r1170683631 ## dev/breeze/src/airflow_breeze/utils/selective_checks.py: ## @@ -606,7 +606,43 @@ def parallel_test_types(self) -> str: ) t

[GitHub] [airflow] github-actions[bot] commented on pull request #29811: Add support for Yandex Dataproc cluster labels

2023-04-18 Thread via GitHub
github-actions[bot] commented on PR #29811: URL: https://github.com/apache/airflow/pull/29811#issuecomment-1513935293 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for you

[GitHub] [airflow] github-actions[bot] commented on pull request #29780: Fix to make use of labels parameter in ClusterGenerator.

2023-04-18 Thread via GitHub
github-actions[bot] commented on PR #29780: URL: https://github.com/apache/airflow/pull/29780#issuecomment-1513935364 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for you

[GitHub] [airflow] potiuk commented on a diff in pull request #30705: Optimize parallel test execution for unit tests

2023-04-18 Thread via GitHub
potiuk commented on code in PR #30705: URL: https://github.com/apache/airflow/pull/30705#discussion_r1170671590 ## dev/breeze/src/airflow_breeze/utils/selective_checks.py: ## @@ -606,7 +606,43 @@ def parallel_test_types(self) -> str: ) t

[GitHub] [airflow] hussein-awala commented on a diff in pull request #30705: Optimize parallel test execution for unit tests

2023-04-18 Thread via GitHub
hussein-awala commented on code in PR #30705: URL: https://github.com/apache/airflow/pull/30705#discussion_r1170669657 ## dev/breeze/src/airflow_breeze/utils/selective_checks.py: ## @@ -606,7 +606,43 @@ def parallel_test_types(self) -> str: )

[GitHub] [airflow] potiuk commented on pull request #30711: Allow a user to collapse the grid view

2023-04-18 Thread via GitHub
potiuk commented on PR #30711: URL: https://github.com/apache/airflow/pull/30711#issuecomment-1513904303 Nice one! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

[airflow] branch main updated: SqlToS3Operator - Add feature to partition SQL table (#30460)

2023-04-18 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new d7cef588d6 SqlToS3Operator - Add feature to partiti

[GitHub] [airflow] potiuk merged pull request #30460: SqlToS3Operator - Add feature to partition SQL table

2023-04-18 Thread via GitHub
potiuk merged PR #30460: URL: https://github.com/apache/airflow/pull/30460 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[GitHub] [airflow] potiuk closed issue #30382: SqlToS3Operator not able to write data with partition_cols provided.

2023-04-18 Thread via GitHub
potiuk closed issue #30382: SqlToS3Operator not able to write data with partition_cols provided. URL: https://github.com/apache/airflow/issues/30382 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[GitHub] [airflow] potiuk closed issue #30499: Launch Airflow Webserver via multiprocessing

2023-04-18 Thread via GitHub
potiuk closed issue #30499: Launch Airflow Webserver via multiprocessing URL: https://github.com/apache/airflow/issues/30499 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [airflow] potiuk commented on issue #30499: Launch Airflow Webserver via multiprocessing

2023-04-18 Thread via GitHub
potiuk commented on issue #30499: URL: https://github.com/apache/airflow/issues/30499#issuecomment-1513896449 Correct. This is how airflow works, and we are not going to change it. Airflow webserver shoudl be using gunicorn, subprocess and forking and converting that into single packaged ex

[GitHub] [airflow] potiuk commented on pull request #30312: Reformat chart templates part 3

2023-04-18 Thread via GitHub
potiuk commented on PR #30312: URL: https://github.com/apache/airflow/pull/30312#issuecomment-1513889625 > I'm not super excited about unit tests for the helm chart. I mean it's quite hard to run and maintain them, but I don't know and cannot find better/faster existing alternative.

[GitHub] [airflow] potiuk commented on pull request #30331: AIP-44 Migrate MetastoreBackend to Internal API

2023-04-18 Thread via GitHub
potiuk commented on PR #30331: URL: https://github.com/apache/airflow/pull/30331#issuecomment-1513887100 Needs conflict resolving :( -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [airflow] potiuk closed issue #30697: requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://login.cac1.pure.cloud/oauth/token

2023-04-18 Thread via GitHub
potiuk closed issue #30697: requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://login.cac1.pure.cloud/oauth/token URL: https://github.com/apache/airflow/issues/30697 -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [airflow] potiuk commented on pull request #30252: Add deferrable mode to `WasbPrefixSensor`

2023-04-18 Thread via GitHub
potiuk commented on PR #30252: URL: https://github.com/apache/airflow/pull/30252#issuecomment-1513877234 Stati checks failure. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[airflow] branch vincbeck/add-tests-to-python-operator updated (08534629fb -> e77b93c932)

2023-04-18 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch vincbeck/add-tests-to-python-operator in repository https://gitbox.apache.org/repos/asf/airflow.git omit 08534629fb Better message on deserialization error add 35a5bbe846 Fix docs: add a

[GitHub] [airflow] potiuk commented on a diff in pull request #30705: Optimize parallel test execution for unit tests

2023-04-18 Thread via GitHub
potiuk commented on code in PR #30705: URL: https://github.com/apache/airflow/pull/30705#discussion_r1170597253 ## dev/breeze/src/airflow_breeze/utils/selective_checks.py: ## @@ -606,7 +606,43 @@ def parallel_test_types(self) -> str: ) t

[GitHub] [airflow] potiuk commented on a diff in pull request #30705: Optimize parallel test execution for unit tests

2023-04-18 Thread via GitHub
potiuk commented on code in PR #30705: URL: https://github.com/apache/airflow/pull/30705#discussion_r1170596292 ## dev/breeze/src/airflow_breeze/utils/selective_checks.py: ## @@ -606,7 +606,43 @@ def parallel_test_types(self) -> str: ) t

[GitHub] [airflow] hussein-awala commented on a diff in pull request #30718: Add a new arg for KPO to only delete the pod when it doesn't fail

2023-04-18 Thread via GitHub
hussein-awala commented on code in PR #30718: URL: https://github.com/apache/airflow/pull/30718#discussion_r1170592987 ## airflow/providers/cncf/kubernetes/operators/pod.py: ## @@ -191,6 +191,8 @@ class KubernetesPodOperator(BaseOperator): :param is_delete_operator_pod: Wha

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #30705: Optimize parallel test execution for unit tests

2023-04-18 Thread via GitHub
vandonr-amz commented on code in PR #30705: URL: https://github.com/apache/airflow/pull/30705#discussion_r1170591283 ## dev/breeze/src/airflow_breeze/utils/selective_checks.py: ## @@ -606,7 +606,43 @@ def parallel_test_types(self) -> str: )

[GitHub] [airflow] potiuk commented on a diff in pull request #30705: Optimize parallel test execution for unit tests

2023-04-18 Thread via GitHub
potiuk commented on code in PR #30705: URL: https://github.com/apache/airflow/pull/30705#discussion_r1170580465 ## dev/breeze/src/airflow_breeze/utils/selective_checks.py: ## @@ -606,7 +606,43 @@ def parallel_test_types(self) -> str: ) t

[GitHub] [airflow] potiuk commented on a diff in pull request #30705: Optimize parallel test execution for unit tests

2023-04-18 Thread via GitHub
potiuk commented on code in PR #30705: URL: https://github.com/apache/airflow/pull/30705#discussion_r1170578263 ## dev/breeze/src/airflow_breeze/utils/selective_checks.py: ## @@ -606,7 +606,43 @@ def parallel_test_types(self) -> str: ) t

[GitHub] [airflow] potiuk commented on a diff in pull request #30705: Optimize parallel test execution for unit tests

2023-04-18 Thread via GitHub
potiuk commented on code in PR #30705: URL: https://github.com/apache/airflow/pull/30705#discussion_r1170573243 ## dev/breeze/src/airflow_breeze/utils/selective_checks.py: ## @@ -606,7 +606,43 @@ def parallel_test_types(self) -> str: ) t

[GitHub] [airflow] potiuk commented on a diff in pull request #30705: Optimize parallel test execution for unit tests

2023-04-18 Thread via GitHub
potiuk commented on code in PR #30705: URL: https://github.com/apache/airflow/pull/30705#discussion_r1170563693 ## dev/breeze/src/airflow_breeze/utils/selective_checks.py: ## @@ -606,7 +606,43 @@ def parallel_test_types(self) -> str: ) t

[GitHub] [airflow] potiuk commented on a diff in pull request #30705: Optimize parallel test execution for unit tests

2023-04-18 Thread via GitHub
potiuk commented on code in PR #30705: URL: https://github.com/apache/airflow/pull/30705#discussion_r1170566245 ## dev/breeze/src/airflow_breeze/utils/selective_checks.py: ## @@ -606,7 +606,43 @@ def parallel_test_types(self) -> str: ) t

[GitHub] [airflow] potiuk commented on a diff in pull request #30705: Optimize parallel test execution for unit tests

2023-04-18 Thread via GitHub
potiuk commented on code in PR #30705: URL: https://github.com/apache/airflow/pull/30705#discussion_r1170566245 ## dev/breeze/src/airflow_breeze/utils/selective_checks.py: ## @@ -606,7 +606,43 @@ def parallel_test_types(self) -> str: ) t

[GitHub] [airflow] potiuk commented on a diff in pull request #30705: Optimize parallel test execution for unit tests

2023-04-18 Thread via GitHub
potiuk commented on code in PR #30705: URL: https://github.com/apache/airflow/pull/30705#discussion_r1170563693 ## dev/breeze/src/airflow_breeze/utils/selective_checks.py: ## @@ -606,7 +606,43 @@ def parallel_test_types(self) -> str: ) t

[GitHub] [airflow] potiuk commented on a diff in pull request #30705: Optimize parallel test execution for unit tests

2023-04-18 Thread via GitHub
potiuk commented on code in PR #30705: URL: https://github.com/apache/airflow/pull/30705#discussion_r1170563693 ## dev/breeze/src/airflow_breeze/utils/selective_checks.py: ## @@ -606,7 +606,43 @@ def parallel_test_types(self) -> str: ) t

[GitHub] [airflow] potiuk commented on a diff in pull request #30705: Optimize parallel test execution for unit tests

2023-04-18 Thread via GitHub
potiuk commented on code in PR #30705: URL: https://github.com/apache/airflow/pull/30705#discussion_r1170563693 ## dev/breeze/src/airflow_breeze/utils/selective_checks.py: ## @@ -606,7 +606,43 @@ def parallel_test_types(self) -> str: ) t

[GitHub] [airflow] vincbeck commented on a diff in pull request #30720: add a stop operator to emr serverless

2023-04-18 Thread via GitHub
vincbeck commented on code in PR #30720: URL: https://github.com/apache/airflow/pull/30720#discussion_r1170563040 ## docs/apache-airflow-providers-amazon/operators/emr/emr_serverless.rst: ## @@ -63,6 +63,18 @@ start an EMR Serverless Job. .. _howto/operator:EmrServerlessDelet

[GitHub] [airflow] potiuk commented on a diff in pull request #30705: Optimize parallel test execution for unit tests

2023-04-18 Thread via GitHub
potiuk commented on code in PR #30705: URL: https://github.com/apache/airflow/pull/30705#discussion_r1170563693 ## dev/breeze/src/airflow_breeze/utils/selective_checks.py: ## @@ -606,7 +606,43 @@ def parallel_test_types(self) -> str: ) t

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #30705: Optimize parallel test execution for unit tests

2023-04-18 Thread via GitHub
vandonr-amz commented on code in PR #30705: URL: https://github.com/apache/airflow/pull/30705#discussion_r1170551519 ## dev/breeze/src/airflow_breeze/utils/selective_checks.py: ## @@ -606,7 +606,43 @@ def parallel_test_types(self) -> str: )

[airflow] branch constraints-main updated: Updating constraints. Github run id:4736373992

2023-04-18 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 9f982290d0 Updating con

[GitHub] [airflow] raphaelauv commented on pull request #30175: Add provider for Apache Kafka

2023-04-18 Thread via GitHub
raphaelauv commented on PR #30175: URL: https://github.com/apache/airflow/pull/30175#issuecomment-1513755494 what do you mean by ? ``` I'm going to need more to go on to address your comment. ``` -- This is an automated message from the Apache Git Service. To respond to the m

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #30720: add a stop operator to emr serverless

2023-04-18 Thread via GitHub
vandonr-amz commented on code in PR #30720: URL: https://github.com/apache/airflow/pull/30720#discussion_r1170533027 ## tests/system/providers/amazon/aws/example_emr_serverless.py: ## @@ -108,6 +109,15 @@ job_run_id=start_job.output, ) # [END howto_sensor_emr_

[GitHub] [airflow] dylanbstorey commented on pull request #30175: Add provider for Apache Kafka

2023-04-18 Thread via GitHub
dylanbstorey commented on PR #30175: URL: https://github.com/apache/airflow/pull/30175#issuecomment-1513752757 I THINK we're done with the substantive feedback. @hussein-awala , can you review and clear as appropriate. (And ping me with anything that needs further discussion please.) --

[GitHub] [airflow] potiuk commented on pull request #30175: Add provider for Apache Kafka

2023-04-18 Thread via GitHub
potiuk commented on PR #30175: URL: https://github.com/apache/airflow/pull/30175#issuecomment-1513749534 FYI. The Kafka Provider has been accepted by Lazy Consensus now, and I see we still have some iterations to complete, but this can now be done with the realm of being "blessed" by the co

[GitHub] [airflow] boring-cyborg[bot] commented on issue #30722: Ability to set automountServiceAccountToken in values.yaml

2023-04-18 Thread via GitHub
boring-cyborg[bot] commented on issue #30722: URL: https://github.com/apache/airflow/issues/30722#issuecomment-1513748423 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for ap

[GitHub] [airflow] fsellecchia opened a new issue, #30722: Ability to set automountServiceAccountToken in values.yaml

2023-04-18 Thread via GitHub
fsellecchia opened a new issue, #30722: URL: https://github.com/apache/airflow/issues/30722 ### Official Helm Chart version 1.6.0 ### Apache Airflow version 2.3.0 ### Kubernetes Version 1.24 ### Helm Chart configuration ``` serviceAccount:

[GitHub] [airflow] vincbeck commented on a diff in pull request #30720: add a stop operator to emr serverless

2023-04-18 Thread via GitHub
vincbeck commented on code in PR #30720: URL: https://github.com/apache/airflow/pull/30720#discussion_r1170506237 ## tests/system/providers/amazon/aws/example_emr_serverless.py: ## @@ -108,6 +109,15 @@ job_run_id=start_job.output, ) # [END howto_sensor_emr_ser

[airflow] branch main updated (221249e872 -> 372a0881d9)

2023-04-18 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 221249e872 Fixed logging issue (#30703) add 372a0881d9 Separate out and clarify policies for providers (#30657) N

[GitHub] [airflow] potiuk merged pull request #30657: Separate out and clarify policies for providers

2023-04-18 Thread via GitHub
potiuk merged PR #30657: URL: https://github.com/apache/airflow/pull/30657 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[GitHub] [airflow] eladkal commented on a diff in pull request #30718: Add a new arg for KPO to only delete the pod when it doesn't fail

2023-04-18 Thread via GitHub
eladkal commented on code in PR #30718: URL: https://github.com/apache/airflow/pull/30718#discussion_r1170501085 ## airflow/providers/cncf/kubernetes/operators/pod.py: ## @@ -191,6 +191,8 @@ class KubernetesPodOperator(BaseOperator): :param is_delete_operator_pod: What to d

[GitHub] [airflow] potiuk commented on issue #22397: Upgrade to FlaskAppBuilder 4.0.*

2023-04-18 Thread via GitHub
potiuk commented on issue #22397: URL: https://github.com/apache/airflow/issues/22397#issuecomment-1513705886 > Hi > > We are using Apache-airflow v2.5.1 which is using with Flask-AppBuilder version 4.1.4 > > A few days back Flask-AppBuilder version 4.1.4 is marked as vulnerabl

[GitHub] [airflow] sainman commented on issue #22397: Upgrade to FlaskAppBuilder 4.0.*

2023-04-18 Thread via GitHub
sainman commented on issue #22397: URL: https://github.com/apache/airflow/issues/22397#issuecomment-1513696796 Hi We are using Apache-airflow v2.5.1 which is using with Flask-AppBuilder version 4.1.4 A few days back Flask-AppBuilder version 4.1.4 is marked as vulnerable with h

[GitHub] [airflow] hussein-awala commented on a diff in pull request #30641: Count mapped upstreams only if all are finished

2023-04-18 Thread via GitHub
hussein-awala commented on code in PR #30641: URL: https://github.com/apache/airflow/pull/30641#discussion_r1170458286 ## airflow/models/xcom_arg.py: ## @@ -309,11 +310,26 @@ def zip(self, *others: XComArg, fillvalue: Any = NOTSET) -> ZipXComArg: return super().zip(*ot

[GitHub] [airflow] vandonr-amz opened a new pull request, #30720: add a stop operator to emr serverless

2023-04-18 Thread via GitHub
vandonr-amz opened a new pull request, #30720: URL: https://github.com/apache/airflow/pull/30720 EMR serverless currently has a `delete` operator that stops the app and deletes it, but there is no operator to just stop an app. Since the delete operator contains a stop step, to avoid d

[GitHub] [airflow] mobuchowski opened a new pull request, #30719: openlineage: add base SQL extractor for SQL-based operators

2023-04-18 Thread via GitHub
mobuchowski opened a new pull request, #30719: URL: https://github.com/apache/airflow/pull/30719 Add base SqlExtractor. This provides base upon which extractors gathering data from operators which derive from `SQLExecuteQueryOperator` are build. This means - passing SQL to SQL parser wi

[GitHub] [airflow] hussein-awala opened a new pull request, #30718: Add a new arg for KPO to only delete the pod when it doesn't fail

2023-04-18 Thread via GitHub
hussein-awala opened a new pull request, #30718: URL: https://github.com/apache/airflow/pull/30718 Add a new argument `delete_when_fails` for `KubernetesPodOperator`, when it is set to `False` and `is_delete_operator_pod` is set to `True`, the task will only delete the pod if it terminates

[GitHub] [airflow] jens-scheffler-bosch commented on pull request #30699: Save scheduler execution time during search for queued dag_runs

2023-04-18 Thread via GitHub
jens-scheffler-bosch commented on PR #30699: URL: https://github.com/apache/airflow/pull/30699#issuecomment-1513608529 > Nice small change with a big impact :) I'm not familiar with SQLAlchemy, so I'll rephrase my comprehension of your change here to make sure I understand well what you're

[GitHub] [airflow] potiuk commented on a diff in pull request #30705: Optimize parallel test execution for unit tests

2023-04-18 Thread via GitHub
potiuk commented on code in PR #30705: URL: https://github.com/apache/airflow/pull/30705#discussion_r1170413102 ## dev/breeze/src/airflow_breeze/params/shell_params.py: ## @@ -267,9 +268,13 @@ def command_passed(self): @property def mssql_data_volume(self) -> str:

[GitHub] [airflow] potiuk commented on a diff in pull request #30705: Optimize parallel test execution for unit tests

2023-04-18 Thread via GitHub
potiuk commented on code in PR #30705: URL: https://github.com/apache/airflow/pull/30705#discussion_r1170411789 ## dev/breeze/src/airflow_breeze/commands/testing_commands.py: ## @@ -434,16 +390,11 @@ def command_for_tests( perform_environment_checks() if run_in_paralle

[GitHub] [airflow] potiuk commented on a diff in pull request #30705: Optimize parallel test execution for unit tests

2023-04-18 Thread via GitHub
potiuk commented on code in PR #30705: URL: https://github.com/apache/airflow/pull/30705#discussion_r1170411074 ## dev/breeze/src/airflow_breeze/commands/testing_commands.py: ## @@ -268,43 +261,12 @@ def run_tests_in_parallel( parallel_test_types_list: list[str], extra

[airflow] branch main updated: Fixed logging issue (#30703)

2023-04-18 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 221249e872 Fixed logging issue (#30703) 221249e872

[GitHub] [airflow] potiuk merged pull request #30703: Fixed logging issue in `DynamoDBValueSensor`

2023-04-18 Thread via GitHub
potiuk merged PR #30703: URL: https://github.com/apache/airflow/pull/30703 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[GitHub] [airflow] potiuk commented on a diff in pull request #30705: Optimize parallel test execution for Google, Amazon, WWW tests

2023-04-18 Thread via GitHub
potiuk commented on code in PR #30705: URL: https://github.com/apache/airflow/pull/30705#discussion_r1170399473 ## BREEZE.rst: ## @@ -788,6 +788,16 @@ For example this will only run provider tests for airbyte and http providers: breeze testing tests --test-type "Providers

[GitHub] [airflow] potiuk commented on a diff in pull request #30705: Optimize parallel test execution for Google, Amazon, WWW tests

2023-04-18 Thread via GitHub
potiuk commented on code in PR #30705: URL: https://github.com/apache/airflow/pull/30705#discussion_r1170399473 ## BREEZE.rst: ## @@ -788,6 +788,16 @@ For example this will only run provider tests for airbyte and http providers: breeze testing tests --test-type "Providers

[airflow] branch main updated: Update system test example_emr to have logs (#30715)

2023-04-18 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new c4f0d9f004 Update system test example_emr to have l

[GitHub] [airflow] potiuk merged pull request #30715: Update system test example_emr to have logs

2023-04-18 Thread via GitHub
potiuk merged PR #30715: URL: https://github.com/apache/airflow/pull/30715 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[GitHub] [airflow] potiuk commented on pull request #30717: Update faq.rst about missing task logs

2023-04-18 Thread via GitHub
potiuk commented on PR #30717: URL: https://github.com/apache/airflow/pull/30717#issuecomment-1513586264 I like the description and good idea to mention it, maybe others would like to add something? -- This is an automated message from the Apache Git Service. To respond to the message, pl

[GitHub] [airflow] RNHTTR opened a new pull request, #30717: Update faq.rst about missing task logs

2023-04-18 Thread via GitHub
RNHTTR opened a new pull request, #30717: URL: https://github.com/apache/airflow/pull/30717 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more info

[GitHub] [airflow] o-nikolas commented on issue #30341: S3ToRedshiftOperator does not support default values on upsert

2023-04-18 Thread via GitHub
o-nikolas commented on issue #30341: URL: https://github.com/apache/airflow/issues/30341#issuecomment-1513571287 @shubham22 @Gowthami03B done :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[GitHub] [airflow] vincbeck commented on a diff in pull request #30705: Optimize parallel test execution for Google, Amazon, WWW tests

2023-04-18 Thread via GitHub
vincbeck commented on code in PR #30705: URL: https://github.com/apache/airflow/pull/30705#discussion_r1170371850 ## BREEZE.rst: ## @@ -788,6 +788,16 @@ For example this will only run provider tests for airbyte and http providers: breeze testing tests --test-type "Provide

[GitHub] [airflow] syun64 opened a new issue, #30716: Metrics Tag produces invalid tags

2023-04-18 Thread via GitHub
syun64 opened a new issue, #30716: URL: https://github.com/apache/airflow/issues/30716 ### Apache Airflow version 2.6.0b1 ### What happened The recently added **dag_processing.processes** _file_path_ metric tag always fails to publish the metric tag because file path del

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #30032: Add support for deferrable operators in AMPP

2023-04-18 Thread via GitHub
vandonr-amz commented on code in PR #30032: URL: https://github.com/apache/airflow/pull/30032#discussion_r1170312053 ## airflow/providers/amazon/aws/hooks/base_aws.py: ## @@ -684,7 +684,6 @@ def conn(self) -> BaseAwsConnection: else: return self.get_resourc

[GitHub] [airflow] shubham22 commented on issue #30341: S3ToRedshiftOperator does not support default values on upsert

2023-04-18 Thread via GitHub
shubham22 commented on issue #30341: URL: https://github.com/apache/airflow/issues/30341#issuecomment-1513486872 @o-nikolas can you assign this to @Gowthami03B? TIA! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #30704: Save scheduler execution time by caching dags

2023-04-18 Thread via GitHub
vandonr-amz commented on code in PR #30704: URL: https://github.com/apache/airflow/pull/30704#discussion_r1170303898 ## airflow/jobs/scheduler_job_runner.py: ## @@ -1083,8 +1083,13 @@ def _do_scheduling(self, session: Session) -> int: callback_tuples = self._schedul

[GitHub] [airflow] vandonr-amz commented on pull request #30699: Save scheduler execution time during search for queued dag_runs

2023-04-18 Thread via GitHub
vandonr-amz commented on PR #30699: URL: https://github.com/apache/airflow/pull/30699#issuecomment-1513458550 Nice small change with a big impact :) I'm not familiar with SQLAlchemy, so I'll rephrase my comprehension of your change here to make sure I understand well what you're doing (an

[GitHub] [airflow] RNHTTR commented on a diff in pull request #30375: Simplify logic to resolve tasks stuck in queued despite stalled_task_timeout

2023-04-18 Thread via GitHub
RNHTTR commented on code in PR #30375: URL: https://github.com/apache/airflow/pull/30375#discussion_r1170250855 ## airflow/executors/celery_executor.py: ## @@ -543,22 +417,28 @@ def try_adopt_task_instances(self, tis: Sequence[TaskInstance]) -> Sequence[Task return n

[GitHub] [airflow] sthakkar-move commented on issue #16610: Dag Dependency page not showing anything

2023-04-18 Thread via GitHub
sthakkar-move commented on issue #16610: URL: https://github.com/apache/airflow/issues/16610#issuecomment-1513397908 @rvatsastro thanks for reporting. did you have to do anything to resolve the issue ? We are on MWAA 2.0.2 and still seeing the same issue https://user-images.githubusercont

[GitHub] [airflow] swapniel99 commented on issue #22194: Paramiko 2.9.x breaking SSH operator for some OpenSSH configurations

2023-04-18 Thread via GitHub
swapniel99 commented on issue #22194: URL: https://github.com/apache/airflow/issues/22194#issuecomment-1513366103 Hi, Just putting this here in case someone else faces this issue. https://askubuntu.com/questions/1409105/ubuntu-22-04-ssh-the-rsa-key-isnt-working-since-upgrading-

[GitHub] [airflow] saulbein commented on pull request #30563: Change alembic logging level to WARN

2023-04-18 Thread via GitHub
saulbein commented on PR #30563: URL: https://github.com/apache/airflow/pull/30563#issuecomment-1513365214 What about setting up `logging_config_class` before running alembic commands and not touching logs with alembic at all? That way we could use the existing logging override mechanism to

[GitHub] [airflow] vincbeck opened a new pull request, #30715: Update system test example_emr to have logs

2023-04-18 Thread via GitHub
vincbeck opened a new pull request, #30715: URL: https://github.com/apache/airflow/pull/30715 Update system test `example_emr` to have logs --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CON

[GitHub] [airflow-site] potiuk commented on pull request #766: Add canonical link to archived docs

2023-04-18 Thread via GitHub
potiuk commented on PR #766: URL: https://github.com/apache/airflow-site/pull/766#issuecomment-1513307048 Cool. I think we will also see when we generate next provider's wave. so we will have to see then. -- This is an automated message from the Apache Git Service. To respond to the messa

[GitHub] [airflow] prakash12 opened a new pull request, #30714: Google cloud code format

2023-04-18 Thread via GitHub
prakash12 opened a new pull request, #30714: URL: https://github.com/apache/airflow/pull/30714 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more i

  1   2   >