[GitHub] [airflow] binnisb commented on issue #29127: Use latest azure-mgmt-containerinstance (10.0.0) which depricates network_profile and adds subnet_ids

2023-01-24 Thread via GitHub
binnisb commented on issue #29127: URL: https://github.com/apache/airflow/issues/29127#issuecomment-1403205812 I can do that if I get it working for our use case. I have 2 week so if I haven't opened a PR by then I will not have time to do it. -- This is an automated message from the

[GitHub] [airflow] Aakcht commented on issue #21891: hive provider support for python 3.9

2023-01-24 Thread via GitHub
Aakcht commented on issue #21891: URL: https://github.com/apache/airflow/issues/21891#issuecomment-1403197225 I didn't personally test it, but I don't see any issues related to hive provider and python 3.10 so I suspect it should work fine (since the latest airflow versions support python

[GitHub] [airflow] BasPH commented on a diff in pull request #29143: Demonstrate usage of the PythonSensor

2023-01-24 Thread via GitHub
BasPH commented on code in PR #29143: URL: https://github.com/apache/airflow/pull/29143#discussion_r1086270155 ## docs/apache-airflow/howto/operator/python.rst: ## @@ -225,11 +225,29 @@ Jinja templating can be used in same way as described for the PythonOperator. PythonSensor

[GitHub] [airflow] sergei3000 commented on issue #21891: hive provider support for python 3.9

2023-01-24 Thread via GitHub
sergei3000 commented on issue #21891: URL: https://github.com/apache/airflow/issues/21891#issuecomment-1403184373 Hi does all this mean Hive provider will work for Python 3.10 as well? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [airflow] BasPH commented on a diff in pull request #29143: Demonstrate usage of the PythonSensor

2023-01-24 Thread via GitHub
BasPH commented on code in PR #29143: URL: https://github.com/apache/airflow/pull/29143#discussion_r1086267570 ## docs/apache-airflow/howto/operator/python.rst: ## @@ -225,11 +225,29 @@ Jinja templating can be used in same way as described for the PythonOperator. PythonSensor

[GitHub] [airflow] o-nikolas commented on pull request #27947: refactored Amazon Redshift-data functionality into the hook

2023-01-24 Thread via GitHub
o-nikolas commented on PR #27947: URL: https://github.com/apache/airflow/pull/27947#issuecomment-1403147389 Hey @yehoshuadimarsky, Any updates on this one, specifically regarding the feedback from Daniel? -- This is an automated message from the Apache Git Service. To respond to

[GitHub] [airflow] o-nikolas commented on a diff in pull request #28187: Add IAM authentication to Amazon Redshift Connection by AWS Connection

2023-01-24 Thread via GitHub
o-nikolas commented on code in PR #28187: URL: https://github.com/apache/airflow/pull/28187#discussion_r1086234925 ## airflow/providers/amazon/aws/hooks/redshift_sql.py: ## @@ -62,6 +80,9 @@ def _get_conn_params(self) -> dict[str, str | int]: conn_params: dict[str,

[GitHub] [airflow] o-nikolas commented on pull request #28282: Mark stepIds for cancel in EmrAddStepsOperator

2023-01-24 Thread via GitHub
o-nikolas commented on PR #28282: URL: https://github.com/apache/airflow/pull/28282#issuecomment-1403134136 Hey @swapz-z, What are your thoughts on the above feedback from Elad, Vincent and Dennis? Also @dacort, would you like to weigh in? -- This is an automated message from

[GitHub] [airflow] o-nikolas commented on pull request #28321: Add support in AWS Batch Operator for multinode jobs

2023-01-24 Thread via GitHub
o-nikolas commented on PR #28321: URL: https://github.com/apache/airflow/pull/28321#issuecomment-1403132380 Hey @camilleanne, Any plans to pick this one up again? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [airflow] o-nikolas commented on pull request #28338: New AWS sensor — DynamoDBValueSensor

2023-01-24 Thread via GitHub
o-nikolas commented on PR #28338: URL: https://github.com/apache/airflow/pull/28338#issuecomment-1403128044 Hey @mrichman, Any plans to make more progress on this one? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [airflow] o-nikolas commented on a diff in pull request #28827: Add option to wait for completion on the EmrCreateJobFlowOperator

2023-01-24 Thread via GitHub
o-nikolas commented on code in PR #28827: URL: https://github.com/apache/airflow/pull/28827#discussion_r1086224824 ## airflow/providers/amazon/aws/operators/emr.py: ## @@ -538,42 +544,76 @@ def __init__( emr_conn_id: str | None = "emr_default",

[GitHub] [airflow] o-nikolas commented on pull request #29053: introduce base class for EKS sensors

2023-01-24 Thread via GitHub
o-nikolas commented on PR #29053: URL: https://github.com/apache/airflow/pull/29053#issuecomment-1403123198 @ferruzzi as our resident EKS expert, do you have time to take a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[airflow] branch main updated: Fixes to how DebugExecutor handles sensors (#28528)

2023-01-24 Thread onikolas
This is an automated email from the ASF dual-hosted git repository. onikolas pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new a35ec9549d Fixes to how DebugExecutor handles

[GitHub] [airflow] o-nikolas merged pull request #28528: Fixes to how DebugExecutor handles sensors

2023-01-24 Thread via GitHub
o-nikolas merged PR #28528: URL: https://github.com/apache/airflow/pull/28528 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] potiuk commented on pull request #29116: Add colors in help outputs of Airfow CLI commands #28789

2023-01-24 Thread via GitHub
potiuk commented on PR #29116: URL: https://github.com/apache/airflow/pull/29116#issuecomment-1403113821 I think performance in this case is very little impacted. The Click change was much more of the problem because click works with python-decorated methods and it has natural tendency to

[GitHub] [airflow] potiuk commented on pull request #29116: Add colors in help outputs of Airfow CLI commands #28789

2023-01-24 Thread via GitHub
potiuk commented on PR #29116: URL: https://github.com/apache/airflow/pull/29116#issuecomment-1403113028 Looks cool. You need to rebase, and fix the static checks and CLI tests (the SSM test errors will likely go away after rebase - it was broken main I think). -- This is an automated

[GitHub] [airflow] o-nikolas commented on a diff in pull request #29147: Rename most pod_id usage to pod_name in KubernetesExecutor

2023-01-24 Thread via GitHub
o-nikolas commented on code in PR #29147: URL: https://github.com/apache/airflow/pull/29147#discussion_r1086215221 ## airflow/executors/kubernetes_executor.py: ## @@ -792,28 +793,31 @@ def try_adopt_task_instances(self, tis: Sequence[TaskInstance]) -> Sequence[Task

[GitHub] [airflow] sudeepgupta90 commented on issue #28779: Built-in ServiceMonitor for PgBouncer and StatsD

2023-01-24 Thread via GitHub
sudeepgupta90 commented on issue #28779: URL: https://github.com/apache/airflow/issues/28779#issuecomment-1403107145 The latter - adding community exporter for pgbouncer -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [airflow] boring-cyborg[bot] commented on issue #29149: Parameter for disabling scheduling

2023-01-24 Thread boring-cyborg
boring-cyborg[bot] commented on issue #29149: URL: https://github.com/apache/airflow/issues/29149#issuecomment-1403094151 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] jedcunningham opened a new pull request, #29148: General cleanup of things around KubernetesExecutor

2023-01-24 Thread via GitHub
jedcunningham opened a new pull request, #29148: URL: https://github.com/apache/airflow/pull/29148 The biggest change here is using `State` or `TaskInstanceState` where appropriate. Even though their strings are the same, it provides useful context, so we should use the correct enum.

[GitHub] [airflow] rishabh643 opened a new issue, #29149: Parameter for disabling scheduling

2023-01-24 Thread via GitHub
rishabh643 opened a new issue, #29149: URL: https://github.com/apache/airflow/issues/29149 ### Description Can we have a parameter in airflow that can disable scheduling for all dags that are present? I know one can disable dags individually or can turn off the scheduler

[GitHub] [airflow] jedcunningham commented on a diff in pull request #29147: Rename most pod_id usage to pod_name in KubernetesExecutor

2023-01-24 Thread via GitHub
jedcunningham commented on code in PR #29147: URL: https://github.com/apache/airflow/pull/29147#discussion_r1086193863 ## airflow/executors/kubernetes_executor.py: ## @@ -353,12 +353,12 @@ def run_next(self, next_job: KubernetesJobType) -> None: self.run_pod_async(pod,

[GitHub] [airflow] jedcunningham opened a new pull request, #29147: Rename most pod_id usage to pod_name in KubernetesExecutor

2023-01-24 Thread via GitHub
jedcunningham opened a new pull request, #29147: URL: https://github.com/apache/airflow/pull/29147 We were using pod_id in a lot of place, where really it is just the pod name. I've renamed it, where it is easy to do so, so things are easier to follow. -- This is an automated message

[GitHub] [airflow] josh-fell commented on a diff in pull request #29143: Demonstrate usage of the PythonSensor

2023-01-24 Thread via GitHub
josh-fell commented on code in PR #29143: URL: https://github.com/apache/airflow/pull/29143#discussion_r1086144868 ## docs/apache-airflow/howto/operator/python.rst: ## @@ -225,11 +225,29 @@ Jinja templating can be used in same way as described for the PythonOperator.

[GitHub] [airflow] josh-fell commented on a diff in pull request #29143: Demonstrate usage of the PythonSensor

2023-01-24 Thread via GitHub
josh-fell commented on code in PR #29143: URL: https://github.com/apache/airflow/pull/29143#discussion_r1086143815 ## docs/apache-airflow/howto/operator/python.rst: ## @@ -225,11 +225,29 @@ Jinja templating can be used in same way as described for the PythonOperator.

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #29146: Add ability to access context in functions decorated by task.sensor

2023-01-24 Thread boring-cyborg
boring-cyborg[bot] commented on PR #29146: URL: https://github.com/apache/airflow/pull/29146#issuecomment-1402983470 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution

[GitHub] [airflow] Thomas-McKanna opened a new pull request, #29146: Add ability to access context in functions decorated by task.sensor

2023-01-24 Thread via GitHub
Thomas-McKanna opened a new pull request, #29146: URL: https://github.com/apache/airflow/pull/29146 Previous implementation did not properly expose Airflow context in functions decorated by `airflow.decorators.task.sensor`:

[GitHub] [airflow] Taragolis commented on pull request #29144: rewrite invoke lamba operator tests

2023-01-24 Thread via GitHub
Taragolis commented on PR #29144: URL: https://github.com/apache/airflow/pull/29144#issuecomment-1402930439 I don't know about removal `moto` in this test, because it is a good framework for testing something more close to AWS implementation. Basically it just wrapper around

[GitHub] [airflow] ferruzzi commented on a diff in pull request #29144: rewrite invoke lamba operator tests

2023-01-24 Thread via GitHub
ferruzzi commented on code in PR #29144: URL: https://github.com/apache/airflow/pull/29144#discussion_r1086120021 ## tests/providers/amazon/aws/operators/test_lambda_function.py: ## @@ -90,65 +84,69 @@ def test_init(self): assert lambda_operator.log_type == "None"

[GitHub] [airflow] Taragolis opened a new pull request, #29145: wip: Attempt to fix integration with codecov

2023-01-24 Thread via GitHub
Taragolis opened a new pull request, #29145: URL: https://github.com/apache/airflow/pull/29145 Change some setting for coverage and CMD arguments. No idea is help or not but right now codecov doesn't happy about our reports

[GitHub] [airflow] dstandish commented on pull request #29117: GCSTaskHandler may use remote log conn id

2023-01-24 Thread via GitHub
dstandish commented on PR #29117: URL: https://github.com/apache/airflow/pull/29117#issuecomment-1402916188 I think it's ok. I've tested it locally and it works and, it is only invoked if you set the conn id so, should be ok 爛 -- This is an automated message from the Apache Git

[GitHub] [airflow] vandonr-amz opened a new pull request, #29144: rewrite invoke lamba operator tests

2023-01-24 Thread via GitHub
vandonr-amz opened a new pull request, #29144: URL: https://github.com/apache/airflow/pull/29144 there was one big test that was mostly testing moto's implementation, and also it was taking a long time and failing on my machine. I replaced it with a test that mocks the hook as we

[GitHub] [airflow] jedcunningham commented on pull request #29136: Dataproc batches

2023-01-24 Thread via GitHub
jedcunningham commented on PR #29136: URL: https://github.com/apache/airflow/pull/29136#issuecomment-1402886139 Thanks for the PR @kristopherkane! Looks like static checks are failing. I'd also suggest that you [set up the pre-commit

[GitHub] [airflow] github-actions[bot] commented on pull request #23560: Add advanced secrets backend configurations

2023-01-24 Thread github-actions
github-actions[bot] commented on PR #23560: URL: https://github.com/apache/airflow/pull/23560#issuecomment-1402861209 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for

[GitHub] [airflow] github-actions[bot] closed pull request #27247: Added extra annotations support

2023-01-24 Thread github-actions
github-actions[bot] closed pull request #27247: Added extra annotations support URL: https://github.com/apache/airflow/pull/27247 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [airflow] github-actions[bot] closed pull request #28056: Move GCP Sheets system tests

2023-01-24 Thread github-actions
github-actions[bot] closed pull request #28056: Move GCP Sheets system tests URL: https://github.com/apache/airflow/pull/28056 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [airflow] github-actions[bot] commented on pull request #28123: Fix SFTP Sensor fails to locate file

2023-01-24 Thread github-actions
github-actions[bot] commented on PR #28123: URL: https://github.com/apache/airflow/pull/28123#issuecomment-1402861028 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for

[airflow] branch main updated: introduce a method to convert dictionaries to boto-style key-value lists (#28816)

2023-01-24 Thread onikolas
This is an automated email from the ASF dual-hosted git repository. onikolas pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 2c4928da40 introduce a method to convert

[GitHub] [airflow] o-nikolas merged pull request #28816: introduce a method to convert dictionaries to boto-style key-value lists

2023-01-24 Thread via GitHub
o-nikolas merged PR #28816: URL: https://github.com/apache/airflow/pull/28816 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] mik-laj commented on pull request #29117: GCSTaskHandler may use remote log conn id

2023-01-24 Thread via GitHub
mik-laj commented on PR #29117: URL: https://github.com/apache/airflow/pull/29117#issuecomment-1402832503 I don't remember the details of why this caused problems and we decided to remove hook support, but I would check if the secret backends are causing problems here. -- This is an

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #28869: rewrite polling code for appflow hook

2023-01-24 Thread via GitHub
vandonr-amz commented on code in PR #28869: URL: https://github.com/apache/airflow/pull/28869#discussion_r1086049943 ## airflow/providers/amazon/aws/operators/appflow.py: ## @@ -93,6 +98,12 @@ def execute(self, context: Context) -> None: self.connector_type =

[airflow] branch main updated: GCSTaskHandler may use remote log conn id (#29117)

2023-01-24 Thread dstandish
This is an automated email from the ASF dual-hosted git repository. dstandish pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new b4c50dadd3 GCSTaskHandler may use remote log

[GitHub] [airflow] dstandish merged pull request #29117: GCSTaskHandler may use remote log conn id

2023-01-24 Thread via GitHub
dstandish merged PR #29117: URL: https://github.com/apache/airflow/pull/29117 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] Taragolis commented on pull request #29087: Unquarantine receive SIGTERM on Task Runner test (second attempt)

2023-01-24 Thread via GitHub
Taragolis commented on PR #29087: URL: https://github.com/apache/airflow/pull/29087#issuecomment-1402820966 Yet another rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [airflow] travis-cook-sfdc commented on issue #28870: Support multiple buttons from `operator_extra_links`

2023-01-24 Thread via GitHub
travis-cook-sfdc commented on issue #28870: URL: https://github.com/apache/airflow/issues/28870#issuecomment-1402800228 I'm interesting in potentially implementing this change, but it's not clear to me exactly what the implementation should look like. Can anyone provide a summary?

[GitHub] [airflow] Taragolis commented on pull request #29142: Decrypt SecureString value obtained by SsmHook

2023-01-24 Thread via GitHub
Taragolis commented on PR #29142: URL: https://github.com/apache/airflow/pull/29142#issuecomment-1402786587 Since we cant be sure where it password and where it is not than better mask all. I will ad it tomorrow. -- This is an automated message from the Apache Git Service. To respond

[GitHub] [airflow] zachliu closed issue #29138: Auto refreshing (or auto tail) of logs in Web UI is erroneous

2023-01-24 Thread via GitHub
zachliu closed issue #29138: Auto refreshing (or auto tail) of logs in Web UI is erroneous URL: https://github.com/apache/airflow/issues/29138 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] zachliu commented on issue #29138: Auto refreshing (or auto tail) of logs in Web UI is erroneous

2023-01-24 Thread via GitHub
zachliu commented on issue #29138: URL: https://github.com/apache/airflow/issues/29138#issuecomment-1402776235 oops, didn't reconcile my own derived class with that [PR](https://github.com/apache/airflow/pull/26169) :sweat_smile: i have a `PatchedS3TaskHandler` to always read from

[GitHub] [airflow] Taragolis commented on issue #29138: Auto refreshing (or auto tail) of logs in Web UI is erroneous

2023-01-24 Thread via GitHub
Taragolis commented on issue #29138: URL: https://github.com/apache/airflow/issues/29138#issuecomment-1402766815 Just wondering are you use `EcsRunTaskOperator` with enable fetching CloudWatch logs to Airflow tasks? -- This is an automated message from the Apache Git Service. To

[GitHub] [airflow] ferruzzi commented on pull request #29142: Decrypt SecureString value obtained by SsmHook

2023-01-24 Thread via GitHub
ferruzzi commented on PR #29142: URL: https://github.com/apache/airflow/pull/29142#issuecomment-1402759146 > > Nice catch! IMHO, if we are decoding by default then masking sounds like the right answer to me. I'm not really up to date on best practices when using SecureString though, so I'm

[GitHub] [airflow] BasPH opened a new pull request, #29143: Demonstrate usage of the PythonSensor

2023-01-24 Thread via GitHub
BasPH opened a new pull request, #29143: URL: https://github.com/apache/airflow/pull/29143 Update of the https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/python.html#pythonsensor paragraph. All other paragraphs in the doc also demonstrate the taskflow-style operator,

[GitHub] [airflow] Taragolis commented on pull request #29142: Decrypt SecureString value obtained by SsmHook

2023-01-24 Thread via GitHub
Taragolis commented on PR #29142: URL: https://github.com/apache/airflow/pull/29142#issuecomment-1402747883 > Nice catch! IMHO, if we are decoding by default then masking sounds like the right answer to me. I'm not really up to date on best practices when using SecureString though, so I'm

[GitHub] [airflow] zachliu commented on issue #29138: Auto refreshing (or auto tail) of logs in Web UI is erroneous

2023-01-24 Thread via GitHub
zachliu commented on issue #29138: URL: https://github.com/apache/airflow/issues/29138#issuecomment-1402735638 i guess throttling would just make it refresh less frequent? the `example_sensors` dag has some long-running tasks (> 1min) that can clearly show this behavior no

[GitHub] [airflow] Taragolis opened a new pull request, #29142: Decrypt SecureString value obtained by SsmHook

2023-01-24 Thread via GitHub
Taragolis opened a new pull request, #29142: URL: https://github.com/apache/airflow/pull/29142 Right now if we try to get SecureString by `SsmHook.get_parameter_value` then we get encrypted value, which is useless for further usage without decryption. ```python from

[GitHub] [airflow] kristopherkane commented on issue #29109: [Google Cloud] DataprocCreateBatchOperator returns incorrect results and does not reattach

2023-01-24 Thread via GitHub
kristopherkane commented on issue #29109: URL: https://github.com/apache/airflow/issues/29109#issuecomment-1402672278 cc @MaksYermak for additional review as the original author. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [airflow] loulsb commented on pull request #29116: Add colors in help outputs of Airfow CLI commands #28789

2023-01-24 Thread via GitHub
loulsb commented on PR #29116: URL: https://github.com/apache/airflow/pull/29116#issuecomment-1402667103 I don't know if there is a specific tool that you use for performance testing, I ran "time airflow cheat-sheet" a bunch of times without the change and with the change and I got this:

[GitHub] [airflow] Taragolis commented on issue #29131: Airflow v2.4.2 - Scheduler fails with BrokenPIPE error (Exiting gracefully upon receiving signal 15)

2023-01-24 Thread via GitHub
Taragolis commented on issue #29131: URL: https://github.com/apache/airflow/issues/29131#issuecomment-1402640500 More like deployment troubleshooting rather then actual issue. Convert to discussion -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [airflow] Taragolis closed issue #29131: Airflow v2.4.2 - Scheduler fails with BrokenPIPE error (Exiting gracefully upon receiving signal 15)

2023-01-24 Thread via GitHub
Taragolis closed issue #29131: Airflow v2.4.2 - Scheduler fails with BrokenPIPE error (Exiting gracefully upon receiving signal 15) URL: https://github.com/apache/airflow/issues/29131 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [airflow] Taragolis commented on issue #29138: Auto refreshing (or auto tail) of logs in Web UI is erroneous

2023-01-24 Thread via GitHub
Taragolis commented on issue #29138: URL: https://github.com/apache/airflow/issues/29138#issuecomment-1402638923 It also might potentially fixed by https://github.com/apache/airflow/pull/28818 Could you provide some simple DAG which might reproduce this behaviour? And do you use

[GitHub] [airflow] jedcunningham commented on a diff in pull request #29058: Optionally export `airflow db clean` data to CSV files

2023-01-24 Thread via GitHub
jedcunningham commented on code in PR #29058: URL: https://github.com/apache/airflow/pull/29058#discussion_r1085907128 ## airflow/utils/db_cleanup.py: ## @@ -159,6 +170,14 @@ def _do_delete(*, query, orm_model, skip_archive, session): logger.debug("delete statement:\n%s",

[GitHub] [airflow] jedcunningham commented on issue #28779: Built-in ServiceMonitor for PgBouncer and StatsD

2023-01-24 Thread via GitHub
jedcunningham commented on issue #28779: URL: https://github.com/apache/airflow/issues/28779#issuecomment-1402634049 Sounds good to me. Does PgBouncer have a prometheus endpoint to point at though, or would this work include adding the community exporter? -- This is an automated message

[GitHub] [airflow] o-nikolas commented on a diff in pull request #29055: [AIP-51] Executors vending CLI commands

2023-01-24 Thread via GitHub
o-nikolas commented on code in PR #29055: URL: https://github.com/apache/airflow/pull/29055#discussion_r1085901682 ## airflow/cli/cli_parser.py: ## @@ -17,2165 +17,35 @@ # specific language governing permissions and limitations # under the License. """Command-line

[GitHub] [airflow] Taragolis commented on issue #29137: Add ability to access context in functions decorated by task.sensor

2023-01-24 Thread via GitHub
Taragolis commented on issue #29137: URL: https://github.com/apache/airflow/issues/29137#issuecomment-1402621242 Feel free to make a PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[airflow] branch main updated: AIP-51 - Executor Coupling in Logging (#28161)

2023-01-24 Thread onikolas
This is an automated email from the ASF dual-hosted git repository. onikolas pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 3b25168c41 AIP-51 - Executor Coupling in Logging

[GitHub] [airflow] o-nikolas merged pull request #28161: AIP-51 - Executor Coupling in Logging

2023-01-24 Thread via GitHub
o-nikolas merged PR #28161: URL: https://github.com/apache/airflow/pull/28161 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] o-nikolas closed issue #27931: AIP-51 - Executor Coupling in Logging

2023-01-24 Thread via GitHub
o-nikolas closed issue #27931: AIP-51 - Executor Coupling in Logging URL: https://github.com/apache/airflow/issues/27931 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [airflow] Taragolis commented on issue #29131: Airflow v2.4.2 - Scheduler fails with BrokenPIPE error (Exiting gracefully upon receiving signal 15)

2023-01-24 Thread via GitHub
Taragolis commented on issue #29131: URL: https://github.com/apache/airflow/issues/29131#issuecomment-1402606802 You could try to check by new [Scheduler Health Check

[GitHub] [airflow] zachliu commented on issue #29138: Auto refreshing (or auto tail) of logs in Web UI is erroneous

2023-01-24 Thread via GitHub
zachliu commented on issue #29138: URL: https://github.com/apache/airflow/issues/29138#issuecomment-1402586173 might be related to https://github.com/apache/airflow/pull/26169 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #29139: Deletion of schema, login, password, port

2023-01-24 Thread boring-cyborg
boring-cyborg[bot] commented on PR #29139: URL: https://github.com/apache/airflow/pull/29139#issuecomment-1402584893 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution

[GitHub] [airflow] atharvmishra1999 opened a new pull request, #29139: Deletion of schema, login, password, port

2023-01-24 Thread via GitHub
atharvmishra1999 opened a new pull request, #29139: URL: https://github.com/apache/airflow/pull/29139 Issue #29113 Deleted schema, login, password, port from docs to resolve the issue. --- **^ Add meaningful description above** Read the **[Pull Request

[GitHub] [airflow] zachliu opened a new issue, #29138: Auto refreshing (or auto tail) of logs in Web UI is erroneous

2023-01-24 Thread via GitHub
zachliu opened a new issue, #29138: URL: https://github.com/apache/airflow/issues/29138 ### Apache Airflow version 2.5.1 ### What happened the entire log section keeps being appended to the existing logs, making this page grow like crazy

[GitHub] [airflow] Thomas-McKanna opened a new issue, #29137: Add ability to access context in functions decorated by task.sensor

2023-01-24 Thread via GitHub
Thomas-McKanna opened a new issue, #29137: URL: https://github.com/apache/airflow/issues/29137 ### Description Hello, I am a new Airflow user. I am requesting a feature in which the airflow context (containing task instance, etc.) be available inside of functions decorated by

[GitHub] [airflow] boring-cyborg[bot] commented on issue #29137: Add ability to access context in functions decorated by task.sensor

2023-01-24 Thread boring-cyborg
boring-cyborg[bot] commented on issue #29137: URL: https://github.com/apache/airflow/issues/29137#issuecomment-1402580643 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] tanuj241088 commented on issue #29131: Airflow v2.4.2 - Scheduler fails with BrokenPIPE error (Exiting gracefully upon receiving signal 15)

2023-01-24 Thread via GitHub
tanuj241088 commented on issue #29131: URL: https://github.com/apache/airflow/issues/29131#issuecomment-1402557912 @Taragolis ... i was also investigating in the same direction. Here is the update: i dont think this is related to Airflow version upgrade. We do have liveliness

[GitHub] [airflow] Taragolis commented on issue #29131: Airflow v2.4.2 - Scheduler fails with BrokenPIPE error (Exiting gracefully upon receiving signal 15)

2023-01-24 Thread via GitHub
Taragolis commented on issue #29131: URL: https://github.com/apache/airflow/issues/29131#issuecomment-1402521976 Do you have any health checks? This might be a main reason why your scheduler terminated. -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] jedcunningham commented on a diff in pull request #29135: Keyfile dict can be dict not str

2023-01-24 Thread via GitHub
jedcunningham commented on code in PR #29135: URL: https://github.com/apache/airflow/pull/29135#discussion_r1085847354 ## airflow/providers/google/common/hooks/base_google.py: ## @@ -491,7 +494,7 @@ def provide_gcp_credential_file_as_context(self) -> Generator[str | None,

[GitHub] [airflow] Taragolis commented on issue #29133: Airflow triggered the upstream task even the downstream task not executed completely.

2023-01-24 Thread via GitHub
Taragolis commented on issue #29133: URL: https://github.com/apache/airflow/issues/29133#issuecomment-1402507768 1. Can you share the DAG code? You can redact some sensitive values or something that you don't want to expose to the public. Without this it would be problematic reproduce.

[GitHub] [airflow] kristopherkane opened a new pull request, #29136: Dataproc batches

2023-01-24 Thread via GitHub
kristopherkane opened a new pull request, #29136: URL: https://github.com/apache/airflow/pull/29136 #29109 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

[airflow] branch main updated (33ba242d7e -> 1fbfd312d9)

2023-01-24 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 33ba242d7e Clarify graph view dynamic task labels (#29042) add 1fbfd312d9 Fix ftp example broken docs link

[GitHub] [airflow] potiuk merged pull request #29134: Fix ftp example broken docs link

2023-01-24 Thread via GitHub
potiuk merged PR #29134: URL: https://github.com/apache/airflow/pull/29134 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] potiuk closed issue #29128: [Doc] Link to examples how to use FTP provider is incorrect

2023-01-24 Thread via GitHub
potiuk closed issue #29128: [Doc] Link to examples how to use FTP provider is incorrect URL: https://github.com/apache/airflow/issues/29128 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] eladkal commented on pull request #28952: Add release flag to Breeze `generate-issue-content`

2023-01-24 Thread via GitHub
eladkal commented on PR #28952: URL: https://github.com/apache/airflow/pull/28952#issuecomment-1402460223 So we can't use the same function because core/helm have different parameters than providers. so I dropped the idea of artifact flag. First step is to change

[GitHub] [airflow] dstandish commented on a diff in pull request #29135: Keyfile dict can be dict not str

2023-01-24 Thread via GitHub
dstandish commented on code in PR #29135: URL: https://github.com/apache/airflow/pull/29135#discussion_r1085761659 ## airflow/providers/google/common/hooks/base_google.py: ## @@ -491,7 +494,7 @@ def provide_gcp_credential_file_as_context(self) -> Generator[str | None, None,

[GitHub] [airflow] jedcunningham commented on a diff in pull request #29135: Keyfile dict can be dict not str

2023-01-24 Thread via GitHub
jedcunningham commented on code in PR #29135: URL: https://github.com/apache/airflow/pull/29135#discussion_r1085734976 ## airflow/providers/google/common/hooks/base_google.py: ## @@ -242,7 +242,10 @@ def get_credentials_and_project_id(self) ->

[GitHub] [airflow] dstandish commented on pull request #29117: GCSTaskHandler may use remote log conn id

2023-01-24 Thread via GitHub
dstandish commented on PR #29117: URL: https://github.com/apache/airflow/pull/29117#issuecomment-1402420531 @mik-laj any reason not to add this back? i know you removed it in the past. for hosted service, you don't necessarily want ADC to be used, and, it may not be feasible / desirable

[GitHub] [airflow] jedcunningham commented on a diff in pull request #29135: Keyfile dict can be dict not str

2023-01-24 Thread via GitHub
jedcunningham commented on code in PR #29135: URL: https://github.com/apache/airflow/pull/29135#discussion_r1085734976 ## airflow/providers/google/common/hooks/base_google.py: ## @@ -242,7 +242,10 @@ def get_credentials_and_project_id(self) ->

[airflow-site] branch gh-pages updated: Deploying to gh-pages from @ 4dbd4b800383d30d28e75d20d0a3515f70bb07e9 

2023-01-24 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch gh-pages in repository https://gitbox.apache.org/repos/asf/airflow-site.git The following commit(s) were added to refs/heads/gh-pages by this push: new 51e208d79d Deploying to gh-pages

[GitHub] [airflow] dstandish opened a new pull request, #29135: Keyfile dict can be dict not str

2023-01-24 Thread via GitHub
dstandish opened a new pull request, #29135: URL: https://github.com/apache/airflow/pull/29135 This makes it much easier to define a connection with json. We can do this: ```json {"extra": {"keyfile_dict": {"foo": "bar", "private_key": "hi"}}} ``` Instead of this:

[GitHub] [airflow] pankajastro commented on issue #29128: [Doc] Link to examples how to use FTP provider is incorrect

2023-01-24 Thread via GitHub
pankajastro commented on issue #29128: URL: https://github.com/apache/airflow/issues/29128#issuecomment-1402389196 https://github.com/apache/airflow/pull/29134 should fix it. meanwhile, you can get the example from

[GitHub] [airflow] pankajastro opened a new pull request, #29134: Fix ftp example broken docs link

2023-01-24 Thread via GitHub
pankajastro opened a new pull request, #29134: URL: https://github.com/apache/airflow/pull/29134 closes: https://github.com/apache/airflow/issues/29128 --- **^ Add meaningful description above** Read the **[Pull Request

[airflow-site] branch main updated: Add Amazon provider package health status dashboard in ecosystem page (#729)

2023-01-24 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow-site.git The following commit(s) were added to refs/heads/main by this push: new 4dbd4b8003 Add Amazon provider package health

[GitHub] [airflow-site] potiuk merged pull request #729: Add Amazon provider package health status dashboard in ecosystem page

2023-01-24 Thread via GitHub
potiuk merged PR #729: URL: https://github.com/apache/airflow-site/pull/729 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] ashish1471 opened a new issue, #29133: Airflow triggered the upstream task even the downstream task not executed completely.

2023-01-24 Thread via GitHub
ashish1471 opened a new issue, #29133: URL: https://github.com/apache/airflow/issues/29133 ### Apache Airflow version Other Airflow 2 version (please specify below) ### What happened Hey Team, I am not sure where I am doing wrong. I am running a DAG and I have set

[GitHub] [airflow] boring-cyborg[bot] commented on issue #29133: Airflow triggered the upstream task even the downstream task not executed completely.

2023-01-24 Thread boring-cyborg
boring-cyborg[bot] commented on issue #29133: URL: https://github.com/apache/airflow/issues/29133#issuecomment-1402368487 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #29132: Allow PATCH role endpoint to remove permissions not included in PATCH call

2023-01-24 Thread boring-cyborg
boring-cyborg[bot] commented on PR #29132: URL: https://github.com/apache/airflow/pull/29132#issuecomment-1402347138 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution

[GitHub] [airflow] tseruga opened a new pull request, #29132: Allow PATCH role endpoint to remove permissions not included in PATCH call

2023-01-24 Thread via GitHub
tseruga opened a new pull request, #29132: URL: https://github.com/apache/airflow/pull/29132 closes: #25734 Rationale: `PATCH /api/v1/roles/[role]` previously would only ever append permissions to a role, never removing permissions that were omitted from the `PATCH` call. This

[GitHub] [airflow] shubham22 commented on issue #28289: Add AWS Neptune hook and operators

2023-01-24 Thread via GitHub
shubham22 commented on issue #28289: URL: https://github.com/apache/airflow/issues/28289#issuecomment-1402345265 @swapz-z - are you still looking into this? Let us know if you need any help. cc: @o-nikolas -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] kristopherkane closed pull request #29111: Dataproc batches

2023-01-24 Thread via GitHub
kristopherkane closed pull request #29111: Dataproc batches URL: https://github.com/apache/airflow/pull/29111 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

[GitHub] [airflow] eladkal commented on issue #29127: Use latest azure-mgmt-containerinstance (10.0.0) which depricates network_profile and adds subnet_ids

2023-01-24 Thread via GitHub
eladkal commented on issue #29127: URL: https://github.com/apache/airflow/issues/29127#issuecomment-1402235462 This is something waiting for one of Azure users to do it

[GitHub] [airflow] jedcunningham commented on a diff in pull request #29058: Optionally export `airflow db clean` data to CSV files

2023-01-24 Thread via GitHub
jedcunningham commented on code in PR #29058: URL: https://github.com/apache/airflow/pull/29058#discussion_r1085539467 ## airflow/utils/db_cleanup.py: ## @@ -123,7 +126,15 @@ def _check_for_rows(*, query: Query, print_rows=False): return num_entities -def _do_delete(*,

  1   2   >