[GitHub] [airflow] uranusjr commented on a diff in pull request #31854: Allow variables to be printed to STDOUT with airflow variables export.

2023-06-13 Thread via GitHub
uranusjr commented on code in PR #31854: URL: https://github.com/apache/airflow/pull/31854#discussion_r1229100836 ## airflow/cli/commands/variable_command.py: ## @@ -76,7 +77,24 @@ def variables_import(args): def variables_export(args): """Exports all the variables to th

[GitHub] [airflow] phanikumv commented on a diff in pull request #31872: Optimize deferrable mode execution for `BigQueryValueCheckOperator`

2023-06-13 Thread via GitHub
phanikumv commented on code in PR #31872: URL: https://github.com/apache/airflow/pull/31872#discussion_r1229102226 ## airflow/providers/google/cloud/operators/bigquery.py: ## @@ -364,19 +364,21 @@ def execute(self, context: Context) -> None: # type: ignore[override]

[GitHub] [airflow] uranusjr commented on a diff in pull request #31854: Allow variables to be printed to STDOUT with airflow variables export.

2023-06-13 Thread via GitHub
uranusjr commented on code in PR #31854: URL: https://github.com/apache/airflow/pull/31854#discussion_r1229100836 ## airflow/cli/commands/variable_command.py: ## @@ -76,7 +77,24 @@ def variables_import(args): def variables_export(args): """Exports all the variables to th

[GitHub] [airflow] phanikumv opened a new pull request, #31888: `DatabricksSubmitRunOperator` : Fix type annotation for `libraries` param

2023-06-13 Thread via GitHub
phanikumv opened a new pull request, #31888: URL: https://github.com/apache/airflow/pull/31888 closes: #31877 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-gui

[GitHub] [airflow] bkossakowska commented on pull request #31284: Add deferrable mode to PubsubPullSensor

2023-06-13 Thread via GitHub
bkossakowska commented on PR #31284: URL: https://github.com/apache/airflow/pull/31284#issuecomment-1590565278 Hi @phanikumv, @potiuk thanks for your comments, changes have been added. Is there anything else I can improve? -- This is an automated message from the Apache Git Service. To

[GitHub] [airflow] bkossakowska commented on a diff in pull request #31457: Fix BIGQUERY_JOB_DETAILS_LINK_FMT in BigQueryConsoleLink

2023-06-13 Thread via GitHub
bkossakowska commented on code in PR #31457: URL: https://github.com/apache/airflow/pull/31457#discussion_r1229084939 ## airflow/providers/google/cloud/operators/bigquery.py: ## @@ -1181,7 +1191,13 @@ def execute(self, context: Context): ] else:

[GitHub] [airflow] uranusjr commented on a diff in pull request #31855: Add Redis task handler

2023-06-13 Thread via GitHub
uranusjr commented on code in PR #31855: URL: https://github.com/apache/airflow/pull/31855#discussion_r1229078343 ## airflow/utils/log/file_task_handler.py: ## @@ -147,7 +147,7 @@ class FileTaskHandler(logging.Handler): def __init__(self, base_log_folder: str, filename_te

[GitHub] [airflow] mrn-aglic commented on issue #31584: BigQueryInsertJobOperator not exiting deferred state

2023-06-13 Thread via GitHub
mrn-aglic commented on issue #31584: URL: https://github.com/apache/airflow/issues/31584#issuecomment-1590521233 @phanikumv Nop, I didn't get any wheel file @pankajastro mentioned. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

[GitHub] [airflow] utkarsharma2 commented on pull request #31887: Adapt Notifier for sla_miss_callback

2023-06-13 Thread via GitHub
utkarsharma2 commented on PR #31887: URL: https://github.com/apache/airflow/pull/31887#issuecomment-1590515248 @ephraimbuddy Please review :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

[GitHub] [airflow] michalc commented on a diff in pull request #31855: Add Redis task handler

2023-06-13 Thread via GitHub
michalc commented on code in PR #31855: URL: https://github.com/apache/airflow/pull/31855#discussion_r122915 ## airflow/utils/log/file_task_handler.py: ## @@ -147,7 +147,7 @@ class FileTaskHandler(logging.Handler): def __init__(self, base_log_folder: str, filename_tem

[GitHub] [airflow] michalc commented on a diff in pull request #31855: Add Redis task handler

2023-06-13 Thread via GitHub
michalc commented on code in PR #31855: URL: https://github.com/apache/airflow/pull/31855#discussion_r122915 ## airflow/utils/log/file_task_handler.py: ## @@ -147,7 +147,7 @@ class FileTaskHandler(logging.Handler): def __init__(self, base_log_folder: str, filename_tem

[GitHub] [airflow] Aakcht commented on pull request #31865: Chart: Propagate global pod security context to statsd and redis

2023-06-13 Thread via GitHub
Aakcht commented on PR #31865: URL: https://github.com/apache/airflow/pull/31865#issuecomment-1590482722 @jedcunningham actually statsd is only half excluded from using globals - since `Values.securityContexts.containers` is getting propagated without issues to statsd container (note how [

[GitHub] [airflow] eladkal commented on pull request #31885: Allow a destination folder to be provided

2023-06-13 Thread via GitHub
eladkal commented on PR #31885: URL: https://github.com/apache/airflow/pull/31885#issuecomment-1590444084 cc @shahar1 WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [airflow] stiak commented on pull request #31885: Allow a destination folder to be provided

2023-06-13 Thread via GitHub
stiak commented on PR #31885: URL: https://github.com/apache/airflow/pull/31885#issuecomment-1590436213 I've added an example for "Copy into an existing folder". I ended up using this pattern for the folder id: ```python FOLDER_ID = os.environ.get("GCP_GDRIVE_FOLDER_ID", "abcd1234")

[GitHub] [airflow] phanikumv commented on pull request #31885: Allow a destination folder to be provided

2023-06-13 Thread via GitHub
phanikumv commented on PR #31885: URL: https://github.com/apache/airflow/pull/31885#issuecomment-1590428789 > > Please add an example for the destination folder movement(similar to `howto_operator_gcs_to_gdrive_move_files`) , and explain this new functionality in `gcs_to_gdrive.rst`. You ma

[GitHub] [airflow] ferruzzi commented on a diff in pull request #31859: AIP-49 OTel Timers

2023-06-13 Thread via GitHub
ferruzzi commented on code in PR #31859: URL: https://github.com/apache/airflow/pull/31859#discussion_r1228953001 ## airflow/metrics/otel_logger.py: ## @@ -233,27 +255,32 @@ def gauge( if self.metrics_validator.test(stat): self.metrics_map.set_gauge_value(f

[GitHub] [airflow] jedcunningham commented on a diff in pull request #31859: AIP-49 OTel Timers

2023-06-13 Thread via GitHub
jedcunningham commented on code in PR #31859: URL: https://github.com/apache/airflow/pull/31859#discussion_r1228942632 ## airflow/metrics/otel_logger.py: ## @@ -131,6 +131,29 @@ def _skip_due_to_rate(rate: float) -> bool: return rate < 1 and random.random() > rate +clas

[GitHub] [airflow] doiken commented on pull request #30523: Fix sla checks being skipped when the later task instance is skipped

2023-06-13 Thread via GitHub
doiken commented on PR #30523: URL: https://github.com/apache/airflow/pull/30523#issuecomment-1590409370 There is a conflicting file, but I will leave it as is for stale. If there is a demand for this PR, please let me know so I can fix it. -- This is an automated message from the Apach

[GitHub] [airflow] doiken commented on pull request #31414: Fix future DagRun rarely triggered by race conditions when max_active_runs reached its upper limit.

2023-06-13 Thread via GitHub
doiken commented on PR #31414: URL: https://github.com/apache/airflow/pull/31414#issuecomment-1590409213 It has been a while since our last conversation, is there anything I can do about this PR? -- This is an automated message from the Apache Git Service. To respond to the message, ple

[GitHub] [airflow] jedcunningham commented on a diff in pull request #31882: Update documentation for constraints installation

2023-06-13 Thread via GitHub
jedcunningham commented on code in PR #31882: URL: https://github.com/apache/airflow/pull/31882#discussion_r1228939074 ## docs/apache-airflow/installation/installing-from-pypi.rst: ## @@ -79,50 +87,86 @@ where: - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version

[GitHub] [airflow] stiak commented on pull request #31885: Allow a destination folder to be provided

2023-06-13 Thread via GitHub
stiak commented on PR #31885: URL: https://github.com/apache/airflow/pull/31885#issuecomment-1590407718 > Please add an example for the destination folder movement(similar to `howto_operator_gcs_to_gdrive_move_files`) , and explain this new functionality in `gcs_to_gdrive.rst`. You may also

[GitHub] [airflow] jedcunningham commented on pull request #31868: Runtime class name

2023-06-13 Thread via GitHub
jedcunningham commented on PR #31868: URL: https://github.com/apache/airflow/pull/31868#issuecomment-1590389960 Also don't forget to add this to `values.yaml` and `files/pod-template-file.kubernetes-helm-yaml`. -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] jedcunningham commented on pull request #31779: Supporting extraAnnotions at pgbouncer deployment level

2023-06-13 Thread via GitHub
jedcunningham commented on PR #31779: URL: https://github.com/apache/airflow/pull/31779#issuecomment-1590385206 How is this different than `pgbouncer.podAnnotations`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [airflow] jedcunningham commented on pull request #31865: Chart: Propagate global pod security context to statsd deployment

2023-06-13 Thread via GitHub
jedcunningham commented on PR #31865: URL: https://github.com/apache/airflow/pull/31865#issuecomment-1590379681 Actually statsd and redis were intentionally excluded from using globals, note how those 2 are the ones using `localPodSecurityContext`. They also aren't "airflow" pods ultimately

[airflow] branch main updated: Align apiVersion and kind order in chart templates (#31850)

2023-06-13 Thread jedcunningham
This is an automated email from the ASF dual-hosted git repository. jedcunningham pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 7e7add0bf4 Align apiVersion and kind order i

[GitHub] [airflow] jedcunningham merged pull request #31850: Align `apiVersion` and `kind` order in chart templates

2023-06-13 Thread via GitHub
jedcunningham merged PR #31850: URL: https://github.com/apache/airflow/pull/31850 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@ai

[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #31885: Allow a destination folder to be provided

2023-06-13 Thread via GitHub
utkarsharma2 commented on code in PR #31885: URL: https://github.com/apache/airflow/pull/31885#discussion_r1228905706 ## airflow/providers/google/suite/transfers/gcs_to_gdrive.py: ## @@ -63,12 +63,12 @@ class GCSToGoogleDriveOperator(BaseOperator): For example, with pre

[GitHub] [airflow] mashirali opened a new issue, #31886: Azure ML integration issue

2023-06-13 Thread via GitHub
mashirali opened a new issue, #31886: URL: https://github.com/apache/airflow/issues/31886 ### Apache Airflow version 2.6.1 ### What happened I was trying to add azure ml workflows in the dags but got an error ### What you think should happen instead Everythi

[GitHub] [airflow] stiak commented on a diff in pull request #31885: Allow a destination folder to be provided

2023-06-13 Thread via GitHub
stiak commented on code in PR #31885: URL: https://github.com/apache/airflow/pull/31885#discussion_r1228896190 ## airflow/providers/google/suite/transfers/gcs_to_gdrive.py: ## @@ -63,12 +63,12 @@ class GCSToGoogleDriveOperator(BaseOperator): For example, with prefix ``f

[GitHub] [airflow] utkarsharma2 commented on pull request #31880: Update breeze.py

2023-06-13 Thread via GitHub
utkarsharma2 commented on PR #31880: URL: https://github.com/apache/airflow/pull/31880#issuecomment-1590339612 @rohan472000 Could you please add the link to the ticket in the PR description? It gives us more context for the PR and helps in the review process :) Also, you can reach ou

[airflow] branch main updated: Add deferrable mode to `BatchSensor` (#30279)

2023-06-13 Thread pankaj
This is an automated email from the ASF dual-hosted git repository. pankaj pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 688f91b330 Add deferrable mode to `BatchSensor` (#

[GitHub] [airflow] pankajastro merged pull request #30279: Add deferrable mode to `BatchSensor`

2023-06-13 Thread via GitHub
pankajastro merged PR #30279: URL: https://github.com/apache/airflow/pull/30279 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airf

[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #31885: Allow a destination folder to be provided

2023-06-13 Thread via GitHub
utkarsharma2 commented on code in PR #31885: URL: https://github.com/apache/airflow/pull/31885#discussion_r1228882844 ## tests/providers/google/suite/transfers/test_gcs_to_gdrive.py: ## @@ -64,11 +64,53 @@ def test_should_copy_single_file(self, mock_named_temporary_file, mock_g

[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #31885: Allow a destination folder to be provided

2023-06-13 Thread via GitHub
utkarsharma2 commented on code in PR #31885: URL: https://github.com/apache/airflow/pull/31885#discussion_r1228882844 ## tests/providers/google/suite/transfers/test_gcs_to_gdrive.py: ## @@ -64,11 +64,53 @@ def test_should_copy_single_file(self, mock_named_temporary_file, mock_g

[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #31885: Allow a destination folder to be provided

2023-06-13 Thread via GitHub
utkarsharma2 commented on code in PR #31885: URL: https://github.com/apache/airflow/pull/31885#discussion_r1228881958 ## airflow/providers/google/suite/transfers/gcs_to_gdrive.py: ## @@ -63,12 +63,12 @@ class GCSToGoogleDriveOperator(BaseOperator): For example, with pre

[GitHub] [airflow] syedahsn closed pull request #31749: S3 Key sensor deferrable

2023-06-13 Thread via GitHub
syedahsn closed pull request #31749: S3 Key sensor deferrable URL: https://github.com/apache/airflow/pull/31749 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #31855: Add Redis task handler

2023-06-13 Thread via GitHub
utkarsharma2 commented on code in PR #31855: URL: https://github.com/apache/airflow/pull/31855#discussion_r1228875873 ## airflow/utils/log/file_task_handler.py: ## @@ -147,7 +147,7 @@ class FileTaskHandler(logging.Handler): def __init__(self, base_log_folder: str, filenam

[GitHub] [airflow] javatarz commented on issue #31875: Ability to sync airflow variables during deployments

2023-06-13 Thread via GitHub
javatarz commented on issue #31875: URL: https://github.com/apache/airflow/issues/31875#issuecomment-1590306133 > create a concatenation scripts This is precisely what I meant in my original post when I said "The individual building blocks for this change are already available. The (

[GitHub] [airflow] javatarz commented on issue #31875: Ability to sync airflow variables during deployments

2023-06-13 Thread via GitHub
javatarz commented on issue #31875: URL: https://github.com/apache/airflow/issues/31875#issuecomment-1590298835 Thank you for the detailed response @potiuk! There's a bunch of perspectives I hadn't thought of so I'll go through as much as I can answer now and for some of it, I'll have to co

[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #31854: Allow variables to be printed to STDOUT with airflow variables export.

2023-06-13 Thread via GitHub
utkarsharma2 commented on code in PR #31854: URL: https://github.com/apache/airflow/pull/31854#discussion_r1228848575 ## airflow/cli/commands/variable_command.py: ## @@ -76,7 +77,24 @@ def variables_import(args): def variables_export(args): """Exports all the variables t

[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #31854: Allow variables to be printed to STDOUT with airflow variables export.

2023-06-13 Thread via GitHub
utkarsharma2 commented on code in PR #31854: URL: https://github.com/apache/airflow/pull/31854#discussion_r1228847858 ## airflow/cli/commands/variable_command.py: ## @@ -76,7 +77,24 @@ def variables_import(args): def variables_export(args): """Exports all the variables t

[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #31854: Allow variables to be printed to STDOUT with airflow variables export.

2023-06-13 Thread via GitHub
utkarsharma2 commented on code in PR #31854: URL: https://github.com/apache/airflow/pull/31854#discussion_r1228846964 ## airflow/cli/commands/variable_command.py: ## @@ -76,7 +77,24 @@ def variables_import(args): def variables_export(args): """Exports all the variables t

[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #31854: Allow variables to be printed to STDOUT with airflow variables export.

2023-06-13 Thread via GitHub
utkarsharma2 commented on code in PR #31854: URL: https://github.com/apache/airflow/pull/31854#discussion_r1228846788 ## airflow/cli/commands/variable_command.py: ## @@ -76,7 +77,24 @@ def variables_import(args): def variables_export(args): """Exports all the variables t

[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #31854: Allow variables to be printed to STDOUT with airflow variables export.

2023-06-13 Thread via GitHub
utkarsharma2 commented on code in PR #31854: URL: https://github.com/apache/airflow/pull/31854#discussion_r1228845111 ## airflow/cli/commands/variable_command.py: ## @@ -76,7 +77,24 @@ def variables_import(args): def variables_export(args): """Exports all the variables t

[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #31854: Allow variables to be printed to STDOUT with airflow variables export.

2023-06-13 Thread via GitHub
utkarsharma2 commented on code in PR #31854: URL: https://github.com/apache/airflow/pull/31854#discussion_r1228845111 ## airflow/cli/commands/variable_command.py: ## @@ -76,7 +77,24 @@ def variables_import(args): def variables_export(args): """Exports all the variables t

[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #31854: Allow variables to be printed to STDOUT with airflow variables export.

2023-06-13 Thread via GitHub
utkarsharma2 commented on code in PR #31854: URL: https://github.com/apache/airflow/pull/31854#discussion_r1228845890 ## airflow/cli/commands/variable_command.py: ## @@ -76,7 +77,24 @@ def variables_import(args): def variables_export(args): """Exports all the variables t

[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #31854: Allow variables to be printed to STDOUT with airflow variables export.

2023-06-13 Thread via GitHub
utkarsharma2 commented on code in PR #31854: URL: https://github.com/apache/airflow/pull/31854#discussion_r1228845111 ## airflow/cli/commands/variable_command.py: ## @@ -76,7 +77,24 @@ def variables_import(args): def variables_export(args): """Exports all the variables t

[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #31854: Allow variables to be printed to STDOUT with airflow variables export.

2023-06-13 Thread via GitHub
utkarsharma2 commented on code in PR #31854: URL: https://github.com/apache/airflow/pull/31854#discussion_r1228845111 ## airflow/cli/commands/variable_command.py: ## @@ -76,7 +77,24 @@ def variables_import(args): def variables_export(args): """Exports all the variables t

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #31885: Allow a destination folder to be provided

2023-06-13 Thread via GitHub
boring-cyborg[bot] commented on PR #31885: URL: https://github.com/apache/airflow/pull/31885#issuecomment-1590261465 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution

[GitHub] [airflow] stiak opened a new pull request, #31885: Allow a destination folder to be provided

2023-06-13 Thread via GitHub
stiak opened a new pull request, #31885: URL: https://github.com/apache/airflow/pull/31885 With the move from `delegate_to` to `impersonation_chain` I'm no longer able to impersonate a user, only a service account. Google drive files created by a service account aren't visible to anyone el

[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #31836: Add --retry and --retry-delay to "airflow db check"

2023-06-13 Thread via GitHub
utkarsharma2 commented on code in PR #31836: URL: https://github.com/apache/airflow/pull/31836#discussion_r1228841822 ## airflow/cli/commands/db_command.py: ## @@ -187,9 +188,18 @@ def shell(args): @cli_utils.action_cli(check_db=False) -def check(_): +def check(args):

[GitHub] [airflow] vandonr-amz opened a new pull request, #31884: remove 'wait_for_completion' from ecs register/deregister operators

2023-06-13 Thread via GitHub
vandonr-amz opened a new pull request, #31884: URL: https://github.com/apache/airflow/pull/31884 those operations are actually completed instantly. the doc to deregister says it explicitely: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deregister-task-definition.html

[GitHub] [airflow] github-actions[bot] commented on pull request #30669: [WIP] Fix try_number calculation and add a new column poke_number

2023-06-13 Thread via GitHub
github-actions[bot] commented on PR #30669: URL: https://github.com/apache/airflow/pull/30669#issuecomment-1590236288 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for you

[GitHub] [airflow] potiuk commented on pull request #31882: Update documentation for constraints installation

2023-06-13 Thread via GitHub
potiuk commented on PR #31882: URL: https://github.com/apache/airflow/pull/31882#issuecomment-1590231730 First pass of review applied. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [airflow] potiuk commented on a diff in pull request #31882: Update documentation for constraints installation

2023-06-13 Thread via GitHub
potiuk commented on code in PR #31882: URL: https://github.com/apache/airflow/pull/31882#discussion_r1228825582 ## docs/apache-airflow/installation/installing-from-pypi.rst: ## @@ -79,53 +87,64 @@ where: - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or

[GitHub] [airflow] potiuk commented on a diff in pull request #31882: Update documentation for constraints installation

2023-06-13 Thread via GitHub
potiuk commented on code in PR #31882: URL: https://github.com/apache/airflow/pull/31882#discussion_r1228822923 ## docs/apache-airflow/installation/installing-from-pypi.rst: ## @@ -79,53 +87,64 @@ where: - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or

[GitHub] [airflow] potiuk commented on a diff in pull request #31882: Update documentation for constraints installation

2023-06-13 Thread via GitHub
potiuk commented on code in PR #31882: URL: https://github.com/apache/airflow/pull/31882#discussion_r1228819317 ## docs/apache-airflow/installation/installing-from-pypi.rst: ## @@ -53,20 +53,28 @@ This is an example, see further for more explanation. Constraints files

[GitHub] [airflow] potiuk commented on a diff in pull request #31882: Update documentation for constraints installation

2023-06-13 Thread via GitHub
potiuk commented on code in PR #31882: URL: https://github.com/apache/airflow/pull/31882#discussion_r1228819098 ## docs/apache-airflow/installation/installing-from-pypi.rst: ## @@ -53,20 +53,28 @@ This is an example, see further for more explanation. Constraints files

[GitHub] [airflow] o-nikolas commented on a diff in pull request #31882: Update documentation for constraints installation

2023-06-13 Thread via GitHub
o-nikolas commented on code in PR #31882: URL: https://github.com/apache/airflow/pull/31882#discussion_r1228802464 ## docs/apache-airflow/installation/installing-from-pypi.rst: ## @@ -53,20 +53,28 @@ This is an example, see further for more explanation. Constraints files '

[GitHub] [airflow] asosnovsky closed pull request #30914: Added support for custom KEDA scaling

2023-06-13 Thread via GitHub
asosnovsky closed pull request #30914: Added support for custom KEDA scaling URL: https://github.com/apache/airflow/pull/30914 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

[GitHub] [airflow] syedahsn commented on a diff in pull request #31657: Deferrable mode for EksCreateFargateProfileOperator and EksDeleteFargateProfileOperator

2023-06-13 Thread via GitHub
syedahsn commented on code in PR #31657: URL: https://github.com/apache/airflow/pull/31657#discussion_r1228786895 ## airflow/providers/amazon/aws/operators/eks.py: ## @@ -401,13 +417,31 @@ def execute(self, context: Context): selectors=self.selectors, *

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #31657: Deferrable mode for EksCreateFargateProfileOperator and EksDeleteFargateProfileOperator

2023-06-13 Thread via GitHub
vandonr-amz commented on code in PR #31657: URL: https://github.com/apache/airflow/pull/31657#discussion_r1228778172 ## airflow/providers/amazon/aws/operators/eks.py: ## @@ -401,13 +417,31 @@ def execute(self, context: Context): selectors=self.selectors,

[GitHub] [airflow] potiuk commented on issue #31823: Sensor with Deferrable not set (default is False) is Deferring

2023-06-13 Thread via GitHub
potiuk commented on issue #31823: URL: https://github.com/apache/airflow/issues/31823#issuecomment-1590140748 PR here: https://github.com/apache/airflow/pull/31882/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

[GitHub] [airflow] syedahsn commented on a diff in pull request #30463: Add custom waiters to EMR Serverless

2023-06-13 Thread via GitHub
syedahsn commented on code in PR #30463: URL: https://github.com/apache/airflow/pull/30463#discussion_r1228765387 ## airflow/providers/amazon/aws/utils/waiter_with_logging.py: ## @@ -0,0 +1,77 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributo

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #31881: Deferrable mode for ECS operators

2023-06-13 Thread via GitHub
vandonr-amz commented on code in PR #31881: URL: https://github.com/apache/airflow/pull/31881#discussion_r1228638295 ## airflow/providers/amazon/aws/operators/ecs.py: ## @@ -490,7 +549,26 @@ def execute(self, context, session=None): if self.reattach: self._

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #30463: Add custom waiters to EMR Serverless

2023-06-13 Thread via GitHub
vandonr-amz commented on code in PR #30463: URL: https://github.com/apache/airflow/pull/30463#discussion_r1228768476 ## airflow/providers/amazon/aws/utils/waiter_with_logging.py: ## @@ -0,0 +1,77 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contrib

[GitHub] [airflow] ferrouswheel commented on pull request #16569: Unpause DAG on manual trigger

2023-06-13 Thread via GitHub
ferrouswheel commented on PR #16569: URL: https://github.com/apache/airflow/pull/16569#issuecomment-1590094266 Just a note that this has caused some problems in production environments, after upgrading airflow and data engineers not being familiar with the changed behaviour. Sometime

[GitHub] [airflow] potiuk opened a new pull request, #31882: Update documentation for constraints installation

2023-06-13 Thread via GitHub
potiuk opened a new pull request, #31882: URL: https://github.com/apache/airflow/pull/31882 Airlow constraints and custom depenencies are a mystery for our users. The reason why we are using constraints is not clear and user are confused when they should and when they shoudl not use constra

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #31881: Deferrable mode for ECS operators

2023-06-13 Thread via GitHub
vandonr-amz commented on code in PR #31881: URL: https://github.com/apache/airflow/pull/31881#discussion_r1228638295 ## airflow/providers/amazon/aws/operators/ecs.py: ## @@ -490,7 +549,26 @@ def execute(self, context, session=None): if self.reattach: self._

[GitHub] [airflow] syedahsn commented on a diff in pull request #31657: Deferrable mode for EksCreateFargateProfileOperator and EksDeleteFargateProfileOperator

2023-06-13 Thread via GitHub
syedahsn commented on code in PR #31657: URL: https://github.com/apache/airflow/pull/31657#discussion_r1228731137 ## tests/providers/amazon/aws/triggers/test_eks.py: ## @@ -0,0 +1,312 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

[GitHub] [airflow] potiuk commented on pull request #31835: Add release management command that generates SBOM information

2023-06-13 Thread via GitHub
potiuk commented on PR #31835: URL: https://github.com/apache/airflow/pull/31835#issuecomment-1590066056 🙏 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

[GitHub] [airflow] vincbeck commented on issue #31759: User AWS postgresql with IAM auth as backend

2023-06-13 Thread via GitHub
vincbeck commented on issue #31759: URL: https://github.com/apache/airflow/issues/31759#issuecomment-1590045847 Did you follow the recommandations as mentioned in the [documentation](https://airflow.apache.org/docs/apache-airflow/stable/howto/set-up-database.html#setting-up-a-postgresql-data

[GitHub] [airflow] syedahsn commented on a diff in pull request #31657: Deferrable mode for EksCreateFargateProfileOperator and EksDeleteFargateProfileOperator

2023-06-13 Thread via GitHub
syedahsn commented on code in PR #31657: URL: https://github.com/apache/airflow/pull/31657#discussion_r1228706671 ## airflow/providers/amazon/aws/triggers/eks.py: ## @@ -0,0 +1,178 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license ag

[GitHub] [airflow] vincbeck commented on a diff in pull request #30463: Add custom waiters to EMR Serverless

2023-06-13 Thread via GitHub
vincbeck commented on code in PR #30463: URL: https://github.com/apache/airflow/pull/30463#discussion_r1228700420 ## airflow/providers/amazon/aws/utils/waiter_with_logging.py: ## @@ -0,0 +1,77 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributo

[GitHub] [airflow] syedahsn commented on a diff in pull request #31657: Deferrable mode for EksCreateFargateProfileOperator and EksDeleteFargateProfileOperator

2023-06-13 Thread via GitHub
syedahsn commented on code in PR #31657: URL: https://github.com/apache/airflow/pull/31657#discussion_r1228699857 ## airflow/providers/amazon/aws/operators/eks.py: ## @@ -401,13 +417,31 @@ def execute(self, context: Context): selectors=self.selectors, *

[GitHub] [airflow] syedahsn commented on a diff in pull request #31657: Deferrable mode for EksCreateFargateProfileOperator and EksDeleteFargateProfileOperator

2023-06-13 Thread via GitHub
syedahsn commented on code in PR #31657: URL: https://github.com/apache/airflow/pull/31657#discussion_r1228695276 ## airflow/providers/amazon/aws/operators/eks.py: ## @@ -371,11 +381,14 @@ def __init__( cluster_name: str, pod_execution_role_arn: str, s

[GitHub] [airflow] syedahsn commented on a diff in pull request #31657: Deferrable mode for EksCreateFargateProfileOperator and EksDeleteFargateProfileOperator

2023-06-13 Thread via GitHub
syedahsn commented on code in PR #31657: URL: https://github.com/apache/airflow/pull/31657#discussion_r1228691009 ## airflow/providers/amazon/aws/operators/eks.py: ## @@ -371,11 +381,14 @@ def __init__( cluster_name: str, pod_execution_role_arn: str, s

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #31881: Deferrable mode for ECS operators

2023-06-13 Thread via GitHub
vandonr-amz commented on code in PR #31881: URL: https://github.com/apache/airflow/pull/31881#discussion_r1228623744 ## airflow/providers/amazon/aws/operators/ecs.py: ## @@ -490,7 +549,26 @@ def execute(self, context, session=None): if self.reattach: self._

[GitHub] [airflow] shahar1 commented on a diff in pull request #31457: Fix BIGQUERY_JOB_DETAILS_LINK_FMT in BigQueryConsoleLink

2023-06-13 Thread via GitHub
shahar1 commented on code in PR #31457: URL: https://github.com/apache/airflow/pull/31457#discussion_r1228601415 ## airflow/providers/google/cloud/operators/bigquery.py: ## @@ -1181,7 +1191,13 @@ def execute(self, context: Context): ] else: rai

[GitHub] [airflow] eladkal commented on a diff in pull request #31840: added difference between Deferrable and Non-Deferrable Operators

2023-06-13 Thread via GitHub
eladkal commented on code in PR #31840: URL: https://github.com/apache/airflow/pull/31840#discussion_r1228593439 ## docs/apache-airflow/authoring-and-scheduling/deferring.rst: ## @@ -166,3 +166,13 @@ Airflow tries to only run triggers in one place at once, and maintains a heart

[GitHub] [airflow] potiuk commented on pull request #30727: Kubernetes Executor Load Time Optimizations

2023-06-13 Thread via GitHub
potiuk commented on PR #30727: URL: https://github.com/apache/airflow/pull/30727#issuecomment-1589879414 > Yeah, as I said above, if I'm resolving conflicts anyway I'm more than happy to resolve any conflicts that RM encounter. In fact I'm more than happy to start doing releases altogether

[GitHub] [airflow] o-nikolas commented on pull request #30727: Kubernetes Executor Load Time Optimizations

2023-06-13 Thread via GitHub
o-nikolas commented on PR #30727: URL: https://github.com/apache/airflow/pull/30727#issuecomment-1589878704 > LGTM. I think we are close enough to merge it. Can you please rebase and check if it still succeeds. Did the rebase yesterday and resolved the conflicts. Looks like the build

[GitHub] [airflow] rohan472000 commented on pull request #31880: Update breeze.py

2023-06-13 Thread via GitHub
rohan472000 commented on PR #31880: URL: https://github.com/apache/airflow/pull/31880#issuecomment-1589878617 > It's quite not what the issue was about. > > You do not need to add new "command" - you need to add new "--flags" to existing breeze `--shell` and `--start-airflow` commands

[GitHub] [airflow] o-nikolas commented on pull request #30727: Kubernetes Executor Load Time Optimizations

2023-06-13 Thread via GitHub
o-nikolas commented on PR #30727: URL: https://github.com/apache/airflow/pull/30727#issuecomment-1589877599 Hey @pierrejeambrun, thanks for weighing in! Most of the items have been discussed above, but here are some quick replies: > We have no policy for optimisation PR but we had iss

[GitHub] [airflow] potiuk merged pull request #31381: openlineage: add tests that confirm that extractors from previous OL versions work

2023-06-13 Thread via GitHub
potiuk merged PR #31381: URL: https://github.com/apache/airflow/pull/31381 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[GitHub] [airflow] potiuk closed issue #29671: Adapt OpenLineage default extractor to properly accept all OL implementation

2023-06-13 Thread via GitHub
potiuk closed issue #29671: Adapt OpenLineage default extractor to properly accept all OL implementation URL: https://github.com/apache/airflow/issues/29671 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[GitHub] [airflow] rohan472000 commented on a diff in pull request #31840: added difference between Deferrable and Non-Deferrable Operators

2023-06-13 Thread via GitHub
rohan472000 commented on code in PR #31840: URL: https://github.com/apache/airflow/pull/31840#discussion_r1228584314 ## docs/apache-airflow/authoring-and-scheduling/deferring.rst: ## @@ -166,3 +166,13 @@ Airflow tries to only run triggers in one place at once, and maintains a h

[GitHub] [airflow] potiuk commented on pull request #31880: Update breeze.py

2023-06-13 Thread via GitHub
potiuk commented on PR #31880: URL: https://github.com/apache/airflow/pull/31880#issuecomment-1589874640 It's quite not what the issue was about. You do not need to add new "command" - you need to add new "--flags" to existing breeze `--shell` and `--start-airflow` commands.

[GitHub] [airflow] eladkal commented on a diff in pull request #31840: added difference between Deferrable and Non-Deferrable Operators

2023-06-13 Thread via GitHub
eladkal commented on code in PR #31840: URL: https://github.com/apache/airflow/pull/31840#discussion_r1228581511 ## docs/apache-airflow/authoring-and-scheduling/deferring.rst: ## @@ -166,3 +166,13 @@ Airflow tries to only run triggers in one place at once, and maintains a heart

[GitHub] [airflow] eladkal commented on a diff in pull request #31840: added difference between Deferrable and Non-Deferrable Operators

2023-06-13 Thread via GitHub
eladkal commented on code in PR #31840: URL: https://github.com/apache/airflow/pull/31840#discussion_r1228581511 ## docs/apache-airflow/authoring-and-scheduling/deferring.rst: ## @@ -166,3 +166,13 @@ Airflow tries to only run triggers in one place at once, and maintains a heart

[GitHub] [airflow] rohan472000 opened a new pull request, #31880: Update breeze.py

2023-06-13 Thread via GitHub
rohan472000 opened a new pull request, #31880: URL: https://github.com/apache/airflow/pull/31880 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more

[GitHub] [airflow] potiuk commented on pull request #31879: Group options in shell and start-airflow commands

2023-06-13 Thread via GitHub
potiuk commented on PR #31879: URL: https://github.com/apache/airflow/pull/31879#issuecomment-1589855236 compare it with: ![image](https://github.com/apache/airflow/assets/595491/66a5452d-d21e-4c83-a279-61b5dedf4e0b) -- This is an automated message from the Apache Git Service.

[GitHub] [airflow] potiuk commented on pull request #31879: Group options in shell and start-airflow commands

2023-06-13 Thread via GitHub
potiuk commented on PR #31879: URL: https://github.com/apache/airflow/pull/31879#issuecomment-1589853973 cc: @utkarsharma2 @pankajkoti @vincbeck ![image](https://github.com/apache/airflow/assets/595491/1d3a8a52-112f-4d79-8d80-2eff291593c1) -- This is an automated message f

[GitHub] [airflow] potiuk opened a new pull request, #31879: Group options in shell and start-airflow commands

2023-06-13 Thread via GitHub
potiuk opened a new pull request, #31879: URL: https://github.com/apache/airflow/pull/31879 The shell and start-airflow commands started to have more options and the "advanced" flags became a bag of everything. This change groups the options in more logical groups at help screen. The option

[GitHub] [airflow] dwreeves commented on a diff in pull request #31846: add more accurate typing for DbApiHook.run method

2023-06-13 Thread via GitHub
dwreeves commented on code in PR #31846: URL: https://github.com/apache/airflow/pull/31846#discussion_r1228540748 ## airflow/providers/exasol/hooks/exasol.py: ## @@ -157,15 +161,39 @@ def get_description(statement: ExaStatement) -> Sequence[Sequence]: ) re

[GitHub] [airflow] potiuk merged pull request #31861: Add executor option to breeze shell command

2023-06-13 Thread via GitHub
potiuk merged PR #31861: URL: https://github.com/apache/airflow/pull/31861 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[airflow] branch main updated (f32371c96b -> 89bed231db)

2023-06-13 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from f32371c96b Clean up BigQuery async docs (#31727) add 89bed231db Add executor option to breeze shell command (#3186

[GitHub] [airflow] rohan472000 commented on issue #31826: Add a command line option to start a triggerer process with breeze

2023-06-13 Thread via GitHub
rohan472000 commented on issue #31826: URL: https://github.com/apache/airflow/issues/31826#issuecomment-1589824333 @potiuk @eladkal @vandonr-amz , To resolve this issue I need to make changes in `main()`, so should I do changes in `breeze.py` by making a `main()` there or edit the existing

[GitHub] [airflow] dwreeves commented on a diff in pull request #31846: add more accurate typing for DbApiHook.run method

2023-06-13 Thread via GitHub
dwreeves commented on code in PR #31846: URL: https://github.com/apache/airflow/pull/31846#discussion_r1228535137 ## airflow/providers/databricks/hooks/databricks_sql.py: ## @@ -138,15 +141,39 @@ def get_conn(self) -> Connection: ) return self._sql_conn +

[GitHub] [airflow] dwreeves commented on a diff in pull request #31846: add more accurate typing for DbApiHook.run method

2023-06-13 Thread via GitHub
dwreeves commented on code in PR #31846: URL: https://github.com/apache/airflow/pull/31846#discussion_r1228531239 ## airflow/providers/databricks/hooks/databricks_sql.py: ## @@ -138,15 +141,39 @@ def get_conn(self) -> Connection: ) return self._sql_conn +

  1   2   3   >