Re: [PR] docs: `Add AzureSynapseRunPipelineOperator` [airflow]

2023-12-30 Thread via GitHub
Lymah123 commented on PR #36495: URL: https://github.com/apache/airflow/pull/36495#issuecomment-1872792827 > LGTM Will merge when tests are green > LGTM Will merge when tests are green Yaay! Thanks for being patient with me. @eladkal ! Like how you give feedback! --

Re: [I] Task execution log window should indicate if contents are being loaded [airflow]

2023-12-30 Thread via GitHub
github-actions[bot] commented on issue #28355: URL: https://github.com/apache/airflow/issues/28355#issuecomment-1872735314 This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last

Re: [PR] docs: `Add AzureSynapseRunPipelineOperator` [airflow]

2023-12-30 Thread via GitHub
eladkal commented on code in PR #36495: URL: https://github.com/apache/airflow/pull/36495#discussion_r1438831923 ## docs/apache-airflow-providers-microsoft-azure/operators/azure_synapse.rst: ## @@ -41,6 +41,17 @@ Below is an example of using this operator to execute a Spark

Re: [PR] docs: `Add AzureSynapseRunPipelineOperator` [airflow]

2023-12-30 Thread via GitHub
Lymah123 commented on PR #36495: URL: https://github.com/apache/airflow/pull/36495#issuecomment-1872669178 @eladkal , you can review it again. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [POC][WIP] Async SQLAlchemy sessions in Airflow [airflow]

2023-12-30 Thread via GitHub
hussein-awala commented on PR #36504: URL: https://github.com/apache/airflow/pull/36504#issuecomment-1872653755 For the drivers: - SQLite: - https://github.com/omnilib/aiosqlite - PostgreSQL: - https://magicstack.github.io/asyncpg/current/ -

Re: [PR] Support encryption for triggers paramaters [airflow]

2023-12-30 Thread via GitHub
hussein-awala commented on PR #36492: URL: https://github.com/apache/airflow/pull/36492#issuecomment-1872652191 > > Ok. As @hussein-awala said, it's worth trying accessing DB using async. This is a healthy discussion. Thank you @potiuk. > > Yep. I am waiting to see it  Therea are

[PR] [POC][WIP] Async SQLAlchemy sessions in Airflow [airflow]

2023-12-30 Thread via GitHub
hussein-awala opened a new pull request, #36504: URL: https://github.com/apache/airflow/pull/36504 Airflow Metadata is the brain of Airflow; it is the central component used to store the state of all operations and enable communication between other components. A lot of the

Re: [PR] Allow timetable to slightly miss catchup cutoff [airflow]

2023-12-30 Thread via GitHub
fatmumuhomer commented on PR #33404: URL: https://github.com/apache/airflow/pull/33404#issuecomment-1872645280 @potiuk, @o-nikolas The original behavior of the CronTriggerTimetable was as described in the docs and as described as the expected behavior in

Re: [PR] Add `DatabricksJobsCreateOperator` [airflow]

2023-12-30 Thread via GitHub
github-actions[bot] closed pull request #32221: Add `DatabricksJobsCreateOperator` URL: https://github.com/apache/airflow/pull/32221 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Adding "job_id" to templated fields in the DatabricksRunNowOperator [airflow]

2023-12-30 Thread via GitHub
github-actions[bot] commented on PR #35576: URL: https://github.com/apache/airflow/pull/35576#issuecomment-1872632292 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for

Re: [PR] Change AirflowTaskTimeout to inherit BaseException [airflow]

2023-12-30 Thread via GitHub
github-actions[bot] commented on PR #35653: URL: https://github.com/apache/airflow/pull/35653#issuecomment-1872632280 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for

Re: [PR] Fix multiple issues in release notes does not have links [airflow]

2023-12-30 Thread via GitHub
potiuk commented on PR #36503: URL: https://github.com/apache/airflow/pull/36503#issuecomment-1872622054 NICE -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

(airflow) branch main updated: Fix multiple issues in release notes does not have links (#36503)

2023-12-30 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new c87ef553bc Fix multiple issues in release notes

Re: [I] Release notes issue reference does not provide link if multiple issues are present [airflow]

2023-12-30 Thread via GitHub
potiuk closed issue #36381: Release notes issue reference does not provide link if multiple issues are present URL: https://github.com/apache/airflow/issues/36381 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Fix multiple issues in release notes does not have links [airflow]

2023-12-30 Thread via GitHub
potiuk merged PR #36503: URL: https://github.com/apache/airflow/pull/36503 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] Status of testing Providers that were prepared on December 28, 2023 [airflow]

2023-12-30 Thread via GitHub
hussein-awala commented on issue #36467: URL: https://github.com/apache/airflow/issues/36467#issuecomment-1872619848 I checked my changes, and all looks good! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[PR] Fix multiple issues in release notes does not have links [airflow]

2023-12-30 Thread via GitHub
shohamy7 opened a new pull request, #36503: URL: https://github.com/apache/airflow/pull/36503 --- closes: #36381 Fix a bug when there are multiple issues in one record in release notes. Before this PR, the links will not be rendered, for example:

Re: [PR] Refactor _manage_executor_state by refreshing TIs in batch [airflow]

2023-12-30 Thread via GitHub
potiuk merged PR #36502: URL: https://github.com/apache/airflow/pull/36502 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(airflow) branch main updated: Refactor _manage_executor_state by refreshing TIs in batch (#36502)

2023-12-30 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 9cf5f6f084 Refactor _manage_executor_state by

Re: [PR] Refactor _manage_executor_state by refreshing TIs in batch [airflow]

2023-12-30 Thread via GitHub
potiuk commented on PR #36502: URL: https://github.com/apache/airflow/pull/36502#issuecomment-1872611495 > I cannot test locally because of the following: But it looks like it works now :) -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Refactor _manage_executor_state by refreshing TIs in batch [airflow]

2023-12-30 Thread via GitHub
potiuk commented on code in PR #36502: URL: https://github.com/apache/airflow/pull/36502#discussion_r1438714482 ## airflow/jobs/backfill_job_runner.py: ## @@ -264,16 +264,46 @@ def _manage_executor_state( :return: An iterable of expanded TaskInstance per MappedTask

Re: [PR] Refactor _manage_executor_state by refreshing TIs in batch [airflow]

2023-12-30 Thread via GitHub
hussein-awala commented on PR #36502: URL: https://github.com/apache/airflow/pull/36502#issuecomment-1872605298 I cannot test locally because of the following: > MSSQL is not supported on ARM architecture -- This is an automated message from the Apache Git Service. To respond to the

(airflow) branch v2-8-test updated: Fixes small issues related to suspended/removed providers (#36501)

2023-12-30 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v2-8-test in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/v2-8-test by this push: new 4bad5e4f95 Fixes small issues related to

(airflow) branch main updated: Fixes small issues related to suspended/removed providers (#36501)

2023-12-30 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new f0e05e8ff6 Fixes small issues related to

Re: [PR] Fixes small issues related to suspended/removed providers [airflow]

2023-12-30 Thread via GitHub
potiuk merged PR #36501: URL: https://github.com/apache/airflow/pull/36501 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(airflow) branch v2-8-test updated (e7ecb2e387 -> 23d06bed8e)

2023-12-30 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch v2-8-test in repository https://gitbox.apache.org/repos/asf/airflow.git discard e7ecb2e387 Speed up autocompletion of Breeze by simplifying provider state (#36499) new 23d06bed8e Speed up

(airflow) branch v2-8-test updated (bf85dbe9b3 -> e7ecb2e387)

2023-12-30 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch v2-8-test in repository https://gitbox.apache.org/repos/asf/airflow.git discard bf85dbe9b3 Speed up autocompletion of Breeze by simplifying provider state (#36499) new e7ecb2e387 Speed up

[PR] Fixes small issues related to suspended/removed providers [airflow]

2023-12-30 Thread via GitHub
potiuk opened a new pull request, #36501: URL: https://github.com/apache/airflow/pull/36501 After speeding up breeze in #36499 there are few small fixes needed for suspended/removed providers. --- **^ Add meaningful description above** Read the

(airflow) branch v2-8-test updated: Speed up autocompletion of Breeze by simplifying provider state (#36499)

2023-12-30 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v2-8-test in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/v2-8-test by this push: new bf85dbe9b3 Speed up autocompletion of

(airflow) branch main updated: Speed up autocompletion of Breeze by simplifying provider state (#36499)

2023-12-30 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 6937ae7647 Speed up autocompletion of Breeze by

Re: [PR] Speed up autocompletion of Breeze by simplifying provider state [airflow]

2023-12-30 Thread via GitHub
potiuk merged PR #36499: URL: https://github.com/apache/airflow/pull/36499 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Speed up autocompletion of Breeze by simplifying provider state [airflow]

2023-12-30 Thread via GitHub
potiuk commented on PR #36499: URL: https://github.com/apache/airflow/pull/36499#issuecomment-1872577644 Merging. the Mssql failures already fixed in main -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

(airflow) branch main updated: Revert "Refactor _manage_executor_state by refreshing TIs in batch (#36418)" (#36500)

2023-12-30 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 72f43fcc83 Revert "Refactor _manage_executor_state

Re: [PR] Revert "Refactor _manage_executor_state by refreshing TIs in batch (#… [airflow]

2023-12-30 Thread via GitHub
potiuk merged PR #36500: URL: https://github.com/apache/airflow/pull/36500 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Revert "Refactor _manage_executor_state by refreshing TIs in batch (#… [airflow]

2023-12-30 Thread via GitHub
potiuk closed pull request #36500: Revert "Refactor _manage_executor_state by refreshing TIs in batch (#… URL: https://github.com/apache/airflow/pull/36500 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Revert "Refactor _manage_executor_state by refreshing TIs in batch (#… [airflow]

2023-12-30 Thread via GitHub
potiuk commented on PR #36500: URL: https://github.com/apache/airflow/pull/36500#issuecomment-1872575696 I guess it's better to revert that one temporarily until mssql failures are fixed ? -- This is an automated message from the Apache Git Service. To respond to the message, please log

[PR] Revert "Refactor _manage_executor_state by refreshing TIs in batch (#… [airflow]

2023-12-30 Thread via GitHub
potiuk opened a new pull request, #36500: URL: https://github.com/apache/airflow/pull/36500 …36418)" This reverts commit 9d45db9e2cca2ad04db72f7e0712c478e5a8e1f1. t# --- **^ Add meaningful description above** Read the **[Pull Request

Re: [I] GCS Sensor for matching glob [airflow]

2023-12-30 Thread via GitHub
eladkal closed issue #32650: GCS Sensor for matching glob URL: https://github.com/apache/airflow/issues/32650 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [I] GCS Sensor for matching glob [airflow]

2023-12-30 Thread via GitHub
eladkal commented on issue #32650: URL: https://github.com/apache/airflow/issues/32650#issuecomment-1872574644 solved in https://github.com/apache/airflow/pull/34137 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] docs: `Add AzureSynapseRunPipelineOperator` [airflow]

2023-12-30 Thread via GitHub
eladkal commented on code in PR #36495: URL: https://github.com/apache/airflow/pull/36495#discussion_r1438670195 ## docs/apache-airflow-providers-microsoft-azure/operators/azure_synapse.rst: ## @@ -41,6 +41,17 @@ Below is an example of using this operator to execute a Spark

Re: [PR] Add pre-commit check to check integrations table. [airflow]

2023-12-30 Thread via GitHub
potiuk commented on PR #36497: URL: https://github.com/apache/airflow/pull/36497#issuecomment-1872574142 > Particularly, I am not sure about auto generating the table, as the description part is missing. Could the description be pulled from the yml files? Generally looks cool. We do

Re: [PR] docs: `Add AzureSynapseRunPipelineOperator` [airflow]

2023-12-30 Thread via GitHub
eladkal commented on code in PR #36495: URL: https://github.com/apache/airflow/pull/36495#discussion_r1438669477 ## tests/system/providers/microsoft/azure/example_synapse_run_pipeline.py: ## @@ -36,15 +36,15 @@ catchup=False, tags=["synapse", "example"], ) as dag: -

Re: [I] Airflow 2.7.3: Mapped task group no longer works as expected [airflow]

2023-12-30 Thread via GitHub
potiuk commented on issue #35541: URL: https://github.com/apache/airflow/issues/35541#issuecomment-1872572199 Not sure :). Have not looked at it . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Speed up autocompletion of Breeze by simplifying provider state [airflow]

2023-12-30 Thread via GitHub
potiuk commented on PR #36499: URL: https://github.com/apache/airflow/pull/36499#issuecomment-1872571541 BTW. py-spy is great :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Speed up autocompletion of Breeze by simplifying provider state [airflow]

2023-12-30 Thread via GitHub
potiuk commented on PR #36499: URL: https://github.com/apache/airflow/pull/36499#issuecomment-1872570351 This had been discussed in https://github.com/apache/airflow/pull/36391 and it turned out that complex state management had real impact on auto-complete: The auto-completion time

Re: [PR] Speed up autocompletion of Breeze by simplifying provider state [airflow]

2023-12-30 Thread via GitHub
potiuk closed pull request #36498: Speed up autocompletion of Breeze by simplifying provider state URL: https://github.com/apache/airflow/pull/36498 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Speed up autocompletion of Breeze by simplifying provider state [airflow]

2023-12-30 Thread via GitHub
potiuk commented on PR #36498: URL: https://github.com/apache/airflow/pull/36498#issuecomment-1872570320 Closing in favour of #36499 - because this change should be done in "apache" repo -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[PR] Speed up autocompletion of Breeze by simplifying provider state [airflow]

2023-12-30 Thread via GitHub
potiuk opened a new pull request, #36499: URL: https://github.com/apache/airflow/pull/36499 Some recent changes, adding removed and suspended state for breeze caused significant slow-down of autocompletion retrieval - as it turned out, because we loaded and parsed all provider yaml files

Re: [PR] docs: `Add AzureSynapseRunPipelineOperator` [airflow]

2023-12-30 Thread via GitHub
Lymah123 commented on PR #36495: URL: https://github.com/apache/airflow/pull/36495#issuecomment-1872570175 Hi @eladkal , I have seen the duplicate you're talking about. You can check again. Thanks for being patient with me. -- This is an automated message from the Apache Git

(airflow) 01/01: Speed up autocompletion of Breeze by simplifying provider state

2023-12-30 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch simplify-provider-state in repository https://gitbox.apache.org/repos/asf/airflow.git commit 7626ab12061ce33efd64f38abdd41c865de3c27c Author: Jarek Potiuk AuthorDate: Sat Dec 30 17:54:44 2023

(airflow) branch simplify-provider-state created (now 7626ab1206)

2023-12-30 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch simplify-provider-state in repository https://gitbox.apache.org/repos/asf/airflow.git at 7626ab1206 Speed up autocompletion of Breeze by simplifying provider state This branch includes

Re: [PR] Speed up autocompletion of Breeze by simplifying provider state [airflow]

2023-12-30 Thread via GitHub
potiuk commented on PR #36498: URL: https://github.com/apache/airflow/pull/36498#issuecomment-1872569691 This had been discussed in https://github.com/apache/airflow/pull/36391 and it turned out that complex state management had real impact on auto-complete: The auto-completion time

[PR] Speed up autocompletion of Breeze by simplifying provider state [airflow]

2023-12-30 Thread via GitHub
potiuk opened a new pull request, #36498: URL: https://github.com/apache/airflow/pull/36498 Some recent changes, adding removed and suspended state for breeze caused significant slow-down of autocompletion retrieval - as it turned out, because we loaded and parsed all provider yaml files

Re: [PR] Add pre-commit check to check integrations table. [airflow]

2023-12-30 Thread via GitHub
timonviola commented on PR #36497: URL: https://github.com/apache/airflow/pull/36497#issuecomment-1872553584 @potiuk Could you give some first feedback on this? Particularly, I am not sure about **auto generating** the table, as the description part is missing. Could the description

Re: [PR] Add pre-commit check to check integrations table. [airflow]

2023-12-30 Thread via GitHub
boring-cyborg[bot] commented on PR #36497: URL: https://github.com/apache/airflow/pull/36497#issuecomment-1872552766 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution

[PR] Add pre-commit check to check integrations table. [airflow]

2023-12-30 Thread via GitHub
timonviola opened a new pull request, #36497: URL: https://github.com/apache/airflow/pull/36497 Add pre-commit check to check if integrations and TESTING documentation are synchronized. --- The new pre-commit test: - reads `integration-*.yml` files - reads the table of

Re: [I] Airflow 2.7.3: Mapped task group no longer works as expected [airflow]

2023-12-30 Thread via GitHub
eladkal commented on issue #35541: URL: https://github.com/apache/airflow/issues/35541#issuecomment-1872551875 @potiuk i think https://github.com/apache/airflow/pull/36462 handles it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Support encryption for triggers paramaters [airflow]

2023-12-30 Thread via GitHub
potiuk commented on PR #36492: URL: https://github.com/apache/airflow/pull/36492#issuecomment-1872549992 > Ok. As @hussein-awala said, it's worth trying accessing DB using async. This is a healthy discussion. Thank you @potiuk. Yep. I am waiting to see it :eyes: Therea are two

Re: [PR] fix backward compatibility `bigquery_conn_id` for `BigQueryExecuteQueryOperator` [airflow]

2023-12-30 Thread via GitHub
RNHTTR closed pull request #33930: fix backward compatibility `bigquery_conn_id` for `BigQueryExecuteQueryOperator` URL: https://github.com/apache/airflow/pull/33930 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Support encryption for triggers paramaters [airflow]

2023-12-30 Thread via GitHub
dirrao commented on PR #36492: URL: https://github.com/apache/airflow/pull/36492#issuecomment-1872547808 > > Yes. You are right. It makes even more sense in the case of request/response APIs. However, in airflow, the trigger might run for seconds/minutes/hours (ex: S3KeyTrigger,

Re: [PR] docs: `Add AzureSynapseRunPipelineOperator` [airflow]

2023-12-30 Thread via GitHub
eladkal commented on code in PR #36495: URL: https://github.com/apache/airflow/pull/36495#discussion_r1438625527 ## tests/system/providers/microsoft/azure/example_synapse_run_pipeline.py: ## @@ -36,6 +36,14 @@ catchup=False, tags=["synapse", "example"], ) as dag: +

Re: [PR] Support encryption for triggers paramaters [airflow]

2023-12-30 Thread via GitHub
potiuk commented on PR #36492: URL: https://github.com/apache/airflow/pull/36492#issuecomment-1872543877 > Yes. You are right. It makes even more sense in the case of request/response APIs. However, in airflow, the trigger might run for seconds/minutes/hours (ex: S3KeyTrigger,

(airflow) branch main updated: Add flask config: `MAX_CONTENT_LENGTH` (#36401)

2023-12-30 Thread husseinawala
This is an automated email from the ASF dual-hosted git repository. husseinawala pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 84063e74fb Add flask config:

Re: [PR] Add flask config: `MAX_CONTENT_LENGTH` [airflow]

2023-12-30 Thread via GitHub
hussein-awala merged PR #36401: URL: https://github.com/apache/airflow/pull/36401 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] docs: `Add AzureSynapseRunPipelineOperator` [airflow]

2023-12-30 Thread via GitHub
Lymah123 commented on code in PR #36495: URL: https://github.com/apache/airflow/pull/36495#discussion_r1438616450 ## tests/system/providers/microsoft/azure/example_synapse_run_pipeline.py: ## @@ -36,6 +36,14 @@ catchup=False, tags=["synapse", "example"], ) as dag: +

(airflow) branch constraints-2-8 updated: Updating constraints. Github run id:7364330476

2023-12-30 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-2-8 in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-2-8 by this push: new d839a98fcd Updating

Re: [PR] Support encryption for triggers paramaters [airflow]

2023-12-30 Thread via GitHub
dirrao commented on PR #36492: URL: https://github.com/apache/airflow/pull/36492#issuecomment-1872537980 > > I have tried accessing the variables from the trigger run function multiple times after multiple context switches in event loop. I can access the variable values without any issues

Re: [PR] docs: `Add AzureSynapseRunPipelineOperator` [airflow]

2023-12-30 Thread via GitHub
eladkal commented on code in PR #36495: URL: https://github.com/apache/airflow/pull/36495#discussion_r1438612830 ## tests/system/providers/microsoft/azure/example_synapse_run_pipeline.py: ## @@ -36,6 +36,14 @@ catchup=False, tags=["synapse", "example"], ) as dag: +

Re: [PR] Support encryption for triggers paramaters [airflow]

2023-12-30 Thread via GitHub
potiuk commented on PR #36492: URL: https://github.com/apache/airflow/pull/36492#issuecomment-1872534863 > I'm running a POC to make this easily possible without blocking the event loop and will create a new AIP soon 爛 :eyes: -- This is an automated message from the Apache Git

Re: [PR] docs: `Add AzureSynapseRunPipelineOperator` [airflow]

2023-12-30 Thread via GitHub
Lymah123 commented on code in PR #36495: URL: https://github.com/apache/airflow/pull/36495#discussion_r1438609313 ## tests/system/providers/microsoft/azure/example_synapse_run_pipeline.py: ## @@ -36,6 +36,14 @@ catchup=False, tags=["synapse", "example"], ) as dag: +

Re: [PR] Support encryption for triggers paramaters [airflow]

2023-12-30 Thread via GitHub
hussein-awala commented on PR #36492: URL: https://github.com/apache/airflow/pull/36492#issuecomment-1872534368 > The problem is that if you do it during the event loop, the access (if not using async/await) will block - in I/O or networking and when it is blocked, no other triggers are

Re: [PR] Support encryption for triggers paramaters [airflow]

2023-12-30 Thread via GitHub
potiuk commented on PR #36492: URL: https://github.com/apache/airflow/pull/36492#issuecomment-1872533458 > I have tried accessing the variables from the trigger run function multiple times after multiple context switches in event loop. I can access the variable values without any issues

Re: [PR] docs: `Add AzureSynapseRunPipelineOperator` [airflow]

2023-12-30 Thread via GitHub
Lymah123 commented on code in PR #36495: URL: https://github.com/apache/airflow/pull/36495#discussion_r1438606960 ## tests/system/providers/microsoft/azure/example_synapse_run_pipeline.py: ## @@ -36,6 +36,14 @@ catchup=False, tags=["synapse", "example"], ) as dag: +

Re: [PR] Support encryption for triggers paramaters [airflow]

2023-12-30 Thread via GitHub
dirrao commented on PR #36492: URL: https://github.com/apache/airflow/pull/36492#issuecomment-1872529072 > > > We usually access secrets from either disk or connections or encrypted variables in airflow > > > > > > Correct. This is possible, no doubt about it. > > But it might

(airflow) 01/01: Bump stalebot to version 9 (#36494)

2023-12-30 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v2-8-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit cff4394256da40028adbe9fa0300288b582e8245 Author: Elad Kalif <45845474+elad...@users.noreply.github.com> AuthorDate:

(airflow) branch v2-8-test updated (0f0fbe3d63 -> cff4394256)

2023-12-30 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch v2-8-test in repository https://gitbox.apache.org/repos/asf/airflow.git discard 0f0fbe3d63 Bump stalebot to version 9 (#36494) new cff4394256 Bump stalebot to version 9 (#36494) This

Re: [PR] Add flask config: `MAX_CONTENT_LENGTH` [airflow]

2023-12-30 Thread via GitHub
yo1956 commented on code in PR #36401: URL: https://github.com/apache/airflow/pull/36401#discussion_r1438595200 ## airflow/config_templates/config.yml: ## @@ -1855,6 +1855,13 @@ webserver: type: boolean example: "False" default: "False" +

Re: [PR] Refactor _manage_executor_state by refreshing TIs in batch [airflow]

2023-12-30 Thread via GitHub
potiuk commented on PR #36418: URL: https://github.com/apache/airflow/pull/36418#issuecomment-1872525803 It fails in main for MSSQL. We are planning to cherry-pick that one for 2.8.1 so I hope it's the last that requires MSSQL special treatment (I guess you need to open fixup PR and set

Re: [PR] Add flask config: `MAX_CONTENT_LENGTH` [airflow]

2023-12-30 Thread via GitHub
yo1956 commented on code in PR #36401: URL: https://github.com/apache/airflow/pull/36401#discussion_r1438594348 ## airflow/www/app.py: ## @@ -75,6 +75,8 @@ def create_app(config=None, testing=False): flask_app.config["PERMANENT_SESSION_LIFETIME"] =

Re: [PR] docs: `Add AzureSynapseRunPipelineOperator` [airflow]

2023-12-30 Thread via GitHub
eladkal commented on code in PR #36495: URL: https://github.com/apache/airflow/pull/36495#discussion_r1438594362 ## tests/system/providers/microsoft/azure/example_synapse_run_pipeline.py: ## @@ -36,6 +36,14 @@ catchup=False, tags=["synapse", "example"], ) as dag: +

Re: [PR] Add flask config: `MAX_CONTENT_LENGTH` [airflow]

2023-12-30 Thread via GitHub
yo1956 commented on code in PR #36401: URL: https://github.com/apache/airflow/pull/36401#discussion_r1438594348 ## airflow/www/app.py: ## @@ -75,6 +75,8 @@ def create_app(config=None, testing=False): flask_app.config["PERMANENT_SESSION_LIFETIME"] =

Re: [PR] Support encryption for triggers paramaters [airflow]

2023-12-30 Thread via GitHub
potiuk commented on code in PR #36492: URL: https://github.com/apache/airflow/pull/36492#discussion_r1438593904 ## airflow/models/trigger.py: ## @@ -90,8 +90,17 @@ def __init__( @internal_api_call def from_object(cls, trigger: BaseTrigger) -> Trigger:

Re: [PR] Support encryption for triggers paramaters [airflow]

2023-12-30 Thread via GitHub
potiuk commented on PR #36492: URL: https://github.com/apache/airflow/pull/36492#issuecomment-1872524904 And resolving the conversation enables merging. All cool :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Support encryption for triggers paramaters [airflow]

2023-12-30 Thread via GitHub
hussein-awala commented on PR #36492: URL: https://github.com/apache/airflow/pull/36492#issuecomment-1872524654 ``` Merging is blocked The base branch requires all conversations on code to be resolved. ```  -- This is an automated message from the Apache Git Service. To

Re: [PR] Fix the .asf.yml indentation [airflow]

2023-12-30 Thread via GitHub
potiuk commented on PR #36496: URL: https://github.com/apache/airflow/pull/36496#issuecomment-1872524532 Works like a charm now: ![image](https://github.com/apache/airflow/assets/595491/6ded0448-77de-4767-82af-981eb0ae8e6b) -- This is an automated message from the Apache Git

(airflow) branch main updated: Fix the .asf.yml indentation (#36496)

2023-12-30 Thread husseinawala
This is an automated email from the ASF dual-hosted git repository. husseinawala pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 465c2f6946 Fix the .asf.yml indentation

Re: [PR] Fix the .asf.yml indentation [airflow]

2023-12-30 Thread via GitHub
hussein-awala merged PR #36496: URL: https://github.com/apache/airflow/pull/36496 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Fix the .asf.yml indentation [airflow]

2023-12-30 Thread via GitHub
potiuk commented on PR #36496: URL: https://github.com/apache/airflow/pull/36496#issuecomment-1872523701 :facepalm: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[PR] Fix the .asf.yml indentation [airflow]

2023-12-30 Thread via GitHub
potiuk opened a new pull request, #36496: URL: https://github.com/apache/airflow/pull/36496 The .asf.yml with "required conversation resolution" had been wrongly indented. --- **^ Add meaningful description above** Read the **[Pull Request

Re: [PR] Support encryption for triggers paramaters [airflow]

2023-12-30 Thread via GitHub
hussein-awala commented on code in PR #36492: URL: https://github.com/apache/airflow/pull/36492#discussion_r1438591710 ## airflow/models/trigger.py: ## @@ -90,8 +90,17 @@ def __init__( @internal_api_call def from_object(cls, trigger: BaseTrigger) -> Trigger:

Re: [PR] Support encryption for triggers paramaters [airflow]

2023-12-30 Thread via GitHub
hussein-awala commented on code in PR #36492: URL: https://github.com/apache/airflow/pull/36492#discussion_r1438590907 ## airflow/models/trigger.py: ## @@ -90,8 +90,17 @@ def __init__( @internal_api_call def from_object(cls, trigger: BaseTrigger) -> Trigger:

Re: [PR] Support encryption for triggers paramaters [airflow]

2023-12-30 Thread via GitHub
potiuk commented on code in PR #36492: URL: https://github.com/apache/airflow/pull/36492#discussion_r1438589955 ## airflow/models/trigger.py: ## @@ -90,8 +90,17 @@ def __init__( @internal_api_call def from_object(cls, trigger: BaseTrigger) -> Trigger:

Re: [PR] Add flask config: `MAX_CONTENT_LENGTH` [airflow]

2023-12-30 Thread via GitHub
eladkal commented on code in PR #36401: URL: https://github.com/apache/airflow/pull/36401#discussion_r1438589629 ## airflow/config_templates/config.yml: ## @@ -1855,6 +1855,13 @@ webserver: type: boolean example: "False" default: "False" +

Re: [PR] docs: `Add AzureSynapseRunPipelineOperator` [airflow]

2023-12-30 Thread via GitHub
Lymah123 commented on PR #36495: URL: https://github.com/apache/airflow/pull/36495#issuecomment-1872522385 @eladkal , I guessed I have worked on your suggestions. Let me know if there is any other thing. Thanks! -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] Add flask config: `MAX_CONTENT_LENGTH` [airflow]

2023-12-30 Thread via GitHub
hussein-awala commented on code in PR #36401: URL: https://github.com/apache/airflow/pull/36401#discussion_r1438582934 ## airflow/www/app.py: ## @@ -75,6 +75,8 @@ def create_app(config=None, testing=False): flask_app.config["PERMANENT_SESSION_LIFETIME"] =

(airflow) 01/01: Bump stalebot to version 9 (#36494)

2023-12-30 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v2-8-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 0f0fbe3d63ec62b066119ccde62eaacdf4bf8569 Author: Elad Kalif <45845474+elad...@users.noreply.github.com> AuthorDate:

(airflow) branch v2-8-test updated (94186a04f1 -> 0f0fbe3d63)

2023-12-30 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch v2-8-test in repository https://gitbox.apache.org/repos/asf/airflow.git discard 94186a04f1 Bump stalebot to version 9 (#36494) new 0f0fbe3d63 Bump stalebot to version 9 (#36494) This

Re: [PR] Add flask config: `MAX_CONTENT_LENGTH` [airflow]

2023-12-30 Thread via GitHub
potiuk commented on code in PR #36401: URL: https://github.com/apache/airflow/pull/36401#discussion_r1438580496 ## airflow/config_templates/config.yml: ## @@ -1855,6 +1855,13 @@ webserver: type: boolean example: "False" default: "False" +

(airflow) 20/27: Update providers metadata 2023-12-28 (#36459)

2023-12-30 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v2-8-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 5576e1151dcabe66b18b350d296574631d4f493e Author: Jarek Potiuk AuthorDate: Thu Dec 28 02:41:04 2023 +0100

Re: [PR] Add flask config: `MAX_CONTENT_LENGTH` [airflow]

2023-12-30 Thread via GitHub
eladkal commented on code in PR #36401: URL: https://github.com/apache/airflow/pull/36401#discussion_r1438580095 ## airflow/config_templates/config.yml: ## @@ -1855,6 +1855,13 @@ webserver: type: boolean example: "False" default: "False" +

  1   2   >