Re: [PR] Support encryption for triggers paramaters [airflow]

2023-12-29 Thread via GitHub
dirrao commented on PR #36492: URL: https://github.com/apache/airflow/pull/36492#issuecomment-1872477618 > > We usually access secrets from either disk or connections or encrypted variables in airflow > > Correct. This is possible, no doubt about it. > > But it might quite a

(airflow) branch main updated: Review and update our Pull Request review process and policy (#36472)

2023-12-29 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 3d6ecdfd9b Review and update our Pull Request

Re: [PR] Review and update our Pull Request review process and policy [airflow]

2023-12-29 Thread via GitHub
potiuk merged PR #36472: URL: https://github.com/apache/airflow/pull/36472 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Review and update our Pull Request review process and policy [airflow]

2023-12-29 Thread via GitHub
potiuk commented on PR #36472: URL: https://github.com/apache/airflow/pull/36472#issuecomment-1872477386 Merging it and sending email /slack message. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Support encryption for triggers paramaters [airflow]

2023-12-29 Thread via GitHub
potiuk commented on PR #36492: URL: https://github.com/apache/airflow/pull/36492#issuecomment-1872474138 > We usually access secrets from either disk or connections or encrypted variables in airflow Correct. This is possible, no doubt about it. But it might quite a bit

Re: [PR] Support encryption for triggers paramaters [airflow]

2023-12-29 Thread via GitHub
dirrao commented on PR #36492: URL: https://github.com/apache/airflow/pull/36492#issuecomment-1872469573 > > Nice work. Can't we access the variables and connections from the trigger run? What is the recommended pattern for accessing the sensitive information from the trigger run? >

Re: [PR] Make it possible to change the default cron timetable [airflow]

2023-12-29 Thread via GitHub
potiuk commented on PR #34851: URL: https://github.com/apache/airflow/pull/34851#issuecomment-1872463906 https://github.com/apache/airflow/assets/595491/8c2bba54-84ca-4680-a33f-c0e19a816347;> -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Make it possible to change the default cron timetable [airflow]

2023-12-29 Thread via GitHub
potiuk commented on PR #34851: URL: https://github.com/apache/airflow/pull/34851#issuecomment-1872463815 > I'm pretty sure the failing test is spurious. I do not have permission to re-run it. Neither do we once there is conflict with another PR. You need to rebase and solve the

Re: [PR] Support encryption for triggers paramaters [airflow]

2023-12-29 Thread via GitHub
potiuk commented on PR #36492: URL: https://github.com/apache/airflow/pull/36492#issuecomment-1872463217 > Nice work. Can't we access the variables and connections from the trigger run? What is the recommended pattern for accessing the sensitive information from the trigger run? We

Re: [PR] Make it possible to change the default cron timetable [airflow]

2023-12-29 Thread via GitHub
collinmcnulty commented on PR #34851: URL: https://github.com/apache/airflow/pull/34851#issuecomment-1872447758 I'm pretty sure the failing test is spurious. I do not have permission to re-run it. -- This is an automated message from the Apache Git Service. To respond to the message,

(airflow) branch constraints-main updated: Updating constraints. Github run id:7361149251

2023-12-29 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new df2b9fc19a Updating

Re: [I] Missing AzureSynapseRunPipelineOperator in docs [airflow]

2023-12-29 Thread via GitHub
killua1zoldyck commented on issue #36488: URL: https://github.com/apache/airflow/issues/36488#issuecomment-1872423907 Hi, I can work on this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

(airflow) branch main updated: Add zlib1g-dev package to Airflow images (#36493)

2023-12-29 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 2bc34ffcb5 Add zlib1g-dev package to Airflow

Re: [PR] Add zlib1g-dev package to Airflow images [airflow]

2023-12-29 Thread via GitHub
potiuk merged PR #36493: URL: https://github.com/apache/airflow/pull/36493 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(airflow) branch add-zlib1g-dev-package-to-image updated (e6d9202079 -> 1be142850e)

2023-12-29 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch add-zlib1g-dev-package-to-image in repository https://gitbox.apache.org/repos/asf/airflow.git omit e6d9202079 Add zlib1g-dev package to Airflow images add 1be142850e Add zlib1g-dev

[PR] Add zlib1g-dev package to Airflow images [airflow]

2023-12-29 Thread via GitHub
potiuk opened a new pull request, #36493: URL: https://github.com/apache/airflow/pull/36493 Seems that when mysql repository is used to install mysql client, it induces libxml compilation for Python 3.8 and 3.9 and this library requires devel version of zlib that is missing in the image.

(airflow) branch add-zlib1g-dev-package-to-image created (now e6d9202079)

2023-12-29 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch add-zlib1g-dev-package-to-image in repository https://gitbox.apache.org/repos/asf/airflow.git at e6d9202079 Add zlib1g-dev package to Airflow images This branch includes the following new

(airflow) 01/01: Add zlib1g-dev package to Airflow images

2023-12-29 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch add-zlib1g-dev-package-to-image in repository https://gitbox.apache.org/repos/asf/airflow.git commit e6d92020793dcc2cc9430e219a5b936abdd8105b Author: Jarek Potiuk AuthorDate: Sat Dec 30 01:15:00

Re: [PR] Pass extra configs to `ClientSecretCredential` and `SparkClient` in `AzureSynapseHook` [airflow]

2023-12-29 Thread via GitHub
github-actions[bot] closed pull request #35024: Pass extra configs to `ClientSecretCredential` and `SparkClient` in `AzureSynapseHook` URL: https://github.com/apache/airflow/pull/35024 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] [WIP] Configure bandit to medium and fix security issues [airflow]

2023-12-29 Thread via GitHub
github-actions[bot] commented on PR #35615: URL: https://github.com/apache/airflow/pull/35615#issuecomment-1872400026 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for

Re: [PR] fix(timetable): CronMixin issue when DST [airflow]

2023-12-29 Thread via GitHub
github-actions[bot] commented on PR #35632: URL: https://github.com/apache/airflow/pull/35632#issuecomment-1872400014 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for

[PR] Support encryption for triggers paramaters [airflow]

2023-12-29 Thread via GitHub
hussein-awala opened a new pull request, #36492: URL: https://github.com/apache/airflow/pull/36492 This PR implements a way to encrypt some of the trigger parameters using fernet before storing them in the database. -- This is an automated message from the Apache Git Service. To respond

Re: [PR] Implement Google Analytics Admin (GA4) operators [airflow]

2023-12-29 Thread via GitHub
potiuk commented on PR #36276: URL: https://github.com/apache/airflow/pull/36276#issuecomment-1872393805 @VladaZakharova The solution will be (likely) to increase the timeout for spellcheck job. The problem is that in this case there are some cross-changes between packages, that makes it

(airflow) branch main updated: remove unecessary templated field (#36491)

2023-12-29 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new f070efa95c remove unecessary templated field

Re: [PR] prevent templated field logic checks in operators `__init__` in `BigQueryToPostgresOperator` operator [airflow]

2023-12-29 Thread via GitHub
potiuk merged PR #36491: URL: https://github.com/apache/airflow/pull/36491 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(airflow) branch main updated: Add branching based on mapped task group example to dynamic-task-mapping.rst (#36480)

2023-12-29 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 0d9a26ceef Add branching based on mapped task

Re: [PR] Add branching based on mapped task group example to dynamic-task-mapping.rst [airflow]

2023-12-29 Thread via GitHub
potiuk merged PR #36480: URL: https://github.com/apache/airflow/pull/36480 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Add branching based on mapped task group example to dynamic-task-mapping.rst [airflow]

2023-12-29 Thread via GitHub
potiuk commented on PR #36480: URL: https://github.com/apache/airflow/pull/36480#issuecomment-1872382584 Started discussion about it in https://lists.apache.org/thread/fgoqj23hpl9mlo1n6gfkgvzjxvhfchzm -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Add branching based on mapped task group example to dynamic-task-mapping.rst [airflow]

2023-12-29 Thread via GitHub
potiuk commented on PR #36480: URL: https://github.com/apache/airflow/pull/36480#issuecomment-1872373348 Actually ... that makes me think to propose to enable the https://pre-commit.ci/ application for us - which should handle such issues by automatically fixing such PRs. I

Re: [PR] Improve TaskInstance typing hints [airflow]

2023-12-29 Thread via GitHub
jscheffl commented on code in PR #36487: URL: https://github.com/apache/airflow/pull/36487#discussion_r1438431448 ## airflow/models/taskinstance.py: ## @@ -3398,10 +3405,12 @@ def clear_db_references(self, session): for table in tables: session.execute(

[PR] prevent templated field logic checks in operators `__init__` in `BigQueryToPostgresOperator` operator [airflow]

2023-12-29 Thread via GitHub
romsharon98 opened a new pull request, #36491: URL: https://github.com/apache/airflow/pull/36491 related: https://github.com/apache/airflow/issues/36484 fix `BigQueryToPostgresOperator` operator for this cherry-picking: https://github.com/apache/airflow/pull/33786

Re: [PR] Add branching based on mapped task group example to dynamic-task-mapping.rst [airflow]

2023-12-29 Thread via GitHub
potiuk commented on PR #36480: URL: https://github.com/apache/airflow/pull/36480#issuecomment-1872363062 > I have it installed; I just tried to sneak one past the tests directly from the browser   . I won't be lazy moving forward. Been there, done that :D -- This is an automated

Re: [PR] Add branching based on mapped task group example to dynamic-task-mapping.rst [airflow]

2023-12-29 Thread via GitHub
RNHTTR commented on PR #36480: URL: https://github.com/apache/airflow/pull/36480#issuecomment-1872362628 > Some static checks. Highly recommend to have `pre-commit` installed :) I have it installed; I just tried to sneak one past the tests directly from the browser   -- This is

(airflow) branch main updated: remove unnecessary templated filed (#36490)

2023-12-29 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new eac3d95584 remove unnecessary templated filed

Re: [PR] prevent templated field logic checks in operators `__init__` in livy operator [airflow]

2023-12-29 Thread via GitHub
potiuk merged PR #36490: URL: https://github.com/apache/airflow/pull/36490 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Add description on the ways how users should approach DB monitoring [airflow]

2023-12-29 Thread via GitHub
potiuk commented on code in PR #36483: URL: https://github.com/apache/airflow/pull/36483#discussion_r1438414061 ## docs/apache-airflow/core-concepts/tasks.rst: ## @@ -244,7 +244,7 @@ Zombie/Undead Tasks No system runs perfectly, and task instances are expected to die once in a

(airflow) branch constraints-main updated: Updating constraints. Github run id:7359589960

2023-12-29 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new b955c3444a Updating

[PR] prevent templated field logic checks in operators `__init__` in livy operator [airflow]

2023-12-29 Thread via GitHub
romsharon98 opened a new pull request, #36490: URL: https://github.com/apache/airflow/pull/36490 related: https://github.com/apache/airflow/issues/36484 fix livy operator for this cherry-picking: https://github.com/apache/airflow/pull/33786 --- **^

(airflow) branch main updated: prevent templated field logic checks in operators __init__ (#36489)

2023-12-29 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 918552acad prevent templated field logic checks in

Re: [PR] prevent templated field logic checks in operators `__init__` in dataproc [airflow]

2023-12-29 Thread via GitHub
potiuk merged PR #36489: URL: https://github.com/apache/airflow/pull/36489 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Add a migration script tooling support to migrate off mssql [airflow]

2023-12-29 Thread via GitHub
jscheffl closed pull request #35861: Add a migration script tooling support to migrate off mssql URL: https://github.com/apache/airflow/pull/35861 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Add a migration script tooling support to migrate off mssql [airflow]

2023-12-29 Thread via GitHub
jscheffl commented on PR #35861: URL: https://github.com/apache/airflow/pull/35861#issuecomment-1872320271 Migrated script to https://github.com/apache/airflow-mssql-migration -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

(airflow-mssql-migration) branch main created (now d6dbdb1)

2023-12-29 Thread jscheffl
This is an automated email from the ASF dual-hosted git repository. jscheffl pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow-mssql-migration.git at d6dbdb1 initial commit This branch includes the following new commits: new d6dbdb1 initial

(airflow-mssql-migration) 01/01: initial commit

2023-12-29 Thread jscheffl
This is an automated email from the ASF dual-hosted git repository. jscheffl pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow-mssql-migration.git commit d6dbdb123ffe5ce2385463740425f0c63ef72938 Author: Jens Scheffler AuthorDate: Fri Dec 29 21:18:57 2023

Re: [PR] Improve TaskInstance typing hints [airflow]

2023-12-29 Thread via GitHub
potiuk commented on code in PR #36487: URL: https://github.com/apache/airflow/pull/36487#discussion_r1438397310 ## airflow/models/taskinstance.py: ## @@ -3398,10 +3405,12 @@ def clear_db_references(self, session): for table in tables: session.execute(

Re: [PR] Add further details to replacement documentation [airflow]

2023-12-29 Thread via GitHub
potiuk merged PR #36485: URL: https://github.com/apache/airflow/pull/36485 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Add further details to replacement documentation [airflow]

2023-12-29 Thread via GitHub
potiuk commented on PR #36485: URL: https://github.com/apache/airflow/pull/36485#issuecomment-1872310023 cc: @uranusjr -> I remember we had discussion about it, this kind of advice is precisely what I was thinking is needed. -- This is an automated message from the Apache Git Service.

Re: [I] Task succeeded but retried --> downstream task fails since upstream task is still running [airflow]

2023-12-29 Thread via GitHub
jscheffl commented on issue #35773: URL: https://github.com/apache/airflow/issues/35773#issuecomment-1872282379 I would assess from the information that this is not a general bug, it is not common. Root cause could be a infrastructure or deployment problem such that local executor was not

[PR] prevent templated field logic checks in operators `__init__` in dataproc [airflow]

2023-12-29 Thread via GitHub
romsharon98 opened a new pull request, #36489: URL: https://github.com/apache/airflow/pull/36489 related: https://github.com/apache/airflow/issues/36484 fix dataproc operator when running: `pre-commit run --files airflow/providers/google/cloud/operators/dataproc.py`

Re: [I] run_as_user task report airflow.exceptions.DagRunNotFound: DagRun for tsy-succ-test with run_id or execution_date of '' not found [airflow]

2023-12-29 Thread via GitHub
jscheffl commented on issue #35906: URL: https://github.com/apache/airflow/issues/35906#issuecomment-1872279189 From the error report it is hard t judge what the root cause is. The only difference i notice from the changed DAG is that you remove the `run_as_user` attribute? Please

Re: [I] Missing AzureSynapseRunPipelineOperator in docs [airflow]

2023-12-29 Thread via GitHub
boring-cyborg[bot] commented on issue #36488: URL: https://github.com/apache/airflow/issues/36488#issuecomment-1872266149 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for

[I] Missing AzureSynapseRunPipelineOperator in docs [airflow]

2023-12-29 Thread via GitHub
KamilKandzia opened a new issue, #36488: URL: https://github.com/apache/airflow/issues/36488 ### What do you see as an issue? There are two operators in the synapse.py file (https://github.com/apache/airflow/blob/main/airflow/providers/microsoft/azure/operators/synapse.py), but on

Re: [PR] Remove MSSQL support from Airflow core [airflow]

2023-12-29 Thread via GitHub
potiuk commented on PR #35868: URL: https://github.com/apache/airflow/pull/35868#issuecomment-1872248350 > @potiuk Yes, before re-basing another 10 times, I'd like to get this off the plate and merged. So your advise and prep help might help: > > * I assume all feedback consistently

Re: [I] Airflow DAG manual runs conflict with scheduled runs [airflow]

2023-12-29 Thread via GitHub
RNHTTR commented on issue #36098: URL: https://github.com/apache/airflow/issues/36098#issuecomment-1872245707 I'm not able to reproduce this. Can you provide an example DAG? Something that stands out is that the first manual run was triggered at 14:05:00 but it didn't start running until

Re: [PR] Add description on the ways how users should approach DB monitoring [airflow]

2023-12-29 Thread via GitHub
potiuk commented on PR #36483: URL: https://github.com/apache/airflow/pull/36483#issuecomment-1872241850 Here it is - after employing Chat GPT (after few iterations and discussion with it) and applying the comments above. @BasPH - does it look better ? -- This is an automated message

Re: [I] Full screen mode for Logs, Graph, Code etc.. [airflow]

2023-12-29 Thread via GitHub
morooshka commented on issue #33012: URL: https://github.com/apache/airflow/issues/33012#issuecomment-1872234771 Hi Team! Is there a way to see Graph / Logs like it was previously? Like this ([v2.6.3](https://pypi.python.org/pypi/apache-airflow/2.6.3))? I mean not at the right of grid

(airflow) branch constraints-main updated: Updating constraints. Github run id:7358269526

2023-12-29 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 75bc2ed873 Updating

Re: [PR] Add description on the ways how users should approach DB monitoring [airflow]

2023-12-29 Thread via GitHub
potiuk commented on PR #36483: URL: https://github.com/apache/airflow/pull/36483#issuecomment-1872229139 > https://www.datadoghq.com/dg/monitor/databases/ is great for this kind of monitoring. I am not sure if we should endorse specific services in our docs, but I will leave enough

Re: [PR] Remove MSSQL support from Airflow core [airflow]

2023-12-29 Thread via GitHub
jscheffl commented on PR #35868: URL: https://github.com/apache/airflow/pull/35868#issuecomment-1872228582 @potiuk Yes, before re-basing another 10 times, I'd like to get this off the plate and merged. So your advise and prep help might help: - I assume all feedback consistently

Re: [I] Airflow Scheduler with Kubernetes Executor has errors in logs and stuck slots with no running tasks [airflow]

2023-12-29 Thread via GitHub
crabio commented on issue #36478: URL: https://github.com/apache/airflow/issues/36478#issuecomment-1872228151 Thank you (@dirrao) very much! I will try and be back soon -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Add description on the ways how users should approach DB monitoring [airflow]

2023-12-29 Thread via GitHub
potiuk commented on PR #36483: URL: https://github.com/apache/airflow/pull/36483#issuecomment-1872226453 > > If we can add a feature to automatically clean the data based on the retention parameter that we set in the airflow configuration. > > IMHO, the `db clean` command available

Re: [PR] Add description on the ways how users should approach DB monitoring [airflow]

2023-12-29 Thread via GitHub
potiuk commented on PR #36483: URL: https://github.com/apache/airflow/pull/36483#issuecomment-1872217674 > Great initiative. I have ideas along similar lines. I have observed in our use case where db grows (1GB per day) indefinitely unless manually run the db clean cli command. So, it will

[PR] Improve TaskInstance typing hints [airflow]

2023-12-29 Thread via GitHub
jscheffl opened a new pull request, #36487: URL: https://github.com/apache/airflow/pull/36487 As I was investigating for bug report #36100 I stumbled over a missing type-hint in task_instance.py, attempting to fix this, mypy raised more alarms which I am trying to fix with this PR.

(airflow) branch main updated: Fix assignment of template field in `__init__` in `hive_to_samba.py` (#36486)

2023-12-29 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 07226795ae Fix assignment of template field in

Re: [PR] Fix assignment of template field in `__init__` in `hive_to_samba.py` [airflow]

2023-12-29 Thread via GitHub
eladkal merged PR #36486: URL: https://github.com/apache/airflow/pull/36486 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Add description on the ways how users should approach DB monitoring [airflow]

2023-12-29 Thread via GitHub
hussein-awala commented on PR #36483: URL: https://github.com/apache/airflow/pull/36483#issuecomment-1872189592 > If we can add a feature to automatically clean the data based on the retention parameter that we set in the airflow configuration. IMHO, the `db clean` command available

[PR] Draft: Fix assignment of template field [airflow]

2023-12-29 Thread via GitHub
romsharon98 opened a new pull request, #36486: URL: https://github.com/apache/airflow/pull/36486 related: #36484 fix: airflow/providers/apache/hive/transfers/hive_to_samba.py --- **^ Add meaningful description above** Read the **[Pull Request

[PR] Add further details to replacement documentation [airflow]

2023-12-29 Thread via GitHub
jscheffl opened a new pull request, #36485: URL: https://github.com/apache/airflow/pull/36485 Add a bit more clarification to template deprecated parameters. related: #36100 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Prevent templated field logic checks in __init__ of operators automatically [airflow]

2023-12-29 Thread via GitHub
shahar1 commented on PR #33786: URL: https://github.com/apache/airflow/pull/33786#issuecomment-1872147699 > > @shahar1 I see that the pre-commit found some invalid usages which is awesome. > > I suggest to handle them in separated PR(s) then once all merged you can rebase this one and

[I] Fix existing custom operators' templated fields that include logic checks [airflow]

2023-12-29 Thread via GitHub
shahar1 opened a new issue, #36484: URL: https://github.com/apache/airflow/issues/36484 ### Apache Airflow version main (development) ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? ### Introduction PR #33786 aims

Re: [I] Invalid kube-config file. Expected key current-context in kube-config when using deferrable=True [airflow]

2023-12-29 Thread via GitHub
bjankie1 commented on issue #34644: URL: https://github.com/apache/airflow/issues/34644#issuecomment-1872146456 Before https://github.com/apache/airflow/pull/29498 the flow was as following: 1. In the operator `config_file` was read and deserialized to a map. 2. Serialized config was

Re: [I] Status of testing Providers that were prepared on December 28, 2023 [airflow]

2023-12-29 Thread via GitHub
shohamy7 commented on issue #36467: URL: https://github.com/apache/airflow/issues/36467#issuecomment-1872085752 Verified #36423 and #36413, looks good -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Add description on the ways how users should approach DB monitoring [airflow]

2023-12-29 Thread via GitHub
BasPH commented on PR #36483: URL: https://github.com/apache/airflow/pull/36483#issuecomment-1872078454 I agree with explaining monitoring of Airflow's meta-database in the docs. However, this chapter contains a lot of (IMO) unnecessary details which makes it difficult to extract the key

Re: [I] prev_execution_date_success deprication warning suggests using prev_data_interval_start_success which is not equivalent [airflow]

2023-12-29 Thread via GitHub
jscheffl commented on issue #36100: URL: https://github.com/apache/airflow/issues/36100#issuecomment-1872065857 Thanks for reporting this issue. In detail I agree that there might be a small difference between the two context variables but it is mainly caused by the fact that Airflow tries

Re: [PR] added cli command to list auth managers under `airflow providers` [airflow]

2023-12-29 Thread via GitHub
Bowrna commented on code in PR #36445: URL: https://github.com/apache/airflow/pull/36445#discussion_r1438242691 ## tests/always/test_providers_manager.py: ## @@ -384,6 +384,11 @@ def test_notification(self): notification_class_names =

Re: [PR] added cli command to list auth managers under `airflow providers` [airflow]

2023-12-29 Thread via GitHub
Bowrna commented on code in PR #36445: URL: https://github.com/apache/airflow/pull/36445#discussion_r1438242540 ## airflow/cli/commands/provider_command.py: ## @@ -189,7 +189,20 @@ def auth_backend_list(args): data=list(ProvidersManager().auth_backend_module_names),

(airflow) branch main updated: change spark connection form and add spark connections docs (#36419)

2023-12-29 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new ed9080ae6a change spark connection form and add

Re: [PR] change spark connection form and add spark connections docs [airflow]

2023-12-29 Thread via GitHub
eladkal merged PR #36419: URL: https://github.com/apache/airflow/pull/36419 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(airflow) branch constraints-main updated: Updating constraints. Github run id:7356343167

2023-12-29 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new bdedb3135d Updating

Re: [PR] Add description on the ways how users should approach DB monitoring [airflow]

2023-12-29 Thread via GitHub
jscheffl commented on code in PR #36483: URL: https://github.com/apache/airflow/pull/36483#discussion_r1438203426 ## docs/apache-airflow/howto/set-up-database.rst: ## @@ -383,6 +383,69 @@ After configuring the database and connecting to it in Airflow configuration, yo

Re: [PR] Remove MSSQL support from Airflow core [airflow]

2023-12-29 Thread via GitHub
potiuk commented on PR #35868: URL: https://github.com/apache/airflow/pull/35868#issuecomment-1871997004 Hey @jscheffl -> do you need my help / actions for that one? I am happy to create the separate repo, if needed and we could already arrange the PR in the way to be "ready to marge" when

(airflow) branch main updated: Check executable permission for entrypoints at breeze start (#36482)

2023-12-29 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 5551e14f6c Check executable permission for

Re: [PR] Check executable permission for entrypoints at breeze start [airflow]

2023-12-29 Thread via GitHub
potiuk merged PR #36482: URL: https://github.com/apache/airflow/pull/36482 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Add description on the ways how users should approach DB monitoring [airflow]

2023-12-29 Thread via GitHub
potiuk commented on PR #36483: URL: https://github.com/apache/airflow/pull/36483#issuecomment-1871984595 Inspired by discussion in https://github.com/apache/airflow/discussions/36454 - whenever user/issue is likely related to excessive DB usage, we should be able to just link the

[PR] Add description on the ways how users should approach DB monitoring [airflow]

2023-12-29 Thread via GitHub
potiuk opened a new pull request, #36483: URL: https://github.com/apache/airflow/pull/36483 Often our users are not aware that they are responsible for setting up and monitoring the database they chose as the metaa-data backend. While details of the tables and database structure of

Re: [PR] Check executable permission for entrypoints at breeze start [airflow]

2023-12-29 Thread via GitHub
potiuk commented on code in PR #36482: URL: https://github.com/apache/airflow/pull/36482#discussion_r1438143812 ## dev/breeze/src/airflow_breeze/utils/docker_command_utils.py: ## @@ -481,10 +482,30 @@ def prepare_broker_url(params, env_variables):

Re: [PR] Check executable permission for entrypoints at breeze start [airflow]

2023-12-29 Thread via GitHub
jscheffl commented on code in PR #36482: URL: https://github.com/apache/airflow/pull/36482#discussion_r1438135048 ## dev/breeze/src/airflow_breeze/utils/docker_command_utils.py: ## @@ -481,10 +482,30 @@ def prepare_broker_url(params, env_variables):

[PR] Check executable permission for entrypoints at breeze start [airflow]

2023-12-29 Thread via GitHub
potiuk opened a new pull request, #36482: URL: https://github.com/apache/airflow/pull/36482 Sometimes our contributors check out Airflow repository on filesystems that are not POSIX compliant and do not have support for executable bits (for example when you check-out the repository in

Re: [PR] Review and update our Pull Request review process and policy [airflow]

2023-12-29 Thread via GitHub
potiuk commented on PR #36472: URL: https://github.com/apache/airflow/pull/36472#issuecomment-1871876978 > I very much like the positive wording about comments and maintainer intentions. Yeah. I understand that our (my) comments sometimes seems rough and even sometimes even on the

Re: [PR] Review and update our Pull Request review process and policy [airflow]

2023-12-29 Thread via GitHub
potiuk commented on PR #36472: URL: https://github.com/apache/airflow/pull/36472#issuecomment-1871872733 Applied @jscheffl comments. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Review and update our Pull Request review process and policy [airflow]

2023-12-29 Thread via GitHub
potiuk commented on code in PR #36472: URL: https://github.com/apache/airflow/pull/36472#discussion_r1438112368 ## CONTRIBUTING.rst: ## @@ -1043,8 +1182,8 @@ as described in the static code checks documentation. Coding style and best practices ===

Re: [PR] Review and update our Pull Request review process and policy [airflow]

2023-12-29 Thread via GitHub
potiuk commented on code in PR #36472: URL: https://github.com/apache/airflow/pull/36472#discussion_r1438111964 ## CONTRIBUTING.rst: ## @@ -461,52 +494,159 @@ Step 5: Pass PR Review :align: center :alt: PR Review -Note that committers will use **Squash and Merge**

Re: [PR] Review and update our Pull Request review process and policy [airflow]

2023-12-29 Thread via GitHub
jscheffl commented on code in PR #36472: URL: https://github.com/apache/airflow/pull/36472#discussion_r1438104454 ## CONTRIBUTING.rst: ## @@ -461,52 +494,159 @@ Step 5: Pass PR Review :align: center :alt: PR Review -Note that committers will use **Squash and Merge**

Re: [PR] Add branching based on mapped task group example to dynamic-task-mapping.rst [airflow]

2023-12-29 Thread via GitHub
potiuk commented on PR #36480: URL: https://github.com/apache/airflow/pull/36480#issuecomment-1871867355 Some static checks. Highly recommend to have `pre-commit` installed :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Refactor _manage_executor_state by refreshing TIs in batch [airflow]

2023-12-29 Thread via GitHub
potiuk commented on PR #36418: URL: https://github.com/apache/airflow/pull/36418#issuecomment-1871846529 Nice one! @hussein-awala -> I just wonder if that could be the reason for https://github.com/apache/airflow/discussions/36454 -- This is an automated message from the Apache Git

Re: [I] Empty Details tab after upgrading to Airflow 2.8.0 (Dynamic Task Mapping) [airflow]

2023-12-29 Thread via GitHub
csp33 commented on issue #36481: URL: https://github.com/apache/airflow/issues/36481#issuecomment-1871836011 > Likely should be address before 2.8.1 (cc: @bbovenzi @pierrejeambrun). > > @csp33 -> can you see any errors in console log (open inspect view in your browser) when you open

Re: [I] Empty Details tab after upgrading to Airflow 2.8.0 (Dynamic Task Mapping) [airflow]

2023-12-29 Thread via GitHub
potiuk commented on issue #36481: URL: https://github.com/apache/airflow/issues/36481#issuecomment-1871829347 Likely should be address before 2.8.1 (cc: @bbovenzi @pierrejeambrun). @csp33 -> can you see any errors in console log (open inspect view in your browser) when you open the

Re: [PR] Fix XSS weakness in airflow/www/views.py [airflow]

2023-12-29 Thread via GitHub
potiuk commented on PR #36469: URL: https://github.com/apache/airflow/pull/36469#issuecomment-1871825549 > If that is indeed the case can you describe where in the pipeline validate_key is called on the request parameters so that we can better identify such cases in future and not report