[PR] Upgrade base node version in pre-commit [airflow]

2024-01-28 Thread via GitHub
potiuk opened a new pull request, #37050: URL: https://github.com/apache/airflow/pull/37050 The base node version in pre-commit was very old 18.6.0. Bumping it to latest released node version --- **^ Add meaningful description above** Read the **[Pull

Re: [PR] Draft: kubernetes version bump [airflow]

2024-01-28 Thread via GitHub
potiuk commented on PR #37040: URL: https://github.com/apache/airflow/pull/37040#issuecomment-1913513257 The problem was that you manually modified the `provider_dependencies.json" file and did not modify `pyproject.toml` (they are both generated from provider.yaml being the single source o

Re: [I] Dag grouping [airflow]

2024-01-28 Thread via GitHub
MatthewStrickland commented on issue #37042: URL: https://github.com/apache/airflow/issues/37042#issuecomment-1913515530 > Just for curiosity, did you try to create a pool for this scenario? I have an issue based when i consume an API and hit the rate limit, pools are good to manage this. A

[PR] Added SIRCLO to the list of companies using Apache Airflow and edit Tokopedia GitHub link [airflow]

2024-01-28 Thread via GitHub
enchant3dmango opened a new pull request, #37051: URL: https://github.com/apache/airflow/pull/37051 - [Added SIRCLO to the list of companies using Apache Airflow](https://github.com/apache/airflow/commit/d8e364077cabc126cd1129c162fc6db1e7801aca) - [Updated Tokopedia GitHub link](https://

Re: [PR] Upgrade base node version in pre-commit [airflow]

2024-01-28 Thread via GitHub
Taragolis commented on code in PR #37050: URL: https://github.com/apache/airflow/pull/37050#discussion_r1468790240 ## .pre-commit-config.yaml: ## @@ -18,7 +18,7 @@ default_stages: [commit, push] default_language_version: python: python3 - node: 18.6.0 + node: 21.6.1 Revi

[PR] Protect against manually updated generated dependencies [airflow]

2024-01-28 Thread via GitHub
potiuk opened a new pull request, #37052: URL: https://github.com/apache/airflow/pull/37052 The "generated/provider_dependencies.json" is regenerated automatically together with pyproject.toml from all the provider.yaml files. The file contains information about dependencies for all provide

Re: [PR] Draft: kubernetes version bump [airflow]

2024-01-28 Thread via GitHub
potiuk commented on PR #37040: URL: https://github.com/apache/airflow/pull/37040#issuecomment-1913519658 Protection against such manual updates here: https://github.com/apache/airflow/pull/37052 -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] Protect against manually updated generated dependencies [airflow]

2024-01-28 Thread via GitHub
potiuk commented on code in PR #37052: URL: https://github.com/apache/airflow/pull/37052#discussion_r1468791803 ## pyproject.toml: ## @@ -533,6 +534,7 @@ alibaba = [ "oss2>=2.14.0", ] amazon = [ + "PyAthena>=3.0.10", Review Comment: BTW. It turned out that PyAthena dep

Re: [PR] Protect against manually updated generated dependencies [airflow]

2024-01-28 Thread via GitHub
potiuk commented on PR #37052: URL: https://github.com/apache/airflow/pull/37052#issuecomment-1913521393 I also added a small thing - in the generated pyproject.toml I've added comment stating where the source of the information is - pointing to the provider.yaml file where the dependencies

[PR] Limit moto to version below 5.0.0 [airflow]

2024-01-28 Thread via GitHub
potiuk opened a new pull request, #37054: URL: https://github.com/apache/airflow/pull/37054 Moto 5 replaced all decorators with single mock_aws decorator and we need to Replace the usage of mock decorators in our tests to be able to use moto 5 Related: #37053

Re: [PR] Upgrade base node version in pre-commit [airflow]

2024-01-28 Thread via GitHub
potiuk commented on code in PR #37050: URL: https://github.com/apache/airflow/pull/37050#discussion_r1468797905 ## .pre-commit-config.yaml: ## @@ -18,7 +18,7 @@ default_stages: [commit, push] default_language_version: python: python3 - node: 18.6.0 + node: 21.6.1 Review

Re: [PR] Upgrade base node version in pre-commit [airflow]

2024-01-28 Thread via GitHub
potiuk commented on code in PR #37050: URL: https://github.com/apache/airflow/pull/37050#discussion_r1468798009 ## .pre-commit-config.yaml: ## @@ -18,7 +18,7 @@ default_stages: [commit, push] default_language_version: python: python3 - node: 18.6.0 + node: 21.6.1 Review

Re: [PR] Upgrade base node version in pre-commit [airflow]

2024-01-28 Thread via GitHub
potiuk commented on code in PR #37050: URL: https://github.com/apache/airflow/pull/37050#discussion_r1468798217 ## .pre-commit-config.yaml: ## @@ -18,7 +18,7 @@ default_stages: [commit, push] default_language_version: python: python3 - node: 18.6.0 + node: 21.6.1 Review

Re: [PR] Upgrade base node version in pre-commit [airflow]

2024-01-28 Thread via GitHub
potiuk commented on code in PR #37050: URL: https://github.com/apache/airflow/pull/37050#discussion_r1468798592 ## .pre-commit-config.yaml: ## @@ -18,7 +18,7 @@ default_stages: [commit, push] default_language_version: python: python3 - node: 18.6.0 + node: 21.6.1 Review

Re: [PR] Upgrade base node version in pre-commit [airflow]

2024-01-28 Thread via GitHub
Taragolis commented on code in PR #37050: URL: https://github.com/apache/airflow/pull/37050#discussion_r1468801621 ## .pre-commit-config.yaml: ## @@ -18,7 +18,7 @@ default_stages: [commit, push] default_language_version: python: python3 - node: 18.6.0 + node: 21.6.1 Revi

Re: [PR] Upgrade base node version in pre-commit [airflow]

2024-01-28 Thread via GitHub
Taragolis commented on code in PR #37050: URL: https://github.com/apache/airflow/pull/37050#discussion_r1468801621 ## .pre-commit-config.yaml: ## @@ -18,7 +18,7 @@ default_stages: [commit, push] default_language_version: python: python3 - node: 18.6.0 + node: 21.6.1 Revi

Re: [PR] Upgrade base node version in pre-commit [airflow]

2024-01-28 Thread via GitHub
Taragolis commented on code in PR #37050: URL: https://github.com/apache/airflow/pull/37050#discussion_r1468804107 ## .pre-commit-config.yaml: ## @@ -18,7 +18,7 @@ default_stages: [commit, push] default_language_version: python: python3 - node: 18.6.0 + node: 21.6.1 Revi

Re: [PR] Add "airflow users reset-password" command [airflow]

2024-01-28 Thread via GitHub
aritra24 commented on code in PR #37044: URL: https://github.com/apache/airflow/pull/37044#discussion_r1468805989 ## airflow/providers/fab/auth_manager/security_manager/override.py: ## @@ -470,7 +470,9 @@ def reset_password(self, userid, password): user = self.get_user_

Re: [PR] Add "airflow users reset-password" command [airflow]

2024-01-28 Thread via GitHub
aritra24 commented on code in PR #37044: URL: https://github.com/apache/airflow/pull/37044#discussion_r1468807594 ## airflow/providers/fab/auth_manager/cli_commands/user_command.py: ## @@ -106,6 +106,27 @@ def _find_user(args): return user +@cli_utils.action_cli +@provi

Re: [PR] Limit moto to version below 5.0.0 [airflow]

2024-01-28 Thread via GitHub
potiuk merged PR #37054: URL: https://github.com/apache/airflow/pull/37054 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

(airflow) branch main updated: Limit moto to version below 5.0.0 (#37054)

2024-01-28 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new d576c72c20 Limit moto to version below 5.0.0 (#3705

Re: [PR] Protect against manually updated generated dependencies [airflow]

2024-01-28 Thread via GitHub
potiuk commented on PR #37052: URL: https://github.com/apache/airflow/pull/37052#issuecomment-1913551396 > Wow, nice one! Oh. Yeah. I really, really like seeing someone problems/struggling and turning it into a solution to prevent them. -- This is an automated message from the Apac

Re: [PR] Move docs for cron basics to Authoring and Scheduling section [airflow]

2024-01-28 Thread via GitHub
ketozhang commented on code in PR #37049: URL: https://github.com/apache/airflow/pull/37049#discussion_r1468818504 ## docs/apache-airflow/core-concepts/dags.rst: ## @@ -190,18 +190,20 @@ DAGs do not *require* a schedule, but it's very common to define one. You define with

Re: [PR] Move docs for cron basics to Authoring and Scheduling section [airflow]

2024-01-28 Thread via GitHub
ketozhang commented on PR #37049: URL: https://github.com/apache/airflow/pull/37049#issuecomment-1913551825 Build seems to fail for unrelated files. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] Upgrade base node version in pre-commit [airflow]

2024-01-28 Thread via GitHub
potiuk commented on code in PR #37050: URL: https://github.com/apache/airflow/pull/37050#discussion_r1468820190 ## .pre-commit-config.yaml: ## @@ -18,7 +18,7 @@ default_stages: [commit, push] default_language_version: python: python3 - node: 18.6.0 + node: 21.6.1 Review

Re: [PR] Upgrade base node version in pre-commit [airflow]

2024-01-28 Thread via GitHub
potiuk commented on code in PR #37050: URL: https://github.com/apache/airflow/pull/37050#discussion_r1468820372 ## .pre-commit-config.yaml: ## @@ -18,7 +18,7 @@ default_stages: [commit, push] default_language_version: python: python3 - node: 18.6.0 + node: 21.6.1 Review

Re: [PR] Upgrade base node version in pre-commit [airflow]

2024-01-28 Thread via GitHub
potiuk commented on code in PR #37050: URL: https://github.com/apache/airflow/pull/37050#discussion_r1468820190 ## .pre-commit-config.yaml: ## @@ -18,7 +18,7 @@ default_stages: [commit, push] default_language_version: python: python3 - node: 18.6.0 + node: 21.6.1 Review

[PR] Add support for catchup="ignore_first" to disable catchup for the first DagRun [airflow]

2024-01-28 Thread via GitHub
hussein-awala opened a new pull request, #35392: URL: https://github.com/apache/airflow/pull/35392 This PR updates the type of catchup to string and adds support for the `ignore_first` value, to ignore catchup only for the first DagRub. ### Motivations: The `start_date` is a ma

[PR] SqlToS3Operator: feat/ add max_rows_per_file parameter [airflow]

2024-01-28 Thread via GitHub
selimchergui opened a new pull request, #37055: URL: https://github.com/apache/airflow/pull/37055 **Description** `SqlToS3Operator` create an S3 object from the output of sql qurey. Unless you specify a groupby_kwargs parameter, the entire data will be written in one object. This could r

Re: [PR] SqlToS3Operator: feat/ add max_rows_per_file parameter [airflow]

2024-01-28 Thread via GitHub
boring-cyborg[bot] commented on PR #37055: URL: https://github.com/apache/airflow/pull/37055#issuecomment-1913559242 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors'

Re: [PR] Add "airflow users reset-password" command [airflow]

2024-01-28 Thread via GitHub
potiuk commented on PR #37044: URL: https://github.com/apache/airflow/pull/37044#issuecomment-1913561301 @jscheffl ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsu

Re: [PR] Protect against manually updated generated dependencies [airflow]

2024-01-28 Thread via GitHub
potiuk commented on PR #37052: URL: https://github.com/apache/airflow/pull/37052#issuecomment-1913563203 AH, I need to close that one and open one from the `apache` repo - because we have the case that `dev` scripts that are used in `pull_request_workflow` are modified. -- This is an aut

Re: [PR] Protect against manually updated generated dependencies [airflow]

2024-01-28 Thread via GitHub
potiuk closed pull request #37052: Protect against manually updated generated dependencies URL: https://github.com/apache/airflow/pull/37052 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

(airflow) 01/01: Protect against manually updated generated dependencies

2024-01-28 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch protect-against-not-regenerated-pyproject-toml in repository https://gitbox.apache.org/repos/asf/airflow.git commit 20906699b6385fd8badc235d80e4b8530cb659d5 Author: Jarek Potiuk AuthorDate: Sun J

(airflow) branch protect-against-not-regenerated-pyproject-toml created (now 20906699b6)

2024-01-28 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch protect-against-not-regenerated-pyproject-toml in repository https://gitbox.apache.org/repos/asf/airflow.git at 20906699b6 Protect against manually updated generated dependencies This bran

[PR] Protect against manually updated generated dependencies [airflow]

2024-01-28 Thread via GitHub
potiuk opened a new pull request, #37056: URL: https://github.com/apache/airflow/pull/37056 The "generated/provider_dependencies.json" is regenerated automatically together with pyproject.toml from all the provider.yaml files. The file contains information about dependencies for all provide

(airflow) branch constraints-main updated: Updating constraints. Github run id:7685090996

2024-01-28 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new a763994740 Updating con

(airflow) branch protect-against-not-regenerated-pyproject-toml updated (20906699b6 -> cf6c125494)

2024-01-28 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch protect-against-not-regenerated-pyproject-toml in repository https://gitbox.apache.org/repos/asf/airflow.git discard 20906699b6 Protect against manually updated generated dependencies add c

Re: [PR] Add "airflow users reset-password" command [airflow]

2024-01-28 Thread via GitHub
jscheffl merged PR #37044: URL: https://github.com/apache/airflow/pull/37044 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

Re: [I] Add "airflow users change-user-password" [airflow]

2024-01-28 Thread via GitHub
jscheffl closed issue #37009: Add "airflow users change-user-password" URL: https://github.com/apache/airflow/issues/37009 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

(airflow) branch main updated: Add "airflow users reset-password" command (#37044)

2024-01-28 Thread jscheffl
This is an automated email from the ASF dual-hosted git repository. jscheffl pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 0fce3b6047 Add "airflow users reset-password" com

Re: [PR] feat: Stream Transfer Enhancement for Airflow SFTP to GCS Operator [airflow]

2024-01-28 Thread via GitHub
he-gong-sardine commented on PR #36545: URL: https://github.com/apache/airflow/pull/36545#issuecomment-1913579619 > > Documentation updates are pending. I am currently seeking guidance on locating the source files for documentation to ensure all necessary updates are made. > > Assumi

[PR] Allow committer builds to use scripts/ci, dev and actions from the PR [airflow]

2024-01-28 Thread via GitHub
potiuk opened a new pull request, #37057: URL: https://github.com/apache/airflow/pull/37057 In our process, we generally do not let the scripts in the "build images" workflow to use `scripts/ci`, `dev` and `action` scripts to come from the PR. This is a security feature that prevent Pull Re

Re: [PR] heartbeat recovery message [airflow]

2024-01-28 Thread via GitHub
Bowrna commented on code in PR #34457: URL: https://github.com/apache/airflow/pull/34457#discussion_r1468854809 ## airflow/jobs/job.py: ## @@ -126,7 +127,19 @@ def executor(self): def heartrate(self) -> float: return Job._heartrate(self.job_type) -def is_aliv

Re: [PR] Protect against manually updated generated dependencies [airflow]

2024-01-28 Thread via GitHub
potiuk commented on PR #37052: URL: https://github.com/apache/airflow/pull/37052#issuecomment-1913593942 All right. I actually found a way, how to avoid having to do that again: https://github.com/apache/airflow/pull/37057 -- This is an automated message from the Apache Git Service. To r

Re: [PR] Allow committer builds to use scripts/ci, dev and actions from the PR [airflow]

2024-01-28 Thread via GitHub
potiuk commented on PR #37057: URL: https://github.com/apache/airflow/pull/37057#issuecomment-1913594227 Once we merge that one, committers will have no more the problem that their PRs modifying `dev` scripts will have to close it and re-open it from the `apache` repository. It should **jus

Re: [PR] feat: Stream Transfer Enhancement for Airflow SFTP to GCS Operator [airflow]

2024-01-28 Thread via GitHub
he-gong-sardine commented on code in PR #36545: URL: https://github.com/apache/airflow/pull/36545#discussion_r1468859918 ## airflow/providers/google/cloud/transfers/sftp_to_gcs.py: ## @@ -168,6 +204,68 @@ def _copy_single_object( self.log.info("Executing delete of %

Re: [PR] feat: Stream Transfer Enhancement for Airflow SFTP to GCS Operator [airflow]

2024-01-28 Thread via GitHub
he-gong-sardine commented on code in PR #36545: URL: https://github.com/apache/airflow/pull/36545#discussion_r1468859918 ## airflow/providers/google/cloud/transfers/sftp_to_gcs.py: ## @@ -168,6 +204,68 @@ def _copy_single_object( self.log.info("Executing delete of %

Re: [PR] Allow committer builds to use scripts/ci, dev and actions from the PR [airflow]

2024-01-28 Thread via GitHub
hussein-awala commented on code in PR #37057: URL: https://github.com/apache/airflow/pull/37057#discussion_r1468865093 ## dev/breeze/src/airflow_breeze/utils/selective_checks.py: ## @@ -1039,3 +1040,9 @@ def providers_compatibility_checks(self) -> str: if check[

Re: [PR] feat: Stream Transfer Enhancement for Airflow SFTP to GCS Operator [airflow]

2024-01-28 Thread via GitHub
he-gong-sardine commented on PR #36545: URL: https://github.com/apache/airflow/pull/36545#issuecomment-1913606077 > > * High Memory (Over 100MB): It is recommended to use Google's upload_from_file method, which is more robust and includes a retry feature. This method is preferable when suff

Re: [PR] Allow committer builds to use scripts/ci, dev and actions from the PR [airflow]

2024-01-28 Thread via GitHub
potiuk commented on code in PR #37057: URL: https://github.com/apache/airflow/pull/37057#discussion_r1468866371 ## dev/breeze/src/airflow_breeze/utils/selective_checks.py: ## @@ -1039,3 +1040,9 @@ def providers_compatibility_checks(self) -> str: if check["python

Re: [PR] feat: Stream Transfer Enhancement for Airflow SFTP to GCS Operator [airflow]

2024-01-28 Thread via GitHub
he-gong-sardine commented on code in PR #36545: URL: https://github.com/apache/airflow/pull/36545#discussion_r1468866487 ## airflow/providers/google/cloud/transfers/sftp_to_gcs.py: ## @@ -160,10 +160,8 @@ def execute(self, context: Context): for file in files:

Re: [PR] Allow committer builds to use scripts/ci, dev and actions from the PR [airflow]

2024-01-28 Thread via GitHub
potiuk commented on code in PR #37057: URL: https://github.com/apache/airflow/pull/37057#discussion_r1468868060 ## dev/breeze/src/airflow_breeze/utils/selective_checks.py: ## @@ -1039,3 +1040,9 @@ def providers_compatibility_checks(self) -> str: if check["python

Re: [PR] Protect against manually updated generated dependencies [airflow]

2024-01-28 Thread via GitHub
potiuk merged PR #37056: URL: https://github.com/apache/airflow/pull/37056 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

(airflow) branch main updated: Protect against manually updated generated dependencies (#37056)

2024-01-28 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 42017f5bd8 Protect against manually updated generat

Re: [PR] feat: Stream Transfer Enhancement for Airflow SFTP to GCS Operator [airflow]

2024-01-28 Thread via GitHub
he-gong-sardine commented on code in PR #36545: URL: https://github.com/apache/airflow/pull/36545#discussion_r1468869304 ## airflow/providers/google/cloud/transfers/sftp_to_gcs.py: ## @@ -168,6 +204,68 @@ def _copy_single_object( self.log.info("Executing delete of %

Re: [PR] feat: Stream Transfer Enhancement for Airflow SFTP to GCS Operator [airflow]

2024-01-28 Thread via GitHub
he-gong-sardine commented on code in PR #36545: URL: https://github.com/apache/airflow/pull/36545#discussion_r1468869381 ## airflow/providers/google/cloud/transfers/sftp_to_gcs.py: ## @@ -168,6 +204,68 @@ def _copy_single_object( self.log.info("Executing delete of %

[PR] AIP-58: Add object storage backend for xcom [airflow]

2024-01-28 Thread via GitHub
bolkedebruin opened a new pull request, #37058: URL: https://github.com/apache/airflow/pull/37058 This adds the possibility to store xcoms on a object storage supported backend. This is an initial implementation and I am open for feedback (duh ;-) ). The BaseXCom is a bit messy and r

[I] Triggerer error "deque mutated during iteration" [airflow]

2024-01-28 Thread via GitHub
zhenghanyang opened a new issue, #37059: URL: https://github.com/apache/airflow/issues/37059 ### Apache Airflow version 2.8.1 ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? When start more than 20 thousands trigger job

Re: [I] WORKER_AUTOSCALE should not allow max workers < min workers [airflow]

2024-01-28 Thread via GitHub
RNHTTR closed issue #35332: WORKER_AUTOSCALE should not allow max workers < min workers URL: https://github.com/apache/airflow/issues/35332 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] Add SIRCLO to the list of companies using Apache Airflow and update Tokopedia GitHub link [airflow]

2024-01-28 Thread via GitHub
jscheffl merged PR #37051: URL: https://github.com/apache/airflow/pull/37051 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

(airflow) branch main updated: Add SIRCLO to the list of companies using Apache Airflow and update Tokopedia GitHub link (#37051)

2024-01-28 Thread jscheffl
This is an automated email from the ASF dual-hosted git repository. jscheffl pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 777dfefccc Add SIRCLO to the list of companies us

Re: [I] WORKER_AUTOSCALE should not allow max workers < min workers [airflow]

2024-01-28 Thread via GitHub
Bowrna commented on issue #35332: URL: https://github.com/apache/airflow/issues/35332#issuecomment-1913621853 @RNHTTR shall we close this issue after I fix the documentation? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Allow committer builds to use scripts/ci, dev and actions from the PR [airflow]

2024-01-28 Thread via GitHub
potiuk commented on code in PR #37057: URL: https://github.com/apache/airflow/pull/37057#discussion_r1468880404 ## dev/breeze/src/airflow_breeze/utils/selective_checks.py: ## @@ -1039,3 +1040,9 @@ def providers_compatibility_checks(self) -> str: if check["python

Re: [PR] Allow committer builds to use scripts/ci, dev and actions from the PR [airflow]

2024-01-28 Thread via GitHub
potiuk commented on PR #37057: URL: https://github.com/apache/airflow/pull/37057#issuecomment-1913626242 I will do some more testing in my own fork, to see if I have everything righ. And I'd appreciate a bit more thorough review for that one. This is a sensitive part of our actions

Re: [I] Dag grouping [airflow]

2024-01-28 Thread via GitHub
RNHTTR commented on issue #37042: URL: https://github.com/apache/airflow/issues/37042#issuecomment-1913636521 Could you use a [cluster DAG policy](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/cluster-policies.html#)? You can add the desired DAGs into a

Re: [I] K8S Spark Operator doesn't delete an older application with the same name anymore [airflow]

2024-01-28 Thread via GitHub
TKul6 commented on issue #32363: URL: https://github.com/apache/airflow/issues/32363#issuecomment-1913639337 Hi Guys, any update on that? it would help me as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] Upgrade base node version in pre-commit [airflow]

2024-01-28 Thread via GitHub
Taragolis commented on code in PR #37050: URL: https://github.com/apache/airflow/pull/37050#discussion_r1468891015 ## .pre-commit-config.yaml: ## @@ -18,7 +18,7 @@ default_stages: [commit, push] default_language_version: python: python3 - node: 18.6.0 + node: 21.6.1 Revi

[PR] Make Amazon Provider tests compatible with `moto>=5` [airflow]

2024-01-28 Thread via GitHub
Taragolis opened a new pull request, #37060: URL: https://github.com/apache/airflow/pull/37060 closes: https://github.com/apache/airflow/issues/37053 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/air

Re: [PR] Fix KPO docs argument precedence [airflow]

2024-01-28 Thread via GitHub
amoghrajesh commented on code in PR #37047: URL: https://github.com/apache/airflow/pull/37047#discussion_r1468895610 ## docs/apache-airflow-providers-cncf-kubernetes/operators.rst: ## @@ -77,8 +77,8 @@ You can print out the Kubernetes manifest for the pod that would be created

Re: [PR] Upgrade base node version in pre-commit [airflow]

2024-01-28 Thread via GitHub
potiuk merged PR #37050: URL: https://github.com/apache/airflow/pull/37050 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

(airflow) branch main updated: Upgrade base node version in pre-commit (#37050)

2024-01-28 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 7bff7a5741 Upgrade base node version in pre-commit

Re: [PR] Move docs for cron basics to Authoring and Scheduling section [airflow]

2024-01-28 Thread via GitHub
potiuk commented on code in PR #37049: URL: https://github.com/apache/airflow/pull/37049#discussion_r1468900940 ## docs/apache-airflow/core-concepts/dags.rst: ## @@ -190,18 +190,20 @@ DAGs do not *require* a schedule, but it's very common to define one. You define with DAG

Re: [PR] SqlToS3Operator: feat/ add max_rows_per_file parameter [airflow]

2024-01-28 Thread via GitHub
potiuk commented on code in PR #37055: URL: https://github.com/apache/airflow/pull/37055#discussion_r1468901595 ## airflow/providers/amazon/aws/transfers/sql_to_s3.py: ## @@ -194,13 +197,32 @@ def execute(self, context: Context) -> None: def _partition_dataframe(self, df:

Re: [PR] AIP-58: Add object storage backend for xcom [airflow]

2024-01-28 Thread via GitHub
potiuk commented on code in PR #37058: URL: https://github.com/apache/airflow/pull/37058#discussion_r1468901766 ## airflow/config_templates/config.yml: ## @@ -375,6 +375,21 @@ core: type: string example: "path.to.CustomXCom" default: "airflow.models.xcom.Bas

[I] Potential TOCTOU Bug [airflow]

2024-01-28 Thread via GitHub
nevercodecorrect opened a new issue, #37061: URL: https://github.com/apache/airflow/issues/37061 ### Apache Airflow version 2.8.1 ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? In [code](https://github.com/apache/airf

Re: [PR] AIP-58: Add object storage backend for xcom [airflow]

2024-01-28 Thread via GitHub
potiuk commented on code in PR #37058: URL: https://github.com/apache/airflow/pull/37058#discussion_r1468902862 ## airflow/config_templates/config.yml: ## @@ -375,6 +375,21 @@ core: type: string example: "path.to.CustomXCom" default: "airflow.models.xcom.Bas

Re: [PR] AIP-58: Add object storage backend for xcom [airflow]

2024-01-28 Thread via GitHub
potiuk commented on code in PR #37058: URL: https://github.com/apache/airflow/pull/37058#discussion_r1468903325 ## airflow/io/xcom.py: ## @@ -0,0 +1,111 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE fi

Re: [PR] AIP-58: Add object storage backend for xcom [airflow]

2024-01-28 Thread via GitHub
potiuk commented on code in PR #37058: URL: https://github.com/apache/airflow/pull/37058#discussion_r1468903437 ## airflow/io/xcom.py: ## @@ -0,0 +1,111 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE fi

Re: [PR] AIP-58: Add object storage backend for xcom [airflow]

2024-01-28 Thread via GitHub
potiuk commented on code in PR #37058: URL: https://github.com/apache/airflow/pull/37058#discussion_r1468903437 ## airflow/io/xcom.py: ## @@ -0,0 +1,111 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE fi

Re: [PR] AIP-58: Add object storage backend for xcom [airflow]

2024-01-28 Thread via GitHub
potiuk commented on code in PR #37058: URL: https://github.com/apache/airflow/pull/37058#discussion_r1468905770 ## airflow/io/xcom.py: ## @@ -0,0 +1,111 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE fi

Re: [PR] Simplify hive client connection [airflow]

2024-01-28 Thread via GitHub
eladkal commented on code in PR #37043: URL: https://github.com/apache/airflow/pull/37043#discussion_r1468906095 ## airflow/providers/apache/hive/CHANGELOG.rst: ## @@ -27,6 +27,18 @@ Changelog - +7.0.0 +. + + +Breaking changes + + +* ``use_beeline

Re: [PR] AIP-58: Add object storage backend for xcom [airflow]

2024-01-28 Thread via GitHub
potiuk commented on code in PR #37058: URL: https://github.com/apache/airflow/pull/37058#discussion_r1468907283 ## airflow/io/xcom.py: ## @@ -0,0 +1,111 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE fi

Re: [PR] Move docs for cron basics to Authoring and Scheduling section [airflow]

2024-01-28 Thread via GitHub
potiuk commented on PR #37049: URL: https://github.com/apache/airflow/pull/37049#issuecomment-1913669251 No. It's relateed and caused by the build error you already know about -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

Re: [PR] Make Amazon Provider tests compatible with `moto>=5` [airflow]

2024-01-28 Thread via GitHub
potiuk commented on PR #37060: URL: https://github.com/apache/airflow/pull/37060#issuecomment-1913670655 Almost :) . There are few non-db tests remaining :D -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] Potential TOCTOU Bug [airflow]

2024-01-28 Thread via GitHub
potiuk closed issue #37061: Potential TOCTOU Bug URL: https://github.com/apache/airflow/issues/37061 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: com

Re: [I] Potential TOCTOU Bug [airflow]

2024-01-28 Thread via GitHub
potiuk commented on issue #37061: URL: https://github.com/apache/airflow/issues/37061#issuecomment-1913671207 Please, -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

Re: [I] Potential TOCTOU Bug [airflow]

2024-01-28 Thread via GitHub
potiuk commented on issue #37061: URL: https://github.com/apache/airflow/issues/37061#issuecomment-1913671291 Please report such things following our policy https://github.com/apache/airflow/security/policy -- This is an automated message from the Apache Git Service. To respond to the mes

Re: [PR] AIP-58: Add object storage backend for xcom [airflow]

2024-01-28 Thread via GitHub
bolkedebruin commented on code in PR #37058: URL: https://github.com/apache/airflow/pull/37058#discussion_r1468917344 ## airflow/config_templates/config.yml: ## @@ -375,6 +375,21 @@ core: type: string example: "path.to.CustomXCom" default: "airflow.models.xc

Re: [PR] AIP-58: Add object storage backend for xcom [airflow]

2024-01-28 Thread via GitHub
bolkedebruin commented on code in PR #37058: URL: https://github.com/apache/airflow/pull/37058#discussion_r1468917399 ## airflow/io/xcom.py: ## @@ -0,0 +1,111 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOT

Re: [PR] AIP-58: Add object storage backend for xcom [airflow]

2024-01-28 Thread via GitHub
bolkedebruin commented on code in PR #37058: URL: https://github.com/apache/airflow/pull/37058#discussion_r1468917596 ## airflow/io/xcom.py: ## @@ -0,0 +1,111 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOT

Re: [PR] prevent celery command autoscale misconfig [airflow]

2024-01-28 Thread via GitHub
Bowrna commented on PR #36576: URL: https://github.com/apache/airflow/pull/36576#issuecomment-1913687050 i have fixed some parts in documentation alone here in this PR. The actual issue is not an issue and therefore I just enhanced the docs alone -- This is an automated message from the A

Re: [PR] AIP-58: Add object storage backend for xcom [airflow]

2024-01-28 Thread via GitHub
bolkedebruin commented on code in PR #37058: URL: https://github.com/apache/airflow/pull/37058#discussion_r1468917908 ## airflow/io/xcom.py: ## @@ -0,0 +1,111 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOT

Re: [PR] AIP-58: Add object storage backend for xcom [airflow]

2024-01-28 Thread via GitHub
potiuk commented on code in PR #37058: URL: https://github.com/apache/airflow/pull/37058#discussion_r1468918789 ## airflow/io/xcom.py: ## @@ -0,0 +1,111 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE fi

Re: [PR] AIP-58: Add object storage backend for xcom [airflow]

2024-01-28 Thread via GitHub
potiuk commented on code in PR #37058: URL: https://github.com/apache/airflow/pull/37058#discussion_r1468918879 ## airflow/config_templates/config.yml: ## @@ -375,6 +375,21 @@ core: type: string example: "path.to.CustomXCom" default: "airflow.models.xcom.Bas

Re: [PR] AIP-58: Add object storage backend for xcom [airflow]

2024-01-28 Thread via GitHub
bolkedebruin commented on code in PR #37058: URL: https://github.com/apache/airflow/pull/37058#discussion_r1468917908 ## airflow/io/xcom.py: ## @@ -0,0 +1,111 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOT

Re: [PR] Allow committer builds to use scripts/ci, dev and actions from the PR [airflow]

2024-01-28 Thread via GitHub
potiuk commented on PR #37057: URL: https://github.com/apache/airflow/pull/37057#issuecomment-1913692464 All right I did some testing and applied a few smal fixes (optimiized away the checkout for target-branch for commiter build as it turned out to be not needed For anyone looking h

Re: [PR] Allow committer builds to use scripts/ci, dev and actions from the PR [airflow]

2024-01-28 Thread via GitHub
potiuk commented on code in PR #37057: URL: https://github.com/apache/airflow/pull/37057#discussion_r1468921710 ## dev/breeze/src/airflow_breeze/utils/selective_checks.py: ## @@ -1039,3 +1040,9 @@ def providers_compatibility_checks(self) -> str: if check["python

Re: [PR] prevent celery command autoscale misconfig [airflow]

2024-01-28 Thread via GitHub
potiuk merged PR #36576: URL: https://github.com/apache/airflow/pull/36576 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

(airflow) branch main updated: prevent celery command autoscale misconfig (#36576)

2024-01-28 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 4fa8e45c92 prevent celery command autoscale misconf

  1   2   >