Re: [PR] Allowing airflow kerberos mode to be configurable [airflow]

2023-10-23 Thread via GitHub
amoghrajesh commented on code in PR #35146: URL: https://github.com/apache/airflow/pull/35146#discussion_r1369697847 ## airflow/config_templates/config.yml: ## @@ -2350,6 +2350,14 @@ triggerer: kerberos: description: ~ options: +mode: + description: | +Sp

[PR] Allowing airflow kerberos mode to be configurable [airflow]

2023-10-23 Thread via GitHub
amoghrajesh opened a new pull request, #35146: URL: https://github.com/apache/airflow/pull/35146 I was trying to run airflow kerberos as an init container for airflow worker pods (I am using K8SExecutor with K8S deployment) Realised that there is no mode that can be define

Re: [PR] AIP-58: Add Airflow ObjectStore (AFS) [airflow]

2023-10-23 Thread via GitHub
bolkedebruin commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1369692131 ## airflow/providers/amazon/aws/fs/s3.py: ## @@ -0,0 +1,123 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agree

Re: [PR] AIP-58: Add Airflow ObjectStore (AFS) [airflow]

2023-10-23 Thread via GitHub
bolkedebruin commented on PR #34729: URL: https://github.com/apache/airflow/pull/34729#issuecomment-1776620355 > * If the plugin interface is extended does it mean "newer" extended providers like AWS will work on older versions e.g. Airflow 2.7.3 (See also the comment in CWiki ([Comment

Re: [PR] Add use_krb5ccache option to SparkSubmitHook [airflow]

2023-10-23 Thread via GitHub
eladkal commented on PR #34386: URL: https://github.com/apache/airflow/pull/34386#issuecomment-1776612174 @zeotuan do we need to expose this parameter in SparkSubmitOperator as well? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

Re: [PR] Move authentication related methods to FAB auth manager [airflow]

2023-10-23 Thread via GitHub
uranusjr commented on code in PR #35038: URL: https://github.com/apache/airflow/pull/35038#discussion_r1369679932 ## airflow/auth/managers/fab/security_manager/override.py: ## @@ -1650,3 +1950,82 @@ def _ldap_calculate_user_roles(self, user_attributes: dict[str, list[bytes]]) -

Re: [PR] Add task parameter to set custom logger name [airflow]

2023-10-23 Thread via GitHub
Joffreybvn commented on code in PR #34964: URL: https://github.com/apache/airflow/pull/34964#discussion_r1369648913 ## airflow/models/baseoperator.py: ## @@ -1218,15 +1225,17 @@ def __deepcopy__(self, memo): result.__instantiated = self.__instantiated return re

Re: [PR] Fix SalesforceHook compatiblity with Pandas 2.x [airflow]

2023-10-23 Thread via GitHub
boring-cyborg[bot] commented on PR #35145: URL: https://github.com/apache/airflow/pull/35145#issuecomment-1776572547 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution

[PR] Fix SalesforceHook compatiblity with Pandas 2.x [airflow]

2023-10-23 Thread via GitHub
dave-pollock opened a new pull request, #35145: URL: https://github.com/apache/airflow/pull/35145 In pandas 2.0.0, `pandas.np` was [removed](https://pandas.pydata.org/pandas-docs/version/2.1.0/whatsnew/v2.0.0.html#removal-of-prior-version-deprecations-changes). Airflow 2.7.0 switched

Re: [PR] Add extra operator links for EMR Serverless [airflow]

2023-10-23 Thread via GitHub
eladkal commented on PR #34225: URL: https://github.com/apache/airflow/pull/34225#issuecomment-1776568909 @dacort are you still working on it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] openlineage, aws: Add OpenLineage support for AthenaOperator. [airflow]

2023-10-23 Thread via GitHub
eladkal commented on code in PR #35090: URL: https://github.com/apache/airflow/pull/35090#discussion_r1369651269 ## airflow/providers/amazon/aws/operators/athena.py: ## @@ -163,3 +167,132 @@ def on_kill(self) -> None: "Polling Athena for query with id %s

Re: [PR] Add task parameter to set custom logger name [airflow]

2023-10-23 Thread via GitHub
Joffreybvn commented on code in PR #34964: URL: https://github.com/apache/airflow/pull/34964#discussion_r1369648913 ## airflow/models/baseoperator.py: ## @@ -1218,15 +1225,17 @@ def __deepcopy__(self, memo): result.__instantiated = self.__instantiated return re

[airflow] branch main updated: Use base aws classes in Amazon Athena Operators/Sensors/Triggers (#35133)

2023-10-23 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new da456065df Use base aws classes in Amazon Athena

Re: [PR] Use base aws classes in Amazon Athena Operators/Sensors/Triggers [airflow]

2023-10-23 Thread via GitHub
eladkal merged PR #35133: URL: https://github.com/apache/airflow/pull/35133 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.

Re: [PR] Add task parameter to set custom logger name [airflow]

2023-10-23 Thread via GitHub
Joffreybvn commented on code in PR #34964: URL: https://github.com/apache/airflow/pull/34964#discussion_r1369648913 ## airflow/models/baseoperator.py: ## @@ -1218,15 +1225,17 @@ def __deepcopy__(self, memo): result.__instantiated = self.__instantiated return re

Re: [PR] change latest to stable to fix broken documentation links [airflow]

2023-10-23 Thread via GitHub
theaadya commented on PR #35143: URL: https://github.com/apache/airflow/pull/35143#issuecomment-1776561134 I have updated the code with the new migration link. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [PR] Sqlalchemy 2.0 changes [airflow]

2023-10-23 Thread via GitHub
phanikumv commented on code in PR #35120: URL: https://github.com/apache/airflow/pull/35120#discussion_r1369640176 ## airflow/operators/trigger_dagrun.py: ## @@ -227,12 +228,10 @@ def execute_complete(self, context: Context, session: Session, event: tuple[str, # This e

Re: [PR] Add task parameter to set custom logger name [airflow]

2023-10-23 Thread via GitHub
uranusjr commented on code in PR #34964: URL: https://github.com/apache/airflow/pull/34964#discussion_r1369637078 ## airflow/models/baseoperator.py: ## @@ -1218,15 +1225,17 @@ def __deepcopy__(self, memo): result.__instantiated = self.__instantiated return resu

Re: [PR] Add task parameter to set custom logger name [airflow]

2023-10-23 Thread via GitHub
uranusjr commented on code in PR #34964: URL: https://github.com/apache/airflow/pull/34964#discussion_r1369637078 ## airflow/models/baseoperator.py: ## @@ -1218,15 +1225,17 @@ def __deepcopy__(self, memo): result.__instantiated = self.__instantiated return resu

Re: [PR] change latest to stable to fix broken documentation links [airflow]

2023-10-23 Thread via GitHub
theaadya commented on PR #35143: URL: https://github.com/apache/airflow/pull/35143#issuecomment-1776541924 Thanks! I'll update it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [PR] fix(helm): set worker safeToEvict properly [airflow]

2023-10-23 Thread via GitHub
jedcunningham commented on PR #35130: URL: https://github.com/apache/airflow/pull/35130#issuecomment-1776502995 @hakuno, Hussein meant a test in our test suit to make sure this doesn't regress. [Here is an example](https://github.com/apache/airflow/blob/f457228bd21b13a7fdb29e154202d5357d024

Re: [I] Support for general OIDC providers or making it clear in document [airflow]

2023-10-23 Thread via GitHub
rqi14 commented on issue #35131: URL: https://github.com/apache/airflow/issues/35131#issuecomment-1776454682 > > This behaviour should be documented. Otherwise, there should be a way to configure other OIDC providers like other projects that support OIDC in a general manner. > > We b

Re: [I] Cryptographic API Misuse Vulnerability: Do not use unverified JWT [airflow]

2023-10-23 Thread via GitHub
gxx777 commented on issue #35126: URL: https://github.com/apache/airflow/issues/35126#issuecomment-1776452730 > This is not the correct way to report security issues. See https://airflow.apache.org/docs/apache-airflow/2.1.1/security/index.html#reporting-vulnerabilities Thanks a lot!

[I] Airflow-2.6.3 Mysql: Deadlock found when trying to get lock; try restarting transaction [airflow]

2023-10-23 Thread via GitHub
liangriyu opened a new issue, #35144: URL: https://github.com/apache/airflow/issues/35144 ### Apache Airflow version Other Airflow 2 version (please specify below) ### What happened 2023-10-24 00:10:20,620 ERROR - Exception when executing SchedulerJob._run_scheduler_loop

Re: [I] Airflow-2.6.3 Mysql: Deadlock found when trying to get lock; try restarting transaction [airflow]

2023-10-23 Thread via GitHub
boring-cyborg[bot] commented on issue #35144: URL: https://github.com/apache/airflow/issues/35144#issuecomment-1776380207 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for ap

Re: [PR] Fix parsing KubernetesPodOperator multiline logs [airflow]

2023-10-23 Thread via GitHub
donPain commented on PR #34412: URL: https://github.com/apache/airflow/pull/34412#issuecomment-1776328318 When will this fix be available in helm stable? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [CSE-1089] Add back dag_run to docs [airflow]

2023-10-23 Thread via GitHub
Taragolis commented on PR #35142: URL: https://github.com/apache/airflow/pull/35142#issuecomment-1776262993 > Yeah. The list might be a good start. Ideally we should provide some copy of DagRun (as well as others) into task context. Something like pydantic models which are use in inte

Re: [PR] [CSE-1089] Add back dag_run to docs [airflow]

2023-10-23 Thread via GitHub
Taragolis commented on PR #35142: URL: https://github.com/apache/airflow/pull/35142#issuecomment-1776257494 @fritz-astronomer You should manually add DagRun toctree into the Public Interface of Airflow, some example: https://github.com/apache/airflow/pull/35032/files#diff-4c4bd448d0ce0ccad9

Re: [PR] fix(helm): set worker safeToEvict properly [airflow]

2023-10-23 Thread via GitHub
hakuno commented on PR #35130: URL: https://github.com/apache/airflow/pull/35130#issuecomment-1776255255 Thank you so, @hussein-awala I tested the Helm Chart itself with `--validate` on a Kubernetes cluster to verify the generated manifests. ``` $ helm template --dry-run --

Re: [I] BigQueryValueCheckOperator does not accept non-default project_id, which lead to jobs with impersonation_chain to fail. [airflow]

2023-10-23 Thread via GitHub
vchiapaikeo commented on issue #32870: URL: https://github.com/apache/airflow/issues/32870#issuecomment-1776238735 There's actually a bit of inconsistency here and `project_id` doesn't always necessarily mean where BQ compute occurs. For example, with BigQueryUpsertTableOperator, the [proje

Re: [PR] fix: Delete warning in get_uri [airflow]

2023-10-23 Thread via GitHub
highpon commented on PR #34446: URL: https://github.com/apache/airflow/pull/34446#issuecomment-1776211831 Hi, reviewers! If you have time, please review this PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[airflow] branch constraints-main updated: Updating constraints. Github run id:6618760910

2023-10-23 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 6a3095c649 Updating con

Re: [PR] change latest to stable to fix broken documentation links [airflow]

2023-10-23 Thread via GitHub
hussein-awala commented on PR #35143: URL: https://github.com/apache/airflow/pull/35143#issuecomment-1776171360 > Also, what to do about this link? It gives a 404 even with `stable`. http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/migration.html

Re: [PR] [CSE-1089] Add back dag_run to docs [airflow]

2023-10-23 Thread via GitHub
potiuk commented on PR #35142: URL: https://github.com/apache/airflow/pull/35142#issuecomment-1776139163 > Difficult to say which one should be accessed in context and which one shouldn't. Yeah. The list might be a good start. -- This is an automated message from the Apache Git Ser

Re: [PR] [CSE-1089] Add back dag_run to docs [airflow]

2023-10-23 Thread via GitHub
potiuk commented on PR #35142: URL: https://github.com/apache/airflow/pull/35142#issuecomment-1776138085 > i wish we could actually flip it and say everything is private unless explicitly declared public > > https://public.readthedocs.io/en/stable/ That was the idea (on module

Re: [PR] [CSE-1089] Add back dag_run to docs [airflow]

2023-10-23 Thread via GitHub
Taragolis commented on PR #35142: URL: https://github.com/apache/airflow/pull/35142#issuecomment-1776133631 > Do you think of some concrete methods that we should also mark as private in this case? Maybe that is a good opportunity to mark them as private indeed. Difficult to say which

Re: [PR] Add task parameter to set custom logger name [airflow]

2023-10-23 Thread via GitHub
potiuk commented on code in PR #34964: URL: https://github.com/apache/airflow/pull/34964#discussion_r1369328935 ## tests/providers/docker/hooks/test_docker.py: ## @@ -36,7 +36,7 @@ TEST_CONN = {"host": "some.docker.registry.com", "login": "some_user", "password": "some_p4$$w0r

Re: [PR] Add task parameter to set custom logger name [airflow]

2023-10-23 Thread via GitHub
potiuk commented on code in PR #34964: URL: https://github.com/apache/airflow/pull/34964#discussion_r1369327438 ## tests/providers/docker/hooks/test_docker.py: ## @@ -36,7 +36,7 @@ TEST_CONN = {"host": "some.docker.registry.com", "login": "some_user", "password": "some_p4$$w0r

Re: [PR] Add task parameter to set custom logger name [airflow]

2023-10-23 Thread via GitHub
potiuk commented on code in PR #34964: URL: https://github.com/apache/airflow/pull/34964#discussion_r1369327438 ## tests/providers/docker/hooks/test_docker.py: ## @@ -36,7 +36,7 @@ TEST_CONN = {"host": "some.docker.registry.com", "login": "some_user", "password": "some_p4$$w0r

Re: [PR] Add task parameter to set custom logger name [airflow]

2023-10-23 Thread via GitHub
potiuk commented on code in PR #34964: URL: https://github.com/apache/airflow/pull/34964#discussion_r1369325786 ## tests/providers/docker/hooks/test_docker.py: ## @@ -36,7 +36,7 @@ TEST_CONN = {"host": "some.docker.registry.com", "login": "some_user", "password": "some_p4$$w0r

Re: [PR] Add task parameter to set custom logger name [airflow]

2023-10-23 Thread via GitHub
potiuk commented on code in PR #34964: URL: https://github.com/apache/airflow/pull/34964#discussion_r1369325665 ## tests/providers/docker/hooks/test_docker.py: ## @@ -36,7 +36,7 @@ TEST_CONN = {"host": "some.docker.registry.com", "login": "some_user", "password": "some_p4$$w0r

Re: [PR] [CSE-1089] Add back dag_run to docs [airflow]

2023-10-23 Thread via GitHub
dstandish commented on PR #35142: URL: https://github.com/apache/airflow/pull/35142#issuecomment-1776122568 i wish we could actually flip it and say everything is private unless explicitly declared public https://public.readthedocs.io/en/stable/ -- This is an automated message fro

Re: [PR] Providers requirements for every python version [airflow]

2023-10-23 Thread via GitHub
pierrejeambrun commented on code in PR #35086: URL: https://github.com/apache/airflow/pull/35086#discussion_r1369322850 ## dev/breeze/src/airflow_breeze/commands/sbom_commands.py: ## @@ -236,23 +241,83 @@ def update_sbom_information( ) +@sbom.command(name="build

Re: [PR] Add task parameter to set custom logger name [airflow]

2023-10-23 Thread via GitHub
Taragolis commented on code in PR #34964: URL: https://github.com/apache/airflow/pull/34964#discussion_r1369320830 ## tests/providers/docker/hooks/test_docker.py: ## @@ -36,7 +36,7 @@ TEST_CONN = {"host": "some.docker.registry.com", "login": "some_user", "password": "some_p4$$

Re: [PR] [CSE-1089] Add back dag_run to docs [airflow]

2023-10-23 Thread via GitHub
fritz-astronomer commented on code in PR #35142: URL: https://github.com/apache/airflow/pull/35142#discussion_r1369321216 ## docs/conf.py: ## @@ -250,6 +250,7 @@ def _get_rst_filepath_from_path(filepath: pathlib.Path): "baseoperator.py", "connection.py",

Re: [PR] Providers requirements for every python version [airflow]

2023-10-23 Thread via GitHub
potiuk commented on PR #35086: URL: https://github.com/apache/airflow/pull/35086#issuecomment-1776114632 Images need to be regenerated after my changes :( -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [PR] Add task parameter to set custom logger name [airflow]

2023-10-23 Thread via GitHub
Taragolis commented on code in PR #34964: URL: https://github.com/apache/airflow/pull/34964#discussion_r1369320830 ## tests/providers/docker/hooks/test_docker.py: ## @@ -36,7 +36,7 @@ TEST_CONN = {"host": "some.docker.registry.com", "login": "some_user", "password": "some_p4$$

Re: [PR] [CSE-1089] Add back dag_run to docs [airflow]

2023-10-23 Thread via GitHub
potiuk commented on code in PR #35142: URL: https://github.com/apache/airflow/pull/35142#discussion_r1369320578 ## docs/conf.py: ## @@ -250,6 +250,7 @@ def _get_rst_filepath_from_path(filepath: pathlib.Path): "baseoperator.py", "connection.py", "dag.py

Re: [PR] [CSE-1089] Add back dag_run to docs [airflow]

2023-10-23 Thread via GitHub
potiuk commented on PR #35142: URL: https://github.com/apache/airflow/pull/35142#issuecomment-1776110828 > Are we want to make all methods of DagRun and DagRunNote as part of Airflow Public interface? Should we explicitly set `:meta private:` for all internal methods? The last time w

Re: [PR] Add back dag_run to docs [airflow]

2023-10-23 Thread via GitHub
Taragolis commented on PR #35142: URL: https://github.com/apache/airflow/pull/35142#issuecomment-1776093882 Are we want to make all methods of DagRun and DagRunNote as part of Airflow Public interface? Should we explicitly set `:meta private:` for all internal methods? -- This is an au

Re: [PR] Make core compatible with `pendulum` 3 [airflow]

2023-10-23 Thread via GitHub
Taragolis commented on code in PR #34744: URL: https://github.com/apache/airflow/pull/34744#discussion_r1369303186 ## airflow/utils/timezone.py: ## @@ -18,14 +18,20 @@ from __future__ import annotations import datetime as dt +from functools import lru_cache from typing impo

Re: [PR] Refactor commands to unify daemon context handling [airflow]

2023-10-23 Thread via GitHub
potiuk commented on PR #34945: URL: https://github.com/apache/airflow/pull/34945#issuecomment-1776076860 @Taragolis ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] change latest to stable to fix broken documentation links [airflow]

2023-10-23 Thread via GitHub
potiuk commented on PR #35143: URL: https://github.com/apache/airflow/pull/35143#issuecomment-1776048502 > Also, what to do about this link? It gives a 404 even with `stable`. http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/migration.html I

Re: [PR] change latest to stable to fix broken documentation links [airflow]

2023-10-23 Thread via GitHub
theaadya commented on PR #35143: URL: https://github.com/apache/airflow/pull/35143#issuecomment-1776011437 Also, what to do about this link? It gives a 404 even with `stable`. http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/migration.html -

[PR] change latest to stable to fix broken documentation links [airflow]

2023-10-23 Thread via GitHub
theaadya opened a new pull request, #35143: URL: https://github.com/apache/airflow/pull/35143 I found that in the dev version ([v2.8.0.dev0](https://pypi.python.org/pypi/apache-airflow/2.8.0.dev0)) of Airflow, the documentation link in the UI is broken. Specifically, this link http://apach

[PR] Add back dag_run to docs [airflow]

2023-10-23 Thread via GitHub
fritz-astronomer opened a new pull request, #35142: URL: https://github.com/apache/airflow/pull/35142 --- Adds in `dag_runs` so it can be linked to and viewed - such as from the `templates.rst` page. Should others be here? Like `pool`? 🤷 -- This is an automated message from

Re: [PR] Add `--database-isolation` as option to run breeze with [airflow]

2023-10-23 Thread via GitHub
potiuk merged PR #35140: URL: https://github.com/apache/airflow/pull/35140 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

Re: [PR] AWS ECS Executor [airflow]

2023-10-23 Thread via GitHub
o-nikolas commented on code in PR #34381: URL: https://github.com/apache/airflow/pull/34381#discussion_r1369235641 ## airflow/providers/amazon/aws/executors/ecs/README.md: ## @@ -0,0 +1,196 @@ + + +# AWS ECS Executor + +This is an Airflow executor powered by Amazon Elastic Conta

Re: [PR] Add `--standalone-dag-processor` option to `start-airflow` command [airflow]

2023-10-23 Thread via GitHub
potiuk merged PR #35139: URL: https://github.com/apache/airflow/pull/35139 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

Re: [I] Cloud not read served logs [airflow]

2023-10-23 Thread via GitHub
potiuk closed issue #35125: Cloud not read served logs URL: https://github.com/apache/airflow/issues/35125 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mai

Re: [PR] Refactor commands to unify daemon context handling [airflow]

2023-10-23 Thread via GitHub
Bisk1 commented on code in PR #34945: URL: https://github.com/apache/airflow/pull/34945#discussion_r1369154410 ## airflow/cli/commands/internal_api_command.py: ## @@ -148,60 +139,54 @@ def kill_proc(signum, _): gunicorn_master_proc.kill() sys.exit(0

Re: [PR] Refactor commands to unify daemon context handling [airflow]

2023-10-23 Thread via GitHub
Bisk1 commented on code in PR #34945: URL: https://github.com/apache/airflow/pull/34945#discussion_r1369154410 ## airflow/cli/commands/internal_api_command.py: ## @@ -148,60 +139,54 @@ def kill_proc(signum, _): gunicorn_master_proc.kill() sys.exit(0

Re: [PR] Refactor commands to unify daemon context handling [airflow]

2023-10-23 Thread via GitHub
Bisk1 commented on code in PR #34945: URL: https://github.com/apache/airflow/pull/34945#discussion_r1369152054 ## airflow/cli/commands/celery_command.py: ## @@ -68,28 +67,7 @@ def flower(args): if args.flower_conf: options.append(f"--conf={args.flower_conf}") -

Re: [PR] Refactor commands to unify daemon context handling [airflow]

2023-10-23 Thread via GitHub
Bisk1 commented on code in PR #34945: URL: https://github.com/apache/airflow/pull/34945#discussion_r1369152541 ## airflow/cli/commands/celery_command.py: ## @@ -214,33 +187,31 @@ def worker(args): # executed. maybe_patch_concurrency(["-P", pool]) -if args

Re: [PR] AIP-58: Add Airflow ObjectStore (AFS) [airflow]

2023-10-23 Thread via GitHub
Taragolis commented on code in PR #34729: URL: https://github.com/apache/airflow/pull/34729#discussion_r1369145057 ## airflow/providers/amazon/aws/fs/s3.py: ## @@ -0,0 +1,123 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreemen

Re: [I] Cloud not read served logs [airflow]

2023-10-23 Thread via GitHub
jens-scheffler-bosch commented on issue #35125: URL: https://github.com/apache/airflow/issues/35125#issuecomment-1775836453 This sounds more like a deployment issue in core, not related to helm-chart. I assume to judge this problem report more details are needed. If a scheduled task r

Re: [PR] Refactor commands to unify daemon context handling [airflow]

2023-10-23 Thread via GitHub
daniel-dylag-openx commented on code in PR #34945: URL: https://github.com/apache/airflow/pull/34945#discussion_r1369137222 ## airflow/cli/commands/celery_command.py: ## @@ -214,33 +187,31 @@ def worker(args): # executed. maybe_patch_concurrency(["-P", pool])

Re: [I] Add HttpToS3Operator [airflow]

2023-10-23 Thread via GitHub
jens-scheffler-bosch commented on issue #35137: URL: https://github.com/apache/airflow/issues/35137#issuecomment-1775825745 The text reads as if this is a PR already. If you have a direct code contribution no explicit issue ticket is needed, just open a PR (I also needed to learn this :-D).

Re: [I] Track SparkSubmitHook Yarn Cluster application with Yarn CLI [airflow]

2023-10-23 Thread via GitHub
tokoko commented on issue #24171: URL: https://github.com/apache/airflow/issues/24171#issuecomment-1775814934 @vchiapaikeo Hey, I went through several iterations of this since this issue. The version I'm trying to adopt right now is deferrable and uses yarn rest api rather than yarn cli, be

[airflow] branch main updated: Fix Scheduler crash looping when dagrun creation fails (#35135)

2023-10-23 Thread ephraimanierobi
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 98618f0c4f Fix Scheduler crash looping whe

Re: [PR] Fix Scheduler crash looping when dagrun creation fails [airflow]

2023-10-23 Thread via GitHub
ephraimbuddy merged PR #35135: URL: https://github.com/apache/airflow/pull/35135 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@air

[PR] Add database isolation to start airflow [airflow]

2023-10-23 Thread via GitHub
potiuk opened a new pull request, #35140: URL: https://github.com/apache/airflow/pull/35140 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)*

Re: [PR] Add `--standalone-dag-processor` option to `start-airflow` command [airflow]

2023-10-23 Thread via GitHub
potiuk commented on code in PR #35139: URL: https://github.com/apache/airflow/pull/35139#discussion_r1369077974 ## dev/breeze/src/airflow_breeze/params/shell_params.py: ## @@ -127,6 +127,8 @@ class ShellParams: only_min_version_update: bool = False regenerate_missing_d

Re: [PR] Add `--standalone-dag-processor` option to `start-airflow` command [airflow]

2023-10-23 Thread via GitHub
potiuk commented on code in PR #35139: URL: https://github.com/apache/airflow/pull/35139#discussion_r1369074131 ## scripts/ci/docker-compose/base.yml: ## @@ -81,6 +81,7 @@ services: - SKIP_ENVIRONMENT_INITIALIZATION=${SKIP_ENVIRONMENT_INITIALIZATION} - SKIP_PROVIDE

[PR] Add `--standalone-dag-processor` option to `start-airflow` command [airflow]

2023-10-23 Thread via GitHub
potiuk opened a new pull request, #35139: URL: https://github.com/apache/airflow/pull/35139 As part of AIP-44 implementation we are going to test more secure combination of airflow so adding standalone dag-processor to breeze will make it easier. --- **

Re: [I] BigQueryValueCheckOperator does not accept non-default project_id, which lead to jobs with impersonation_chain to fail. [airflow]

2023-10-23 Thread via GitHub
GaryLiuTelus commented on issue #32870: URL: https://github.com/apache/airflow/issues/32870#issuecomment-1775731577 Thanks for the sharing. And it works! Appreciate! But still, I think this is more like a workaround. But I think if project_id can be provided as a parameter would be be

Re: [PR] Move authentication related methods to FAB auth manager [airflow]

2023-10-23 Thread via GitHub
vincbeck commented on code in PR #35038: URL: https://github.com/apache/airflow/pull/35038#discussion_r1369048804 ## airflow/auth/managers/fab/security_manager/override.py: ## @@ -103,6 +112,16 @@ class FabAirflowSecurityManagerOverride(AirflowSecurityManagerV2): permissio

Re: [PR] Move authentication related methods to FAB auth manager [airflow]

2023-10-23 Thread via GitHub
vincbeck commented on PR #35038: URL: https://github.com/apache/airflow/pull/35038#issuecomment-1775716918 > Rebase needed. Done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] Prevent assignment of non JSON serializable values to DagRun.conf dict [airflow]

2023-10-23 Thread via GitHub
jens-scheffler-bosch commented on code in PR #35096: URL: https://github.com/apache/airflow/pull/35096#discussion_r1369048257 ## airflow/models/dagrun.py: ## @@ -126,7 +150,7 @@ class DagRun(Base, LoggingMixin): creating_job_id = Column(Integer) external_trigger = Colu

Re: [PR] Migrate provider-specific executor docs to providers [airflow]

2023-10-23 Thread via GitHub
potiuk commented on PR #34809: URL: https://github.com/apache/airflow/pull/34809#issuecomment-1775699568 Sphinxes often speak in riddles, this is something I often repeat (too often I am afraid) -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] Migrate provider-specific executor docs to providers [airflow]

2023-10-23 Thread via GitHub
RNHTTR commented on PR #34809: URL: https://github.com/apache/airflow/pull/34809#issuecomment-1775697475 Locally, I get the following ☹️ ``` The documentation has errors. Packages errors summary Package name

Re: [PR] Reset stats logger in daemon context to reinitialize in child process [airflow]

2023-10-23 Thread via GitHub
potiuk commented on PR #34769: URL: https://github.com/apache/airflow/pull/34769#issuecomment-1775678016 I think it's bette to redo this fix after #34945 - that one seems almost ready and I think fixing the problem you mentioned will be a bit different once it is merged. -- This is an au

Re: [PR] Fix service account impersonation with computeEngineSSHHook (google provider) [airflow]

2023-10-23 Thread via GitHub
ginolegigot commented on PR #35136: URL: https://github.com/apache/airflow/pull/35136#issuecomment-1775674632 Roger! I'll try to understand how to write some unit tests in my particular context using airflow tests libs -- This is an automated message from the Apache Git Service. To res

Re: [PR] Fix service account impersonation with computeEngineSSHHook (google provider) [airflow]

2023-10-23 Thread via GitHub
potiuk commented on PR #35136: URL: https://github.com/apache/airflow/pull/35136#issuecomment-1775670817 It will need unit test. We rarely accept changes without corresponding unit tests - mostly when they are difficult to write. This one seems quite reasonable. -- This is an automated m

Re: [I] Support for general OIDC providers or making it clear in document [airflow]

2023-10-23 Thread via GitHub
potiuk commented on issue #35131: URL: https://github.com/apache/airflow/issues/35131#issuecomment-1775666379 > This behaviour should be documented. Otherwise, there should be a way to configure other OIDC providers like other projects that support OIDC in a general manner. We base o

Re: [PR] Use `is_authorized_*` APIs in views [airflow]

2023-10-23 Thread via GitHub
potiuk commented on code in PR #35000: URL: https://github.com/apache/airflow/pull/35000#discussion_r1368989214 ## airflow/www/security_manager.py: ## @@ -290,23 +323,36 @@ def has_access(self, action_name: str, resource_name: str, user=None) -> bool: Example actions

Re: [PR] Migrate provider-specific executor docs to providers [airflow]

2023-10-23 Thread via GitHub
potiuk commented on PR #34809: URL: https://github.com/apache/airflow/pull/34809#issuecomment-1775621186 My wild guess is that you have label reference somewhere in the code left (of the form `_executor:celeryexecutor`). -- This is an automated message from the Apache Git Service. To res

Re: [PR] Move authentication related methods to FAB auth manager [airflow]

2023-10-23 Thread via GitHub
vincbeck commented on code in PR #35038: URL: https://github.com/apache/airflow/pull/35038#discussion_r1368982785 ## airflow/auth/managers/fab/security_manager/override.py: ## @@ -103,6 +112,16 @@ class FabAirflowSecurityManagerOverride(AirflowSecurityManagerV2): permissio

Re: [PR] Migrate provider-specific executor docs to providers [airflow]

2023-10-23 Thread via GitHub
potiuk commented on PR #34809: URL: https://github.com/apache/airflow/pull/34809#issuecomment-1775614880 Yep. Looks better. I think those are not spellcheck errors but some regular sphinx riddiles. Sphinx does not seem to like SOMETHING in the docs and not tell exactly where. The usual thin

Re: [PR] Use `is_authorized_*` APIs in views [airflow]

2023-10-23 Thread via GitHub
vincbeck commented on code in PR #35000: URL: https://github.com/apache/airflow/pull/35000#discussion_r1368975417 ## airflow/www/security_manager.py: ## @@ -290,23 +323,36 @@ def has_access(self, action_name: str, resource_name: str, user=None) -> bool: Example actio

Re: [PR] Use `is_authorized_*` APIs in views [airflow]

2023-10-23 Thread via GitHub
vincbeck commented on code in PR #35000: URL: https://github.com/apache/airflow/pull/35000#discussion_r1368975417 ## airflow/www/security_manager.py: ## @@ -290,23 +323,36 @@ def has_access(self, action_name: str, resource_name: str, user=None) -> bool: Example actio

Re: [PR] Use `is_authorized_*` APIs in views [airflow]

2023-10-23 Thread via GitHub
potiuk commented on code in PR #35000: URL: https://github.com/apache/airflow/pull/35000#discussion_r1368972060 ## airflow/www/security_manager.py: ## @@ -290,23 +323,36 @@ def has_access(self, action_name: str, resource_name: str, user=None) -> bool: Example actions

Re: [I] Add HttpToS3Operator [airflow]

2023-10-23 Thread via GitHub
boring-cyborg[bot] commented on issue #35137: URL: https://github.com/apache/airflow/issues/35137#issuecomment-1775598156 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for ap

[I] Add HttpToS3Operator [airflow]

2023-10-23 Thread via GitHub
simonprydden opened a new issue, #35137: URL: https://github.com/apache/airflow/issues/35137 ### Description This operator allows users to effortlessly transfer data from HTTP sources to Amazon S3, with minimal coding effort. Whether you need to ingest web-based content, receive data

[PR] Fix service account impersonation with computeEngineSSHHook (google provider) [airflow]

2023-10-23 Thread via GitHub
ginolegigot opened a new pull request, #35136: URL: https://github.com/apache/airflow/pull/35136 Hello ! This PR aims to solve the issue where we can't use service account impersonation when we want to connect with ssh between 2 Google Compute Engine instances. It is a new version o

Re: [PR] Refactor commands to unify daemon context handling [airflow]

2023-10-23 Thread via GitHub
potiuk commented on code in PR #34945: URL: https://github.com/apache/airflow/pull/34945#discussion_r1368962020 ## airflow/cli/commands/internal_api_command.py: ## @@ -148,60 +139,54 @@ def kill_proc(signum, _): gunicorn_master_proc.kill() sys.exit(

Re: [PR] Use `is_authorized_*` APIs in views [airflow]

2023-10-23 Thread via GitHub
vincbeck commented on code in PR #35000: URL: https://github.com/apache/airflow/pull/35000#discussion_r1368960096 ## airflow/www/security_manager.py: ## @@ -290,23 +323,36 @@ def has_access(self, action_name: str, resource_name: str, user=None) -> bool: Example actio

Re: [PR] Use `is_authorized_*` APIs in views [airflow]

2023-10-23 Thread via GitHub
vincbeck commented on code in PR #35000: URL: https://github.com/apache/airflow/pull/35000#discussion_r1368959528 ## airflow/www/security_manager.py: ## @@ -290,23 +323,36 @@ def has_access(self, action_name: str, resource_name: str, user=None) -> bool: Example actio

[PR] Fix Scheduler crash looping when dagrun creation fails [airflow]

2023-10-23 Thread via GitHub
ephraimbuddy opened a new pull request, #35135: URL: https://github.com/apache/airflow/pull/35135 Exception from dag.create_dagrun should not crash the scheduler. Instead, the scheduler should catch the exception and log it. This PR provides that behaviour -- This is an aut

Re: [PR] Refactor commands to unify daemon context handling [airflow]

2023-10-23 Thread via GitHub
potiuk commented on code in PR #34945: URL: https://github.com/apache/airflow/pull/34945#discussion_r1368938322 ## airflow/cli/commands/celery_command.py: ## @@ -214,33 +187,31 @@ def worker(args): # executed. maybe_patch_concurrency(["-P", pool]) -if arg

  1   2   3   >