Re: [PR] Switch instance type to m7g for ARM image building [airflow]

2024-02-01 Thread via GitHub
Taragolis commented on code in PR #37134: URL: https://github.com/apache/airflow/pull/37134#discussion_r1475679106 ## scripts/ci/images/ci_start_arm_instance_and_connect_to_docker.sh: ## @@ -20,7 +20,7 @@ SCRIPTS_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd)" WORKING_DI

Re: [PR] Add Doctrine in users [airflow]

2024-02-01 Thread via GitHub
anteverse commented on PR #37135: URL: https://github.com/apache/airflow/pull/37135#issuecomment-1923225869 Thanks a lot! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Replace usage of `datetime.utcnow` and `datetime.utcfromtimestamp` in providers [airflow]

2024-02-01 Thread via GitHub
Taragolis commented on code in PR #37138: URL: https://github.com/apache/airflow/pull/37138#discussion_r1475665219 ## tests/providers/google/cloud/sensors/test_gcs.py: ## @@ -72,7 +72,7 @@ def context(): """ Creates an empty context. """ -context = {"data_inte

(airflow) branch main updated: Add clarification about DAG author capabilities to security model (#37141)

2024-02-01 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new e1b032d05b Add clarification about DAG author capab

Re: [PR] Add clarification about DAG author capabilities to security model [airflow]

2024-02-01 Thread via GitHub
potiuk merged PR #37141: URL: https://github.com/apache/airflow/pull/37141 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

Re: [PR] Replace usage of `datetime.utcnow` and `datetime.utcfromtimestamp` in providers [airflow]

2024-02-01 Thread via GitHub
Taragolis commented on code in PR #37138: URL: https://github.com/apache/airflow/pull/37138#discussion_r1475662271 ## tests/providers/amazon/aws/hooks/test_sagemaker.py: ## @@ -528,7 +528,9 @@ def test_secondary_training_status_changed_false(self): def test_secondary_traini

Re: [PR] DRAFT Add conditional logic for dataset triggering [airflow]

2024-02-01 Thread via GitHub
dstandish commented on PR #37016: URL: https://github.com/apache/airflow/pull/37016#issuecomment-1922964226 @sunank200 1. fix test on main PR 2. look at the tests on the PR and determine which should be added or removed 3. performance - why 10 minutes in main? -

Re: [PR] Replace usage of `datetime.utcnow` and `datetime.utcfromtimestamp` in providers [airflow]

2024-02-01 Thread via GitHub
uranusjr commented on code in PR #37138: URL: https://github.com/apache/airflow/pull/37138#discussion_r1475586114 ## tests/providers/google/cloud/sensors/test_gcs.py: ## @@ -72,7 +72,7 @@ def context(): """ Creates an empty context. """ -context = {"data_inter

Re: [PR] Replace usage of `datetime.utcnow` and `datetime.utcfromtimestamp` in providers [airflow]

2024-02-01 Thread via GitHub
uranusjr commented on code in PR #37138: URL: https://github.com/apache/airflow/pull/37138#discussion_r1475585134 ## tests/providers/amazon/aws/hooks/test_sagemaker.py: ## @@ -528,7 +528,9 @@ def test_secondary_training_status_changed_false(self): def test_secondary_trainin

Re: [PR] Fix KPO docs argument precedence [airflow]

2024-02-01 Thread via GitHub
dirrao commented on code in PR #37047: URL: https://github.com/apache/airflow/pull/37047#discussion_r1475580760 ## docs/apache-airflow-providers-cncf-kubernetes/operators.rst: ## @@ -77,8 +77,8 @@ You can print out the Kubernetes manifest for the pod that would be created at r

Re: [PR] Implement AIP-60 Dataset URI formats [airflow]

2024-02-01 Thread via GitHub
uranusjr commented on PR #37005: URL: https://github.com/apache/airflow/pull/37005#issuecomment-1922824911 Finally got this to a point Iā€™m somewhat satisfied with. The logic should be mostly there. Not exactly sure about docs yet, please provide some ideas on where to add things. -- This

[PR] Add clarification about DAG author capabilities to security model [airflow]

2024-02-01 Thread via GitHub
amoghrajesh opened a new pull request, #37141: URL: https://github.com/apache/airflow/pull/37141 Add clarification about DAG author capabilities to security model --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.c

Re: [PR] Add deferrable param in SFTPSensor [airflow]

2024-02-01 Thread via GitHub
Lee-W commented on code in PR #37117: URL: https://github.com/apache/airflow/pull/37117#discussion_r1475487123 ## airflow/providers/sftp/hooks/sftp.py: ## @@ -400,3 +406,177 @@ def get_files_by_pattern(self, path, fnmatch_pattern) -> list[str]: matched_files.ap

Re: [PR] The task is stuck in a queued state forever in case of pod launch errors [airflow]

2024-02-01 Thread via GitHub
dirrao commented on PR #36882: URL: https://github.com/apache/airflow/pull/36882#issuecomment-1922753144 > I had similar problems and thought about something like that Ok. I would suggest to use dedicated pool slots per namespace. pool slots should depicts the namespace resources. So,

[PR] Remove SCHEDULED_DEPS which is no longer used anywhere since 2.0.0 [airflow]

2024-02-01 Thread via GitHub
XD-DENG opened a new pull request, #37140: URL: https://github.com/apache/airflow/pull/37140 `SCHEDULED_DEPS` is no longer used anywhere in the codebase, since `2.0.0` (the last version where it was used is `1.10.15`) --- **^ Add meaningful description a

Re: [I] `PythonVirtualenvOperator` crashes if any `python_callable` function is defined in the same source as DAG [airflow]

2024-02-01 Thread via GitHub
rawwar commented on issue #35529: URL: https://github.com/apache/airflow/issues/35529#issuecomment-1922587725 in `_execute_python_callable_in_subprocess` --> when calling `self._write_args(input_path)` it seems some object in `op_kwargs` has a reference to modified dag module name.. which s

Re: [PR] WIP KPO log cleanup [airflow]

2024-02-01 Thread via GitHub
github-actions[bot] commented on PR #36227: URL: https://github.com/apache/airflow/pull/36227#issuecomment-1922529710 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for you

Re: [I] Airflow Scheduler with Kubernetes Executor has errors in logs and stuck slots with no running tasks [airflow]

2024-02-01 Thread via GitHub
github-actions[bot] commented on issue #36478: URL: https://github.com/apache/airflow/issues/36478#issuecomment-1922529692 This issue has been closed because it has not received response from the issue author. -- This is an automated message from the Apache Git Service. To respond to the

Re: [I] Airflow Scheduler with Kubernetes Executor has errors in logs and stuck slots with no running tasks [airflow]

2024-02-01 Thread via GitHub
github-actions[bot] closed issue #36478: Airflow Scheduler with Kubernetes Executor has errors in logs and stuck slots with no running tasks URL: https://github.com/apache/airflow/issues/36478 -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [PR] Fixed the hardcoded default namespace value for GCP Data Fusion links. [airflow]

2024-02-01 Thread via GitHub
github-actions[bot] commented on PR #35379: URL: https://github.com/apache/airflow/pull/35379#issuecomment-1922529782 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for you

Re: [PR] Support configuring multiple namespace for multiNamespaceMode [airflow]

2024-02-01 Thread via GitHub
github-actions[bot] commented on PR #35639: URL: https://github.com/apache/airflow/pull/35639#issuecomment-1922529747 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for you

Re: [PR] Merge all ECS executor configs following recursive python dict update [airflow]

2024-02-01 Thread via GitHub
o-nikolas commented on PR #37137: URL: https://github.com/apache/airflow/pull/37137#issuecomment-1922506188 @mshober Can you have a look at this PR and see if it satisfies your use case? -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [I] Airflow URL returning 404 when using Prefix [airflow]

2024-02-01 Thread via GitHub
potiuk commented on issue #28362: URL: https://github.com/apache/airflow/issues/28362#issuecomment-1922506078 It's not needed, I thinl - this is an issue for a 2 years old chart which it is strange to refer to. In the latest chart you have this: > path: {{ if .Values.config.webserver.

Re: [PR] Merge all ECS executor configs following recursive python dict update [airflow]

2024-02-01 Thread via GitHub
o-nikolas commented on code in PR #37137: URL: https://github.com/apache/airflow/pull/37137#discussion_r1475312683 ## airflow/providers/amazon/aws/executors/ecs/ecs_executor.py: ## @@ -406,17 +407,17 @@ def _run_task_kwargs( One last chance to modify Boto3's "run_task

Re: [PR] Merge all ECS executor configs following recursive python dict update [airflow]

2024-02-01 Thread via GitHub
ferruzzi commented on code in PR #37137: URL: https://github.com/apache/airflow/pull/37137#discussion_r1475314154 ## airflow/providers/amazon/aws/executors/ecs/ecs_executor.py: ## @@ -406,17 +407,17 @@ def _run_task_kwargs( One last chance to modify Boto3's "run_task"

Re: [PR] Merge all ECS executor configs following recursive python dict update [airflow]

2024-02-01 Thread via GitHub
o-nikolas commented on code in PR #37137: URL: https://github.com/apache/airflow/pull/37137#discussion_r1475312683 ## airflow/providers/amazon/aws/executors/ecs/ecs_executor.py: ## @@ -406,17 +407,17 @@ def _run_task_kwargs( One last chance to modify Boto3's "run_task

Re: [PR] Switch instance type to m7g for ARM image building [airflow]

2024-02-01 Thread via GitHub
potiuk commented on code in PR #37134: URL: https://github.com/apache/airflow/pull/37134#discussion_r1475306158 ## scripts/ci/images/ci_start_arm_instance_and_connect_to_docker.sh: ## @@ -20,7 +20,7 @@ SCRIPTS_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd)" WORKING_DIR="

Re: [PR] Switch instance type to m7g for ARM image building [airflow]

2024-02-01 Thread via GitHub
potiuk commented on code in PR #37134: URL: https://github.com/apache/airflow/pull/37134#discussion_r1475306158 ## scripts/ci/images/ci_start_arm_instance_and_connect_to_docker.sh: ## @@ -20,7 +20,7 @@ SCRIPTS_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd)" WORKING_DIR="

Re: [PR] Switch instance type to m7g for ARM image building [airflow]

2024-02-01 Thread via GitHub
potiuk commented on code in PR #37134: URL: https://github.com/apache/airflow/pull/37134#discussion_r1475300324 ## scripts/ci/images/ci_start_arm_instance_and_connect_to_docker.sh: ## @@ -20,7 +20,7 @@ SCRIPTS_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd)" WORKING_DIR="

Re: [PR] Merge all ECS executor configs following recursive python dict update [airflow]

2024-02-01 Thread via GitHub
ferruzzi commented on code in PR #37137: URL: https://github.com/apache/airflow/pull/37137#discussion_r1475299681 ## airflow/providers/amazon/aws/executors/ecs/ecs_executor.py: ## @@ -406,17 +407,17 @@ def _run_task_kwargs( One last chance to modify Boto3's "run_task"

(airflow) branch main updated: Add Doctrine in users (#37135)

2024-02-01 Thread ferruzzi
This is an automated email from the ASF dual-hosted git repository. ferruzzi pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new ecb7e3a56e Add Doctrine in users (#37135) ecb7e3a

Re: [PR] Add Doctrine in users [airflow]

2024-02-01 Thread via GitHub
boring-cyborg[bot] commented on PR #37135: URL: https://github.com/apache/airflow/pull/37135#issuecomment-1922434319 Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional contributions.

Re: [PR] Add Doctrine in users [airflow]

2024-02-01 Thread via GitHub
ferruzzi merged PR #37135: URL: https://github.com/apache/airflow/pull/37135 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

Re: [PR] Celery provider: Add task_acks_late setting [airflow]

2024-02-01 Thread via GitHub
ninsbl commented on PR #37066: URL: https://github.com/apache/airflow/pull/37066#issuecomment-1922420520 Hmm, it seems the spell checking does ot like the new key name: (` Incorrect Spelling: 'acks' Line with Error: 'task_acks_late' `)... -- This is an automated message fro

Re: [PR] Celery provider: Add task_acks_late setting [airflow]

2024-02-01 Thread via GitHub
ninsbl commented on PR #37066: URL: https://github.com/apache/airflow/pull/37066#issuecomment-1922297044 Anything I can do to move this forward? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] Merge all ECS executor configs following recursive python dict update [airflow]

2024-02-01 Thread via GitHub
o-nikolas commented on code in PR #37137: URL: https://github.com/apache/airflow/pull/37137#discussion_r1475149018 ## airflow/providers/amazon/aws/executors/ecs/utils.py: ## @@ -264,3 +264,16 @@ def camelize_dict_keys(nested_dict) -> dict: else: result[new_

Re: [I] GCSSynchronizeBucketsOperator fails on 30-second timeout on large files [airflow]

2024-02-01 Thread via GitHub
kevgeo commented on issue #27488: URL: https://github.com/apache/airflow/issues/27488#issuecomment-1922233990 Hi @jmelot, are you still working on this issue? I was able to reproduce to it and can provide a fix. -- This is an automated message from the Apache Git Service. To respond to th

(airflow) branch main updated: fix: using endpoint from connection if not specified (#37076)

2024-02-01 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new cea58c fix: using endpoint from connection i

Re: [PR] fix: using endpoint from connection if not specified [airflow]

2024-02-01 Thread via GitHub
Taragolis merged PR #37076: URL: https://github.com/apache/airflow/pull/37076 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflo

Re: [I] DatabricksSQLOperator struggles with parsing the data [airflow]

2024-02-01 Thread via GitHub
odykstra commented on issue #36839: URL: https://github.com/apache/airflow/issues/36839#issuecomment-1922153837 Hi there. I'm running into this same issue. I'm running a merge statement that only returns one row with four columns. Appreciate your attention to this. Old version

Re: [PR] Replace usage of `datetime.utcnow` and `datetime.utcfromtimestamp` in providers [airflow]

2024-02-01 Thread via GitHub
Taragolis commented on code in PR #37138: URL: https://github.com/apache/airflow/pull/37138#discussion_r1475075880 ## airflow/providers/google/common/hooks/base_google.py: ## @@ -670,7 +670,7 @@ async def acquire_access_token(self, timeout: int = 10) -> None: self.ac

[PR] Replace usage of `datetime.utcnow` and `datetime.utcfromtimestamp` in providers [airflow]

2024-02-01 Thread via GitHub
Taragolis opened a new pull request, #37138: URL: https://github.com/apache/airflow/pull/37138 Related: https://github.com/apache/airflow/issues/32344 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/ai

Re: [I] ECS Executor - Overriding Additional ECS Task Properties [airflow]

2024-02-01 Thread via GitHub
o-nikolas commented on issue #35490: URL: https://github.com/apache/airflow/issues/35490#issuecomment-1922096583 I ended up going with a very opinionated approach which very closely approximates standard python update. What @shubham22 proposed [here](https://github.com/apache/airflow/issues

Re: [PR] Merge all ECS executor configs following recursive python dict update [airflow]

2024-02-01 Thread via GitHub
Taragolis commented on code in PR #37137: URL: https://github.com/apache/airflow/pull/37137#discussion_r1475050095 ## airflow/providers/amazon/aws/executors/ecs/utils.py: ## @@ -264,3 +264,16 @@ def camelize_dict_keys(nested_dict) -> dict: else: result[new_

Re: [PR] Merge all ECS executor configs following recursive python dict update [airflow]

2024-02-01 Thread via GitHub
o-nikolas commented on PR #37137: URL: https://github.com/apache/airflow/pull/37137#issuecomment-1922088649 CC @syedahsn (since I can't seem to add you via the reviewers list) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

Re: [PR] Merge all ECS executor configs following recursive python dict update [airflow]

2024-02-01 Thread via GitHub
o-nikolas commented on code in PR #37137: URL: https://github.com/apache/airflow/pull/37137#discussion_r1475045195 ## tests/providers/amazon/aws/executors/ecs/test_ecs_executor.py: ## @@ -1147,3 +1148,220 @@ def test_providing_no_capacity_provider_no_lunch_type_no_cluster_defau

Re: [PR] fix broken link [airflow]

2024-02-01 Thread via GitHub
Taragolis commented on code in PR #37121: URL: https://github.com/apache/airflow/pull/37121#discussion_r1475039241 ## docs/apache-airflow-providers-amazon/connections/athena.rst: ## @@ -29,7 +29,7 @@ The Athena connection type enables DB API 2.0 integrations with Athena. Authe

[PR] Merge all ECS executor configs following recursive python dict update [airflow]

2024-02-01 Thread via GitHub
o-nikolas opened a new pull request, #37137: URL: https://github.com/apache/airflow/pull/37137 This PR changes how `exec_config` works for the ECS executor. Previously the `exec_config` was only applied to item zero of the `containerOverrides` list (the container that receives the Airflow t

(airflow) branch main updated: Updates to ECS Docs (#37125)

2024-02-01 Thread onikolas
This is an automated email from the ASF dual-hosted git repository. onikolas pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 55cc7bda56 Updates to ECS Docs (#37125) 55cc7bda5

Re: [PR] Updates to ECS Executor Docs [airflow]

2024-02-01 Thread via GitHub
o-nikolas merged PR #37125: URL: https://github.com/apache/airflow/pull/37125 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflo

[I] external_python_task decorator doesn't work if airflow isn't in external python env [airflow]

2024-02-01 Thread via GitHub
JCoder01 opened a new issue, #37136: URL: https://github.com/apache/airflow/issues/37136 ### Apache Airflow version Other Airflow 2 version (please specify below) ### If "Other Airflow 2 version" selected, which one? 2.7.3 ### What happened? ``` 2024-02-0

Re: [I] DatabricksSQLOperator struggles with parsing the data [airflow]

2024-02-01 Thread via GitHub
odykstra commented on issue #36839: URL: https://github.com/apache/airflow/issues/36839#issuecomment-1921951126 Hi there. I'm running into exactly the same issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [I] Datasets: Allow time based force dependency when dataset is not ready. [airflow]

2024-02-01 Thread via GitHub
cmarteepants commented on issue #30974: URL: https://github.com/apache/airflow/issues/30974#issuecomment-1921943496 > Hi, just to provide a bit more info on one of the use cases that inspired this issue: > > * We have two datasets, say A and B, both of which are provided by third par

Re: [PR] Add deferrable param in SFTPSensor [airflow]

2024-02-01 Thread via GitHub
pankajkoti commented on code in PR #37117: URL: https://github.com/apache/airflow/pull/37117#discussion_r1474844227 ## airflow/providers/sftp/sensors/sftp.py: ## @@ -119,3 +124,50 @@ def poke(self, context: Context) -> PokeReturnValue | bool: xcom_value={"files_

Re: [I] Log Links Aren't Clickable in Grid Details Logs Tab [airflow]

2024-02-01 Thread via GitHub
bbovenzi commented on issue #35280: URL: https://github.com/apache/airflow/issues/35280#issuecomment-1921812768 Closed by https://github.com/apache/airflow/pull/36816 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [I] Log Links Aren't Clickable in Grid Details Logs Tab [airflow]

2024-02-01 Thread via GitHub
bbovenzi closed issue #35280: Log Links Aren't Clickable in Grid Details Logs Tab URL: https://github.com/apache/airflow/issues/35280 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [I] Log links in grid view [airflow]

2024-02-01 Thread via GitHub
bbovenzi commented on issue #28163: URL: https://github.com/apache/airflow/issues/28163#issuecomment-1921812028 Closed by https://github.com/apache/airflow/pull/36816 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [I] Log links in grid view [airflow]

2024-02-01 Thread via GitHub
bbovenzi closed issue #28163: Log links in grid view URL: https://github.com/apache/airflow/issues/28163 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Add deferrable param in SFTPSensor [airflow]

2024-02-01 Thread via GitHub
utkarsharma2 commented on PR #37117: URL: https://github.com/apache/airflow/pull/37117#issuecomment-1921785285 > looks like CI is failing Please take a look, they are fixed now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Remove pid arg from celery option to fix duplicate pid issue, Move celery command to provider package [airflow]

2024-02-01 Thread via GitHub
pavansharma36 commented on PR #36794: URL: https://github.com/apache/airflow/pull/36794#issuecomment-1921699122 @potiuk I tried copying daemon func but it goes way down where so many utils are not present. Its better if someone with better codebase understanding do this. Let me know

Re: [I] Kubernetes executor pod reconcilation is slow due to recurssive deep copy [airflow]

2024-02-01 Thread via GitHub
dirrao commented on issue #37131: URL: https://github.com/apache/airflow/issues/37131#issuecomment-1921697564 `index 9f01231b0c..3ab52d6d71 100644 --- a/airflow/providers/cncf/kubernetes/pod_generator.py +++ b/airflow/providers/cncf/kubernetes/pod_generator.py @@ -258,7 +258,7 @@ cla

Re: [PR] Switch instance type to m7g for ARM image building [airflow]

2024-02-01 Thread via GitHub
Taragolis commented on code in PR #37134: URL: https://github.com/apache/airflow/pull/37134#discussion_r1474691877 ## scripts/ci/images/ci_start_arm_instance_and_connect_to_docker.sh: ## @@ -20,7 +20,7 @@ SCRIPTS_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd)" WORKING_DI

Re: [PR] Switch instance type to m7g for ARM image building [airflow]

2024-02-01 Thread via GitHub
Taragolis commented on code in PR #37134: URL: https://github.com/apache/airflow/pull/37134#discussion_r1474687232 ## scripts/ci/images/ci_start_arm_instance_and_connect_to_docker.sh: ## @@ -20,7 +20,7 @@ SCRIPTS_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd)" WORKING_DI

Re: [PR] Add watchdog for immediately processing changes in the DAGs folder [airflow]

2024-02-01 Thread via GitHub
cmarteepants commented on PR #34487: URL: https://github.com/apache/airflow/pull/34487#issuecomment-1921605523 @BasPH Just curious where we are - do you think you'll have time to get this in for 2.9? -- This is an automated message from the Apache Git Service. To respond to the message, p

Re: [PR] Switch instance type to m7g for ARM image building [airflow]

2024-02-01 Thread via GitHub
potiuk commented on code in PR #37134: URL: https://github.com/apache/airflow/pull/37134#discussion_r1474625756 ## scripts/ci/images/ci_start_arm_instance_and_connect_to_docker.sh: ## @@ -20,7 +20,7 @@ SCRIPTS_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd)" WORKING_DIR="

Re: [PR] Switch instance type to m7g for ARM image building [airflow]

2024-02-01 Thread via GitHub
potiuk commented on code in PR #37134: URL: https://github.com/apache/airflow/pull/37134#discussion_r1474625756 ## scripts/ci/images/ci_start_arm_instance_and_connect_to_docker.sh: ## @@ -20,7 +20,7 @@ SCRIPTS_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd)" WORKING_DIR="

Re: [PR] Switch instance type to m7g for ARM image building [airflow]

2024-02-01 Thread via GitHub
Taragolis commented on code in PR #37134: URL: https://github.com/apache/airflow/pull/37134#discussion_r1474616257 ## scripts/ci/images/ci_start_arm_instance_and_connect_to_docker.sh: ## @@ -20,7 +20,7 @@ SCRIPTS_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd)" WORKING_DI

Re: [PR] Switch instance type to m7g for ARM image building [airflow]

2024-02-01 Thread via GitHub
Taragolis commented on code in PR #37134: URL: https://github.com/apache/airflow/pull/37134#discussion_r1474616257 ## scripts/ci/images/ci_start_arm_instance_and_connect_to_docker.sh: ## @@ -20,7 +20,7 @@ SCRIPTS_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd)" WORKING_DI

Re: [PR] Added ability to enable/disable scheduler and webserver [airflow]

2024-02-01 Thread via GitHub
kajanajustin commented on PR #36991: URL: https://github.com/apache/airflow/pull/36991#issuecomment-1921512317 @jedcunningham comment has been removed can we merge this in -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [PR] Remove pid arg from celery option to fix duplicate pid issue, Move celery command to provider package [airflow]

2024-02-01 Thread via GitHub
potiuk commented on PR #36794: URL: https://github.com/apache/airflow/pull/36794#issuecomment-1921511093 > @potiuk can you tell me why its failing to import module from airflow, core modules should be available to providers right > > ` from airflow.cli.commands.daemon_utils import ru

Re: [PR] Remove pid arg from celery option to fix duplicate pid issue, Move celery command to provider package [airflow]

2024-02-01 Thread via GitHub
pavansharma36 commented on PR #36794: URL: https://github.com/apache/airflow/pull/36794#issuecomment-1921498640 @potiuk can you tell me why its failing to import module from airflow, core modules should be available to providers right ` from airflow.cli.commands.daemon_utils import

[PR] Add Doctrine in users [airflow]

2024-02-01 Thread via GitHub
anteverse opened a new pull request, #37135: URL: https://github.com/apache/airflow/pull/37135 Add the company Doctrine as a user. --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pul

Re: [PR] Add Doctrine in users [airflow]

2024-02-01 Thread via GitHub
boring-cyborg[bot] commented on PR #37135: URL: https://github.com/apache/airflow/pull/37135#issuecomment-1921492718 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors'

Re: [PR] Switch instance type to m7g for ARM image building [airflow]

2024-02-01 Thread via GitHub
potiuk commented on PR #37134: URL: https://github.com/apache/airflow/pull/37134#issuecomment-1921482094 Following up after this morning v2-8-stable change that was needed to re-release Airflow Image with updated base image. -- This is an automated message from the Apache Git Service. To

[PR] Switch instance type to m7g for ARM image building [airflow]

2024-02-01 Thread via GitHub
potiuk opened a new pull request, #37134: URL: https://github.com/apache/airflow/pull/37134 Seems that it gets harder and harder to get the old instance types for ARM image building. This one changes it to newer ones. In a short term we will also likely switch to the public ARM inst

(airflow) branch constraints-main updated: Updating constraints. Github run id:7742157710

2024-02-01 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 3f3495d6ab Updating con

Re: [PR] Attempt to update Connexion library to version 3 [airflow]

2024-02-01 Thread via GitHub
lewijw commented on PR #36052: URL: https://github.com/apache/airflow/pull/36052#issuecomment-1921462946 FYI. I was able to get past the prior issue by adding a "with": def _handle_api_not_found( request: ConnexionRequest, ex: starlette.exceptions.HTTPException )

Re: [PR] The task is stuck in a queued state forever in case of pod launch errors [airflow]

2024-02-01 Thread via GitHub
devscheffer commented on PR #36882: URL: https://github.com/apache/airflow/pull/36882#issuecomment-1921445378 I had similar problems and thought about something like that ``` from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator from airflo

Re: [PR] Add new run type "triggered" to distinguish manual DAG runs and triggered DAG runs [airflow]

2024-02-01 Thread via GitHub
molcay commented on PR #37087: URL: https://github.com/apache/airflow/pull/37087#issuecomment-1921398205 > Not sure if it belongs to this PR but if it's not too much work can u add it to the cluster activity? Hi @romsharon98, Thank you for pointing out, it looks like it is an rathe

Re: [PR] Introduce `DatasetOrTimeSchedule` [airflow]

2024-02-01 Thread via GitHub
phanikumv merged PR #36710: URL: https://github.com/apache/airflow/pull/36710 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflo

(airflow) branch main updated: Introduce `DatasetOrTimeSchedule` (#36710)

2024-02-01 Thread phanikumv
This is an automated email from the ASF dual-hosted git repository. phanikumv pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new fb27898a77 Introduce `DatasetOrTimeSchedule` (#3

Re: [I] Triggering DAG with parameters is mandatory when show_trigger_form_if_no_params is enabled [airflow]

2024-02-01 Thread via GitHub
jscheffl commented on issue #36843: URL: https://github.com/apache/airflow/issues/36843#issuecomment-1921298598 Hi, You need to set it as described here: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#show-trigger-form-if-no-params Either in airfl

Re: [I] Kubernetes executor pod reconcilation is slow due to recurssive deep copy [airflow]

2024-02-01 Thread via GitHub
potiuk commented on issue #37131: URL: https://github.com/apache/airflow/issues/37131#issuecomment-1921215369 @dirrao - can you please describe the actual change that you've done to get it (code snippets?) that would be great to make sure that we are talking about the same things :) -- T

Re: [I] Kubernetes executor pod reconcilation is slow due to recurssive deep copy [airflow]

2024-02-01 Thread via GitHub
dirrao commented on issue #37131: URL: https://github.com/apache/airflow/issues/37131#issuecomment-1921136542 @hussein-awala / @jedcunningham I would like to hear your thoughts on this. -- This is an automated message from the Apache Git Service. To respond to the message, please log

[I] Kubernetes executor pod reconcilation is slow due to recurssive deep copy [airflow]

2024-02-01 Thread via GitHub
dirrao opened a new issue, #37131: URL: https://github.com/apache/airflow/issues/37131 ### Apache Airflow Provider(s) cncf-kubernetes ### Versions of Apache Airflow Providers 7.1.4.0 ### Apache Airflow version 2.8.1 ### Operating System Cent OS

Re: [I] Triggering DAG with parameters is mandatory when show_trigger_form_if_no_params is enabled [airflow]

2024-02-01 Thread via GitHub
wilk85 commented on issue #36843: URL: https://github.com/apache/airflow/issues/36843#issuecomment-1921047769 Hello, where i need to set show_trigger_form_if_no_params? and also i have no trigger with config in 2.7.3 enveironment varaible AIRFLOW__WEBSERVER__SHOW_TRIGGER_FORM_IF_NO_PARAM

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-02-01 Thread via GitHub
pateash commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1474223517 ## airflow/models/dagrun.py: ## @@ -498,6 +498,37 @@ def fetch_task_instances( tis = tis.where(TI.task_id.in_(task_ids)) return session.scalars(

Re: [PR] allows users to write dag_id and task_id in their national characters, added display name for dag / task [airflow]

2024-02-01 Thread via GitHub
xgao1023 commented on PR #32520: URL: https://github.com/apache/airflow/pull/32520#issuecomment-1920992158 > I tried to add a commit to address the error mentioned above, but there are multiple linting errors here and a merge conflict that must be fixed first. @uranusjr I was tryi

Re: [PR] Python3.12 [airflow]

2024-02-01 Thread via GitHub
potiuk commented on PR #36755: URL: https://github.com/apache/airflow/pull/36755#issuecomment-1920985769 > nice to see more pythonista pushing the envolpe and adopting 3.12, I know it's teidius but it really help the community in general, thanks for those efforts. Thanks @fruch . Muc

Re: [PR] Python3.12 [airflow]

2024-02-01 Thread via GitHub
fruch commented on PR #36755: URL: https://github.com/apache/airflow/pull/36755#issuecomment-1920943564 > > Hmm.. I looks like 3.29 version of [cassandra-driver](https://github.com/datastax/python-driver/blob/master/CHANGELOG.rst) should support Python 3.12 > > In a way. Looking to t

Re: [PR] Switch instance type to m7g [airflow]

2024-02-01 Thread via GitHub
potiuk merged PR #37127: URL: https://github.com/apache/airflow/pull/37127 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

(airflow) branch v2-8-stable updated: Switch instance type to m7g (#37127)

2024-02-01 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v2-8-stable in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/v2-8-stable by this push: new 1745dbe907 Switch instance type to m7

[PR] Switch instance type to m7g [airflow]

2024-02-01 Thread via GitHub
potiuk opened a new pull request, #37127: URL: https://github.com/apache/airflow/pull/37127 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pul

Re: [PR] Create GKEStartJobOperator and KubernetesJobOperator [airflow]

2024-02-01 Thread via GitHub
dirrao commented on PR #36847: URL: https://github.com/apache/airflow/pull/36847#issuecomment-1920906086 > šŸ‘ > > would be great to also add some doc there -> #36847 > > like KPO vs KOJ ... Yes. It would be great to document when to prefer KJO over KPO. -- This is an au

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-02-01 Thread via GitHub
dirrao commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1474086299 ## airflow/models/dagrun.py: ## @@ -498,6 +498,37 @@ def fetch_task_instances( tis = tis.where(TI.task_id.in_(task_ids)) return session.scalars(t