[GitHub] [airflow] eladkal commented on a diff in pull request #28538: Move Hive macros to the provider

2022-12-22 Thread GitBox
eladkal commented on code in PR #28538: URL: https://github.com/apache/airflow/pull/28538#discussion_r1056114079 ## airflow/providers/apache/hive/plugins/hive_plugin.py: ## @@ -0,0 +1,42 @@ +# Licensed to the Apache Software Foundation (ASF) under one Review Comment: You

[GitHub] [airflow] potiuk closed issue #28550: strange order of db migration job launch

2022-12-22 Thread GitBox
potiuk closed issue #28550: strange order of db migration job launch URL: https://github.com/apache/airflow/issues/28550 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [airflow] potiuk commented on issue #25032: Dynamic Task Mapping - Depth-first execution

2022-12-22 Thread GitBox
potiuk commented on issue #25032: URL: https://github.com/apache/airflow/issues/25032#issuecomment-1363694326 > Is it possible to expand a taskgroup from task result in release 2.5? I believe it is @uranusjr - or I misunderstood it ? -- This is an automated message from the Apache

[GitHub] [airflow] jaegwonseo opened a new issue, #28550: strange order of db migration job launch

2022-12-22 Thread GitBox
jaegwonseo opened a new issue, #28550: URL: https://github.com/apache/airflow/issues/28550 ### Official Helm Chart version 1.7.0 (latest released) ### Apache Airflow version 2.4 ### Kubernetes Version 1.2.3 ### Helm Chart configuration

[GitHub] [airflow] VladaZakharova commented on pull request #28284: Fix for issue with reading schema fields for JSON files in GCSToBigQueryOperator

2022-12-22 Thread GitBox
VladaZakharova commented on PR #28284: URL: https://github.com/apache/airflow/pull/28284#issuecomment-1363692360 Thank you for helping me to improve my code :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [airflow] potiuk commented on a diff in pull request #28541: Speed up provider validation pre-commit

2022-12-22 Thread GitBox
potiuk commented on code in PR #28541: URL: https://github.com/apache/airflow/pull/28541#discussion_r1056101696 ## scripts/ci/pre_commit/pre_commit_check_provider_yaml_files.py: ## @@ -38,6 +39,9 @@ from airflow_breeze.utils.docker_command_utils import

[GitHub] [airflow] jens-scheffler-bosch commented on pull request #27063: Feature/26215 AIP-50 Trigger UI based on FAB

2022-12-22 Thread GitBox
jens-scheffler-bosch commented on PR #27063: URL: https://github.com/apache/airflow/pull/27063#issuecomment-1363689224 Hi @pierrejeambrun - thanks again for the review! I now closed the gaps and made the pipeline green - functional-wise the AIP-50 implementation is completed now. Now time

[GitHub] [airflow] potiuk commented on a diff in pull request #28541: Speed up provider validation pre-commit

2022-12-22 Thread GitBox
potiuk commented on code in PR #28541: URL: https://github.com/apache/airflow/pull/28541#discussion_r1056098444 ## scripts/ci/pre_commit/pre_commit_check_provider_yaml_files.py: ## @@ -38,6 +39,9 @@ from airflow_breeze.utils.docker_command_utils import

[GitHub] [airflow] dstandish commented on a diff in pull request #26639: Multi-threads support for processing diff queues in Kubernetes Executor

2022-12-22 Thread GitBox
dstandish commented on code in PR #26639: URL: https://github.com/apache/airflow/pull/26639#discussion_r1056086480 ## airflow/executors/kubernetes_executor.py: ## @@ -62,6 +64,50 @@ KubernetesWatchType = Tuple[str, str, Optional[str], Dict[str, str], str] +def

[GitHub] [airflow] ephraimbuddy commented on a diff in pull request #28469: Support using a list of callbacks in `on_*_callback/sla_miss_callback`s

2022-12-22 Thread GitBox
ephraimbuddy commented on code in PR #28469: URL: https://github.com/apache/airflow/pull/28469#discussion_r1056077448 ## airflow/dag_processing/processor.py: ## @@ -473,13 +473,21 @@ def manage_slas(self, dag: DAG, session: Session = None) -> None:

[GitHub] [airflow] dstandish commented on a diff in pull request #26639: Multi-threads support for processing diff queues in Kubernetes Executor

2022-12-22 Thread GitBox
dstandish commented on code in PR #26639: URL: https://github.com/apache/airflow/pull/26639#discussion_r1056077346 ## airflow/executors/kubernetes_executor.py: ## @@ -62,6 +64,50 @@ KubernetesWatchType = Tuple[str, str, Optional[str], Dict[str, str], str] +def

[GitHub] [airflow] dstandish commented on a diff in pull request #26639: Multi-threads support for processing diff queues in Kubernetes Executor

2022-12-22 Thread GitBox
dstandish commented on code in PR #26639: URL: https://github.com/apache/airflow/pull/26639#discussion_r1056073815 ## airflow/executors/kubernetes_executor.py: ## @@ -62,6 +64,50 @@ KubernetesWatchType = Tuple[str, str, Optional[str], Dict[str, str], str] +def

[GitHub] [airflow] dstandish commented on a diff in pull request #26639: Multi-threads support for processing diff queues in Kubernetes Executor

2022-12-22 Thread GitBox
dstandish commented on code in PR #26639: URL: https://github.com/apache/airflow/pull/26639#discussion_r1056073449 ## airflow/executors/kubernetes_executor.py: ## @@ -62,6 +64,50 @@ KubernetesWatchType = Tuple[str, str, Optional[str], Dict[str, str], str] +def

[GitHub] [airflow] dstandish commented on a diff in pull request #26639: Multi-threads support for processing diff queues in Kubernetes Executor

2022-12-22 Thread GitBox
dstandish commented on code in PR #26639: URL: https://github.com/apache/airflow/pull/26639#discussion_r1056064275 ## airflow/executors/kubernetes_executor.py: ## @@ -370,7 +416,17 @@ def sync(self) -> None: """ self.log.debug("Syncing KubernetesExecutor")

[GitHub] [airflow] dstandish commented on a diff in pull request #28546: Use labels instead of pod name for pod log read in k8s exec

2022-12-22 Thread GitBox
dstandish commented on code in PR #28546: URL: https://github.com/apache/airflow/pull/28546#discussion_r105590 ## airflow/kubernetes/pod_generator.py: ## @@ -430,6 +419,70 @@ def construct_pod( except Exception as e: raise PodReconciliationError from e

[GitHub] [airflow] wangyaopw commented on issue #25032: Dynamic Task Mapping - Depth-first execution

2022-12-22 Thread GitBox
wangyaopw commented on issue #25032: URL: https://github.com/apache/airflow/issues/25032#issuecomment-1363603332 Hi, @potiuk I don't think it is possible to expand a taskgroup from the result of a task yet in 2.5.0. For example ``` from airflow import DAG from

[GitHub] [airflow] IKholopov commented on pull request #28284: Fix for issue with reading schema fields for JSON files in GCSToBigQueryOperator

2022-12-22 Thread GitBox
IKholopov commented on PR #28284: URL: https://github.com/apache/airflow/pull/28284#issuecomment-1363535466 Thanks for fixing it up, Vlada! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] dstandish commented on a diff in pull request #28546: Use labels instead of pod name for pod log read in k8s exec

2022-12-22 Thread GitBox
dstandish commented on code in PR #28546: URL: https://github.com/apache/airflow/pull/28546#discussion_r105590 ## airflow/kubernetes/pod_generator.py: ## @@ -430,6 +419,70 @@ def construct_pod( except Exception as e: raise PodReconciliationError from e

[GitHub] [airflow] pingzh commented on a diff in pull request #28477: Ensure correct log dir in file task handler

2022-12-22 Thread GitBox
pingzh commented on code in PR #28477: URL: https://github.com/apache/airflow/pull/28477#discussion_r1055962697 ## airflow/utils/log/file_task_handler.py: ## @@ -318,6 +319,34 @@ def read(self, task_instance, try_number=None, metadata=None): return logs,

[GitHub] [airflow] Taragolis closed issue #28547: Snowflake Connection - Failing

2022-12-22 Thread GitBox
Taragolis closed issue #28547: Snowflake Connection - Failing URL: https://github.com/apache/airflow/issues/28547 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [airflow] Taragolis commented on issue #28547: Snowflake Connection - Failing

2022-12-22 Thread GitBox
Taragolis commented on issue #28547: URL: https://github.com/apache/airflow/issues/28547#issuecomment-1363468965 > [2022-12-22, 22:38:41 UTC] {local_task_job.py:159} INFO - Task exited with return code Negsignal.SIGSEGV This could reference to known issue with fork process in MacOS,

[GitHub] [airflow] Taragolis commented on a diff in pull request #28525: Add CloudRunExecuteJobOperator

2022-12-22 Thread GitBox
Taragolis commented on code in PR #28525: URL: https://github.com/apache/airflow/pull/28525#discussion_r1055947023 ## tests/providers/google/cloud/hooks/test_cloud_run.py: ## @@ -0,0 +1,78 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more

[GitHub] [airflow] Taragolis closed issue #28549: ImportError: cannot import name 'Markup' from 'jinja2.utils'

2022-12-22 Thread GitBox
Taragolis closed issue #28549: ImportError: cannot import name 'Markup' from 'jinja2.utils' URL: https://github.com/apache/airflow/issues/28549 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] Taragolis commented on issue #28549: ImportError: cannot import name 'Markup' from 'jinja2.utils'

2022-12-22 Thread GitBox
Taragolis commented on issue #28549: URL: https://github.com/apache/airflow/issues/28549#issuecomment-1363460238 The error happen in `jinjasql`, the relevant issue open in their repo: https://github.com/sripathikrishnan/jinjasql/issues/50. However `jinjasql` is not a part of Airflow

[GitHub] [airflow] github-actions[bot] commented on pull request #27481: Add "user_updated_state" column in taskinstance

2022-12-22 Thread GitBox
github-actions[bot] commented on PR #27481: URL: https://github.com/apache/airflow/pull/27481#issuecomment-1363458206 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for

[GitHub] [airflow] Amphagory opened a new issue, #28549: ImportError: cannot import name 'Markup' from 'jinja2.utils'

2022-12-22 Thread GitBox
Amphagory opened a new issue, #28549: URL: https://github.com/apache/airflow/issues/28549 ### Apache Airflow version 2.5.0 ### What happened I am upgrading from V2.3.2 to the latest version at this time 2.5.0. Everything seems to have upgraded fine, until I see DAG

[GitHub] [airflow] boring-cyborg[bot] commented on issue #28549: ImportError: cannot import name 'Markup' from 'jinja2.utils'

2022-12-22 Thread GitBox
boring-cyborg[bot] commented on issue #28549: URL: https://github.com/apache/airflow/issues/28549#issuecomment-1363453233 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] jedcunningham commented on a diff in pull request #28546: Use labels instead of pod name for pod log read in k8s exec

2022-12-22 Thread GitBox
jedcunningham commented on code in PR #28546: URL: https://github.com/apache/airflow/pull/28546#discussion_r1055936849 ## airflow/kubernetes/pod_generator.py: ## @@ -430,6 +419,70 @@ def construct_pod( except Exception as e: raise PodReconciliationError

[GitHub] [airflow] dstandish commented on a diff in pull request #28546: Use labels instead of pod name for pod log read in k8s exec

2022-12-22 Thread GitBox
dstandish commented on code in PR #28546: URL: https://github.com/apache/airflow/pull/28546#discussion_r1055927732 ## airflow/kubernetes/pod_generator.py: ## @@ -430,6 +419,70 @@ def construct_pod( except Exception as e: raise PodReconciliationError from e

[GitHub] [airflow] dstandish commented on a diff in pull request #28546: Use labels instead of pod name for pod log read in k8s exec

2022-12-22 Thread GitBox
dstandish commented on code in PR #28546: URL: https://github.com/apache/airflow/pull/28546#discussion_r1055927346 ## airflow/kubernetes/pod_generator.py: ## @@ -430,6 +419,70 @@ def construct_pod( except Exception as e: raise PodReconciliationError from e

[GitHub] [airflow] jedcunningham commented on a diff in pull request #28546: Use labels instead of pod name for pod log read in k8s exec

2022-12-22 Thread GitBox
jedcunningham commented on code in PR #28546: URL: https://github.com/apache/airflow/pull/28546#discussion_r1055927271 ## airflow/kubernetes/pod_generator.py: ## @@ -430,6 +419,70 @@ def construct_pod( except Exception as e: raise PodReconciliationError

[GitHub] [airflow] dstandish commented on a diff in pull request #28546: Use labels instead of pod name for pod log read in k8s exec

2022-12-22 Thread GitBox
dstandish commented on code in PR #28546: URL: https://github.com/apache/airflow/pull/28546#discussion_r1055927080 ## airflow/kubernetes/pod_generator.py: ## @@ -430,6 +419,70 @@ def construct_pod( except Exception as e: raise PodReconciliationError from e

[GitHub] [airflow] dstandish commented on a diff in pull request #28546: Use labels instead of pod name for pod log read in k8s exec

2022-12-22 Thread GitBox
dstandish commented on code in PR #28546: URL: https://github.com/apache/airflow/pull/28546#discussion_r1055923312 ## airflow/kubernetes/pod_generator.py: ## @@ -430,6 +419,70 @@ def construct_pod( except Exception as e: raise PodReconciliationError from e

[GitHub] [airflow] jedcunningham commented on a diff in pull request #28546: Use labels instead of pod name for pod log read in k8s exec

2022-12-22 Thread GitBox
jedcunningham commented on code in PR #28546: URL: https://github.com/apache/airflow/pull/28546#discussion_r1055926509 ## airflow/kubernetes/pod_generator.py: ## @@ -430,6 +419,70 @@ def construct_pod( except Exception as e: raise PodReconciliationError

[GitHub] [airflow] dstandish commented on a diff in pull request #28546: Use labels instead of pod name for pod log read in k8s exec

2022-12-22 Thread GitBox
dstandish commented on code in PR #28546: URL: https://github.com/apache/airflow/pull/28546#discussion_r1055926291 ## airflow/utils/log/file_task_handler.py: ## @@ -191,19 +191,29 @@ def _read(self, ti: TaskInstance, try_number: int, metadata: dict[str, Any] | No

[GitHub] [airflow] dstandish commented on a diff in pull request #28546: Use labels instead of pod name for pod log read in k8s exec

2022-12-22 Thread GitBox
dstandish commented on code in PR #28546: URL: https://github.com/apache/airflow/pull/28546#discussion_r1055926224 ## airflow/utils/log/file_task_handler.py: ## @@ -191,19 +191,29 @@ def _read(self, ti: TaskInstance, try_number: int, metadata: dict[str, Any] | No

[GitHub] [airflow] dstandish commented on a diff in pull request #28546: Use labels instead of pod name for pod log read in k8s exec

2022-12-22 Thread GitBox
dstandish commented on code in PR #28546: URL: https://github.com/apache/airflow/pull/28546#discussion_r1055925550 ## airflow/utils/log/file_task_handler.py: ## @@ -191,19 +191,29 @@ def _read(self, ti: TaskInstance, try_number: int, metadata: dict[str, Any] | No

[GitHub] [airflow] jedcunningham commented on a diff in pull request #28546: Use labels instead of pod name for pod log read in k8s exec

2022-12-22 Thread GitBox
jedcunningham commented on code in PR #28546: URL: https://github.com/apache/airflow/pull/28546#discussion_r1055925414 ## airflow/utils/log/file_task_handler.py: ## @@ -191,19 +191,29 @@ def _read(self, ti: TaskInstance, try_number: int, metadata: dict[str, Any] | No

[GitHub] [airflow] Taragolis commented on a diff in pull request #28484: Add execution role parameter to AddStepsOperator

2022-12-22 Thread GitBox
Taragolis commented on code in PR #28484: URL: https://github.com/apache/airflow/pull/28484#discussion_r1055925278 ## tests/providers/amazon/aws/operators/test_emr_add_steps.py: ## @@ -224,4 +224,5 @@ def test_wait_for_completion(self, mock_add_job_flow_steps):

[GitHub] [airflow] dstandish commented on a diff in pull request #28546: Use labels instead of pod name for pod log read in k8s exec

2022-12-22 Thread GitBox
dstandish commented on code in PR #28546: URL: https://github.com/apache/airflow/pull/28546#discussion_r1055924742 ## airflow/utils/log/file_task_handler.py: ## @@ -191,19 +191,29 @@ def _read(self, ti: TaskInstance, try_number: int, metadata: dict[str, Any] | No

[GitHub] [airflow] uranusjr commented on a diff in pull request #28540: Using pypi proxy for speeding up development

2022-12-22 Thread GitBox
uranusjr commented on code in PR #28540: URL: https://github.com/apache/airflow/pull/28540#discussion_r1055924147 ## docs/docker-stack/entrypoint.rst: ## @@ -415,3 +415,16 @@ Example: apache/airflow:2.6.0.dev0-python3.8 webserver This method is only available starting

[GitHub] [airflow] dstandish commented on a diff in pull request #28546: Use labels instead of pod name for pod log read in k8s exec

2022-12-22 Thread GitBox
dstandish commented on code in PR #28546: URL: https://github.com/apache/airflow/pull/28546#discussion_r1055923961 ## airflow/kubernetes/pod_generator.py: ## @@ -430,6 +419,70 @@ def construct_pod( except Exception as e: raise PodReconciliationError from e

[GitHub] [airflow] Taragolis commented on pull request #28484: Add execution role parameter to AddStepsOperator

2022-12-22 Thread GitBox
Taragolis commented on PR #28484: URL: https://github.com/apache/airflow/pull/28484#issuecomment-1363431974 >Made a minor change with how the new parameter gets used in the hook. Because we are using @mock_emr decorator in some of the tests, it applies boto3 validation rules on the

[GitHub] [airflow] dstandish commented on a diff in pull request #28546: Use labels instead of pod name for pod log read in k8s exec

2022-12-22 Thread GitBox
dstandish commented on code in PR #28546: URL: https://github.com/apache/airflow/pull/28546#discussion_r1055923312 ## airflow/kubernetes/pod_generator.py: ## @@ -430,6 +419,70 @@ def construct_pod( except Exception as e: raise PodReconciliationError from e

[GitHub] [airflow] uranusjr commented on a diff in pull request #28541: Speed up provider validation pre-commit

2022-12-22 Thread GitBox
uranusjr commented on code in PR #28541: URL: https://github.com/apache/airflow/pull/28541#discussion_r1055922836 ## scripts/ci/pre_commit/pre_commit_check_provider_yaml_files.py: ## @@ -38,6 +39,9 @@ from airflow_breeze.utils.docker_command_utils import

[GitHub] [airflow] jedcunningham commented on a diff in pull request #28546: Use labels instead of pod name for pod log read in k8s exec

2022-12-22 Thread GitBox
jedcunningham commented on code in PR #28546: URL: https://github.com/apache/airflow/pull/28546#discussion_r1055921589 ## airflow/kubernetes/pod_generator.py: ## @@ -430,6 +419,70 @@ def construct_pod( except Exception as e: raise PodReconciliationError

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #28548: Move comment to the right position

2022-12-22 Thread GitBox
boring-cyborg[bot] commented on PR #28548: URL: https://github.com/apache/airflow/pull/28548#issuecomment-1363430028 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution

[GitHub] [airflow] federicsp opened a new pull request, #28548: Move comment to the right position

2022-12-22 Thread GitBox
federicsp opened a new pull request, #28548: URL: https://github.com/apache/airflow/pull/28548 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more

[GitHub] [airflow] dstandish commented on a diff in pull request #28546: Use labels instead of pod name for pod log read in k8s exec

2022-12-22 Thread GitBox
dstandish commented on code in PR #28546: URL: https://github.com/apache/airflow/pull/28546#discussion_r1055921963 ## airflow/utils/log/file_task_handler.py: ## @@ -191,19 +191,29 @@ def _read(self, ti: TaskInstance, try_number: int, metadata: dict[str, Any] | No

[GitHub] [airflow] ashb commented on a diff in pull request #28546: Use labels instead of pod name for pod log read in k8s exec

2022-12-22 Thread GitBox
ashb commented on code in PR #28546: URL: https://github.com/apache/airflow/pull/28546#discussion_r1055920303 ## airflow/utils/log/file_task_handler.py: ## @@ -191,19 +191,29 @@ def _read(self, ti: TaskInstance, try_number: int, metadata: dict[str, Any] | No

[GitHub] [airflow] Taragolis commented on a diff in pull request #28545: Fix Provider Amazon Test Connection when using custom endpoint url

2022-12-22 Thread GitBox
Taragolis commented on code in PR #28545: URL: https://github.com/apache/airflow/pull/28545#discussion_r1055917750 ## airflow/providers/amazon/aws/hooks/base_aws.py: ## @@ -754,7 +754,9 @@ def test_connection(self): """ try: session =

[GitHub] [airflow] boring-cyborg[bot] commented on issue #28547: Snowflake Connection - Failing

2022-12-22 Thread GitBox
boring-cyborg[bot] commented on issue #28547: URL: https://github.com/apache/airflow/issues/28547#issuecomment-1363423525 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] mgovindasamy opened a new issue, #28547: Snowflake Connection - Failing

2022-12-22 Thread GitBox
mgovindasamy opened a new issue, #28547: URL: https://github.com/apache/airflow/issues/28547 ### Apache Airflow version 2.5.0 ### What happened Snowflake Connection Cursor is failing all the time with SIGSEGV, the same code just calling as pyscript is working fine

[GitHub] [airflow] dstandish opened a new pull request, #28546: Use labels instead of pod name for pod log read in k8s exec

2022-12-22 Thread GitBox
dstandish opened a new pull request, #28546: URL: https://github.com/apache/airflow/pull/28546 This means we don't have to use ti.hostname as a proxy for pod name, and allows us to lift the 63 charcter limit, which was a consequence of getting pod name through hostname. -- This is

[GitHub] [airflow] potiuk commented on pull request #28538: Move Hive macros to the provider

2022-12-22 Thread GitBox
potiuk commented on PR #28538: URL: https://github.com/apache/airflow/pull/28538#issuecomment-1363414047 Still need to sort out versioning, and test it, but it's close to be ready IMHO -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [airflow] potiuk commented on a diff in pull request #28538: Move Hive macros to the provider

2022-12-22 Thread GitBox
potiuk commented on code in PR #28538: URL: https://github.com/apache/airflow/pull/28538#discussion_r1055911192 ## airflow/providers/apache/hive/plugins/hive.py: ## @@ -0,0 +1,41 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #28545: Fix Provider Amazon Test Connection when using custom endpoint url

2022-12-22 Thread GitBox
boring-cyborg[bot] commented on PR #28545: URL: https://github.com/apache/airflow/pull/28545#issuecomment-1363405163 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution

[GitHub] [airflow] confusedpublic opened a new pull request, #28545: Fix Provider Amazon Test Connection when using custom endpoint url

2022-12-22 Thread GitBox
confusedpublic opened a new pull request, #28545: URL: https://github.com/apache/airflow/pull/28545 Original `test_connection` behaviour did not account for the use of custom endpoint urls. This would lead to all attempts to test such connections to fail, as the calls to

[GitHub] [airflow] ashb commented on a diff in pull request #28534: Ensure that pod_mutation_hook is called before logging the pod name

2022-12-22 Thread GitBox
ashb commented on code in PR #28534: URL: https://github.com/apache/airflow/pull/28534#discussion_r1055897468 ## airflow/kubernetes/pod_generator.py: ## @@ -349,6 +354,7 @@ def construct_pod( scheduler_job_id: str, run_id: str | None = None,

[GitHub] [airflow] ashb commented on a diff in pull request #28534: Ensure that pod_mutation_hook is called before logging the pod name

2022-12-22 Thread GitBox
ashb commented on code in PR #28534: URL: https://github.com/apache/airflow/pull/28534#discussion_r1055897338 ## airflow/executors/kubernetes_executor.py: ## @@ -346,6 +340,8 @@ def run_next(self, next_job: KubernetesJobType) -> None: args=command,

[GitHub] [airflow] potiuk commented on pull request #28544: Add back join to zombie query that was dropped in #28198

2022-12-22 Thread GitBox
potiuk commented on PR #28544: URL: https://github.com/apache/airflow/pull/28544#issuecomment-1363397103 Saw it today in main .. Was about to take a look :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [airflow] jedcunningham opened a new pull request, #28544: Add back join to zombie query that was dropped in #28198

2022-12-22 Thread GitBox
jedcunningham opened a new pull request, #28544: URL: https://github.com/apache/airflow/pull/28544 #28198 accidentally dropped a join in a query, leading to this: airflow/jobs/scheduler_job.py:1547 SAWarning: SELECT statement has a cartesian product between FROM eleme

[GitHub] [airflow] potiuk commented on a diff in pull request #28538: Move Hive macros to the provider

2022-12-22 Thread GitBox
potiuk commented on code in PR #28538: URL: https://github.com/apache/airflow/pull/28538#discussion_r1055895315 ## airflow/providers/apache/hive/plugins/hive.py: ## @@ -0,0 +1,41 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

[GitHub] [airflow] potiuk commented on a diff in pull request #28538: Move Hive macros to the provider

2022-12-22 Thread GitBox
potiuk commented on code in PR #28538: URL: https://github.com/apache/airflow/pull/28538#discussion_r1055894887 ## airflow/macros/__init__.py: ## @@ -26,8 +26,6 @@ import dateutil # noqa from pendulum import DateTime -from airflow.macros import hive # noqa Review

[airflow] branch main updated: Cleanup and do housekeeping with plugin examples (#28537)

2022-12-22 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 66eb282b7d Cleanup and do housekeeping with plugin

[GitHub] [airflow] potiuk merged pull request #28537: Cleanup and do housekeeping with plugin examples

2022-12-22 Thread GitBox
potiuk merged PR #28537: URL: https://github.com/apache/airflow/pull/28537 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] dstandish commented on pull request #28198: Fix scheduler orm DetachedInstanceError when find zombies in standalone dag processor mode

2022-12-22 Thread GitBox
dstandish commented on PR #28198: URL: https://github.com/apache/airflow/pull/28198#issuecomment-1363386201 looks like this produces this error ``` /Users/dstandish/code/airflow/airflow/jobs/scheduler_job.py:1536 SAWarning: SELECT statement has a cartesian product between FROM

[GitHub] [airflow] syedahsn commented on pull request #28484: Add execution role parameter to AddStepsOperator

2022-12-22 Thread GitBox
syedahsn commented on PR #28484: URL: https://github.com/apache/airflow/pull/28484#issuecomment-1363382860 Made a minor change with how the new parameter gets used in the hook. Because we are using `@mock_emr` decorator in some of the tests, it applies boto3 validation rules on the

[GitHub] [airflow] potiuk commented on pull request #28541: Speed up provider validation pre-commit

2022-12-22 Thread GitBox
potiuk commented on PR #28541: URL: https://github.com/apache/airflow/pull/28541#issuecomment-1363376218 All green. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [airflow] dstandish commented on a diff in pull request #28454: Fix bad pods pickled in executor_config

2022-12-22 Thread GitBox
dstandish commented on code in PR #28454: URL: https://github.com/apache/airflow/pull/28454#discussion_r1055854652 ## airflow/utils/sqlalchemy.py: ## @@ -153,6 +156,75 @@ def process_result_value(self, value, dialect): return BaseSerialization.deserialize(value)

[GitHub] [airflow] ashb opened a new pull request, #28543: Emit warnings for `conf.get*` from the right source location

2022-12-22 Thread GitBox
ashb opened a new pull request, #28543: URL: https://github.com/apache/airflow/pull/28543 `getboolean` and other typed get functions were issuing warnings from "inside" themselves. Before: ``` $ python ./airflow/airflow/kubernetes/kube_client.py

[GitHub] [airflow] ashb opened a new pull request, #28542: Mark all our local sphinx exts as parallel safe

2022-12-22 Thread GitBox
ashb opened a new pull request, #28542: URL: https://github.com/apache/airflow/pull/28542 We don't do anything fancy in any of them that wouldn't work across multiple processes. By default/as invoked right now this makes no difference to performance, but if we add `-j auto` it

[GitHub] [airflow] michaelmicheal commented on a diff in pull request #28481: dagwarning, add error handling to purge_inactive_dag_warnings

2022-12-22 Thread GitBox
michaelmicheal commented on code in PR #28481: URL: https://github.com/apache/airflow/pull/28481#discussion_r1055846581 ## airflow/models/dagwarning.py: ## @@ -79,7 +83,12 @@ def purge_inactive_dag_warnings(cls, session: Session = NEW_SESSION) -> None: query =

[GitHub] [airflow] potiuk commented on a diff in pull request #28537: Cleanup and do housekeeping with plugin examples

2022-12-22 Thread GitBox
potiuk commented on code in PR #28537: URL: https://github.com/apache/airflow/pull/28537#discussion_r1055846277 ## docs/apache-airflow/howto/custom-view-plugin.rst: ## @@ -48,73 +43,32 @@ Custom view Registration A custom view with object reference

[GitHub] [airflow] dstandish commented on a diff in pull request #28454: Fix bad pods pickled in executor_config

2022-12-22 Thread GitBox
dstandish commented on code in PR #28454: URL: https://github.com/apache/airflow/pull/28454#discussion_r1055844290 ## airflow/utils/sqlalchemy.py: ## @@ -153,6 +156,75 @@ def process_result_value(self, value, dialect): return BaseSerialization.deserialize(value)

[GitHub] [airflow] potiuk commented on a diff in pull request #28537: Cleanup and do housekeeping with plugin examples

2022-12-22 Thread GitBox
potiuk commented on code in PR #28537: URL: https://github.com/apache/airflow/pull/28537#discussion_r1055841466 ## docs/apache-airflow/howto/custom-view-plugin.rst: ## @@ -48,73 +43,32 @@ Custom view Registration A custom view with object reference

[GitHub] [airflow] potiuk closed issue #25032: Dynamic Task Mapping - Depth-first execution

2022-12-22 Thread GitBox
potiuk closed issue #25032: Dynamic Task Mapping - Depth-first execution URL: https://github.com/apache/airflow/issues/25032 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [airflow] potiuk commented on issue #25032: Dynamic Task Mapping - Depth-first execution

2022-12-22 Thread GitBox
potiuk commented on issue #25032: URL: https://github.com/apache/airflow/issues/25032#issuecomment-1363330299 It's already implemented in 2.5.0 - via TaskGroup Mapping. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [airflow] jedcunningham commented on a diff in pull request #28538: Move Hive macros to the provider

2022-12-22 Thread GitBox
jedcunningham commented on code in PR #28538: URL: https://github.com/apache/airflow/pull/28538#discussion_r1055837152 ## airflow/macros/__init__.py: ## @@ -26,8 +26,6 @@ import dateutil # noqa from pendulum import DateTime -from airflow.macros import hive # noqa Review

[GitHub] [airflow] jedcunningham commented on a diff in pull request #28537: Cleanup and do housekeeping with plugin examples

2022-12-22 Thread GitBox
jedcunningham commented on code in PR #28537: URL: https://github.com/apache/airflow/pull/28537#discussion_r1055827387 ## docs/apache-airflow/howto/custom-view-plugin.rst: ## @@ -48,73 +43,32 @@ Custom view Registration A custom view with object

[GitHub] [airflow] potiuk commented on a diff in pull request #28538: Move Hive macros to the provider

2022-12-22 Thread GitBox
potiuk commented on code in PR #28538: URL: https://github.com/apache/airflow/pull/28538#discussion_r1055825466 ## airflow/providers/apache/hive/plugins/hive_plugin.py: ## @@ -0,0 +1,42 @@ +# Licensed to the Apache Software Foundation (ASF) under one Review Comment: Should

[GitHub] [airflow] XD-DENG commented on pull request #28540: Using pypi proxy for speeding up development

2022-12-22 Thread GitBox
XD-DENG commented on PR #28540: URL: https://github.com/apache/airflow/pull/28540#issuecomment-1363302299 > I would not be as strict here - we already endorse a number of development tools in our docs. (and this is such a tool). But at least it should be confirmed to be

[GitHub] [airflow] potiuk commented on pull request #28540: Using pypi proxy for speeding up development

2022-12-22 Thread GitBox
potiuk commented on PR #28540: URL: https://github.com/apache/airflow/pull/28540#issuecomment-1363300844 > > Fair enough, so your recommended change is just to note that this can be done with pypi proxies but avoid mentioning one? > > Yes I believe we need to avoid mentioning one.

[GitHub] [airflow] potiuk commented on pull request #28534: Ensure that pod_mutation_hook is called before logging the pod name

2022-12-22 Thread GitBox
potiuk commented on PR #28534: URL: https://github.com/apache/airflow/pull/28534#issuecomment-1363295309 > wow this may be record for most approved PR :) Let me approve it again to make it closer to record. -- This is an automated message from the Apache Git Service. To respond to

[GitHub] [airflow] ashb commented on pull request #28511: allow a configurable prefix for pod names created by kubernetes executor

2022-12-22 Thread GitBox
ashb commented on PR #28511: URL: https://github.com/apache/airflow/pull/28511#issuecomment-1363285671 Given pod mutation hook exists this just adds config sprawl for a very uncommon use case - I'm closing this PR -- This is an automated message from the Apache Git Service. To respond to

[GitHub] [airflow] ashb closed pull request #28511: allow a configurable prefix for pod names created by kubernetes executor

2022-12-22 Thread GitBox
ashb closed pull request #28511: allow a configurable prefix for pod names created by kubernetes executor URL: https://github.com/apache/airflow/pull/28511 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [airflow] wangyaopw commented on issue #25032: Dynamic Task Mapping - Depth-first execution

2022-12-22 Thread GitBox
wangyaopw commented on issue #25032: URL: https://github.com/apache/airflow/issues/25032#issuecomment-1363282394 +1 There is a highly desirable feature in the community. It would be awesome if Airflow can support this in the near future. -- This is an automated message from the

[GitHub] [airflow] vincbeck commented on a diff in pull request #28502: Migrate DagFileProcessor.manage_slas to Internal API

2022-12-22 Thread GitBox
vincbeck commented on code in PR #28502: URL: https://github.com/apache/airflow/pull/28502#discussion_r1055788988 ## airflow/dag_processing/processor.py: ## @@ -365,8 +367,10 @@ def __init__(self, dag_ids: list[str] | None, dag_directory: str, log: logging.L

[GitHub] [airflow] vincbeck commented on a diff in pull request #28476: Migrate DagFileProcessorManager._deactivate_stale_dags to Internal API

2022-12-22 Thread GitBox
vincbeck commented on code in PR #28476: URL: https://github.com/apache/airflow/pull/28476#discussion_r1055787153 ## airflow/api_internal/endpoints/rpc_api_endpoint.py: ## @@ -68,7 +70,10 @@ def internal_airflow_api( log.debug("Calling method %.", {method_name})

[GitHub] [airflow] dimberman commented on a diff in pull request #28454: Fix bad pods pickled in executor_config

2022-12-22 Thread GitBox
dimberman commented on code in PR #28454: URL: https://github.com/apache/airflow/pull/28454#discussion_r1055774132 ## airflow/utils/sqlalchemy.py: ## @@ -153,6 +156,75 @@ def process_result_value(self, value, dialect): return BaseSerialization.deserialize(value)

[GitHub] [airflow] potiuk commented on pull request #28538: Move Hive macros to the provider

2022-12-22 Thread GitBox
potiuk commented on PR #28538: URL: https://github.com/apache/airflow/pull/28538#issuecomment-1363243094 > Oh... I just more interested in a hypothetical possibility rather than move existed one or implement new one. > > Anyway this approach could add ability of add provider specific

[GitHub] [airflow] potiuk opened a new pull request, #28541: Speed up provider validation pre-commit

2022-12-22 Thread GitBox
potiuk opened a new pull request, #28541: URL: https://github.com/apache/airflow/pull/28541 The provider validation pre-commit is now a lot slower after #28516 but it turned out that it has been also doing a little too much. It worked in the way (against the original design) that

[GitHub] [airflow] Taragolis commented on pull request #28538: Move Hive macros to the provider

2022-12-22 Thread GitBox
Taragolis commented on PR #28538: URL: https://github.com/apache/airflow/pull/28538#issuecomment-1363225566 Oh... I just more interested in a hypothetical possibility rather than move existed one or implement new one. Anyway this approach could add ability of add provider specific

[GitHub] [airflow] potiuk commented on pull request #28300: Add Public Interface description to Airflow documentation

2022-12-22 Thread GitBox
potiuk commented on PR #28300: URL: https://github.com/apache/airflow/pull/28300#issuecomment-1363215643 One other point (which very much concurs with what I described in my proposal about executor is actually something that @Taragolis noticed.

[GitHub] [airflow] potiuk commented on pull request #28538: Move Hive macros to the provider

2022-12-22 Thread GitBox
potiuk commented on PR #28538: URL: https://github.com/apache/airflow/pull/28538#issuecomment-1363208676 > Whoa!!! I never thought from this side. > > That mean potentially we could have CeleryExecutor into celery package and K8S Executors inside cncf.kubernetes package?

[GitHub] [airflow] dstandish commented on pull request #28237: Keep pod name for k8s executor under 63 characters

2022-12-22 Thread GitBox
dstandish commented on PR #28237: URL: https://github.com/apache/airflow/pull/28237#issuecomment-1363195912 i am not sure it does. how does it? FTH gets the pod name from ti.hostname. If you store IP in ti.hostname, then now you are in even worse situation. And

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #28528: Fixes to how DebugExecutor handles sensors

2022-12-22 Thread GitBox
vandonr-amz commented on code in PR #28528: URL: https://github.com/apache/airflow/pull/28528#discussion_r1055725594 ## airflow/sensors/base.py: ## @@ -256,14 +256,13 @@ def _get_next_poke_interval( return new_interval def prepare_for_execution(self) ->

[GitHub] [airflow] VinceLegendre commented on a diff in pull request #28525: Add CloudRunExecuteJobOperator

2022-12-22 Thread GitBox
VinceLegendre commented on code in PR #28525: URL: https://github.com/apache/airflow/pull/28525#discussion_r1055724844 ## tests/providers/google/cloud/hooks/test_cloud_run.py: ## @@ -0,0 +1,78 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more

[GitHub] [airflow] XD-DENG commented on pull request #28540: Using pypi proxy for speeding up development

2022-12-22 Thread GitBox
XD-DENG commented on PR #28540: URL: https://github.com/apache/airflow/pull/28540#issuecomment-1363191782 > Fair enough, so your recommended change is just to note that this can be done with pypi proxies but avoid mentioning one? Yes I believe we need to avoid mentioning one.

[GitHub] [airflow] dstandish commented on a diff in pull request #28534: Ensure that pod_mutation_hook is called before logging the pod name

2022-12-22 Thread GitBox
dstandish commented on code in PR #28534: URL: https://github.com/apache/airflow/pull/28534#discussion_r1055721343 ## airflow/executors/kubernetes_executor.py: ## @@ -346,6 +340,7 @@ def run_next(self, next_job: KubernetesJobType) -> None: args=command,

  1   2   3   >