[GitHub] [airflow] farhan0syakir opened a new pull request, #28799: introduce dag processor job fix #27140

2023-01-08 Thread GitBox
farhan0syakir opened a new pull request, #28799: URL: https://github.com/apache/airflow/pull/28799 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for

[GitHub] [airflow] uranusjr commented on a diff in pull request #28764: Add support for running a Beam Go pipeline with an executable binary

2023-01-08 Thread GitBox
uranusjr commented on code in PR #28764: URL: https://github.com/apache/airflow/pull/28764#discussion_r1064353800 ## airflow/providers/apache/beam/operators/beam.py: ## @@ -563,8 +582,13 @@ def __init__( ) self.dataflow_support_impersonation = False +

[GitHub] [airflow] uranusjr commented on a diff in pull request #28764: Add support for running a Beam Go pipeline with an executable binary

2023-01-08 Thread GitBox
uranusjr commented on code in PR #28764: URL: https://github.com/apache/airflow/pull/28764#discussion_r1064352909 ## airflow/providers/apache/beam/operators/beam.py: ## @@ -563,8 +582,13 @@ def __init__( ) self.dataflow_support_impersonation = False +

[GitHub] [airflow] uranusjr commented on issue #28798: Processor unable to parse DAG with non english characters

2023-01-08 Thread GitBox
uranusjr commented on issue #28798: URL: https://github.com/apache/airflow/issues/28798#issuecomment-1375214889 This looks like a database configuration issue. Did you configure the database’s character set and collation correctly?

[GitHub] [airflow] uranusjr commented on issue #26933: Low priority tasks are scheduled before high priority tasks

2023-01-08 Thread GitBox
uranusjr commented on issue #26933: URL: https://github.com/apache/airflow/issues/26933#issuecomment-1375212752 I think this is intended, at least from my reading of the implementation. Currently the scheduler checks how many free slots in the pool, and schedule at most that number of task

[GitHub] [airflow] Avphy opened a new issue, #28798: Processor unable to parse DAG with non english characters

2023-01-08 Thread GitBox
Avphy opened a new issue, #28798: URL: https://github.com/apache/airflow/issues/28798 ### Apache Airflow version Other Airflow 2 version (please specify below) ### What happened Airflow version: 2.3.2 OS: Linux When processor tries to load DAG files with non

[GitHub] [airflow] VladaZakharova commented on issue #28343: BigQueryColumnCheckOperator doesn't actually implement use_legacy_sql kwarg

2023-01-08 Thread GitBox
VladaZakharova commented on issue #28343: URL: https://github.com/apache/airflow/issues/28343#issuecomment-1375208220 @vchiapaikeo Thank you for the quick PR for this fix! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [airflow] uranusjr commented on a diff in pull request #27428: Handle exception during serializing incompatible executor_config object.

2023-01-08 Thread GitBox
uranusjr commented on code in PR #27428: URL: https://github.com/apache/airflow/pull/27428#discussion_r1064342332 ## airflow/api_connexion/schemas/task_instance_schema.py: ## @@ -33,6 +33,14 @@ from airflow.utils.state import State +class

[GitHub] [airflow] rajaths010494 commented on a diff in pull request #28763: Add deferrable ``GCSObjectExistenceSensorAsync``

2023-01-08 Thread GitBox
rajaths010494 commented on code in PR #28763: URL: https://github.com/apache/airflow/pull/28763#discussion_r1064341941 ## airflow/providers/google/cloud/hooks/gcs.py: ## @@ -32,18 +32,21 @@ from typing import IO, Callable, Generator, Sequence, TypeVar, cast, overload from

[GitHub] [airflow] uranusjr commented on a diff in pull request #28763: Add deferrable ``GCSObjectExistenceSensorAsync``

2023-01-08 Thread GitBox
uranusjr commented on code in PR #28763: URL: https://github.com/apache/airflow/pull/28763#discussion_r1064334417 ## airflow/providers/google/cloud/hooks/gcs.py: ## @@ -32,18 +32,21 @@ from typing import IO, Callable, Generator, Sequence, TypeVar, cast, overload from

[airflow] branch main updated (ce677862be -> 6ca67ba98e)

2023-01-08 Thread uranusjr
This is an automated email from the ASF dual-hosted git repository. uranusjr pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from ce677862be Fix Incorrect `await_container_completion` (#28771) add 6ca67ba98e Update dynamic-task-mapping.rst

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #28797: Update dynamic-task-mapping.rst

2023-01-08 Thread GitBox
boring-cyborg[bot] commented on PR #28797: URL: https://github.com/apache/airflow/pull/28797#issuecomment-1375195803 Awesome work, congrats on your first merged pull request! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [airflow] uranusjr merged pull request #28797: Update dynamic-task-mapping.rst

2023-01-08 Thread GitBox
uranusjr merged PR #28797: URL: https://github.com/apache/airflow/pull/28797 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] Avphy commented on issue #10954: Serialized Dags with MySQL Latin1 encoded airflow db

2023-01-08 Thread GitBox
Avphy commented on issue #10954: URL: https://github.com/apache/airflow/issues/10954#issuecomment-1375193978 We are also facing similar error with Arabic text in DAG file, did anybody find solution to this issue? `2023-01-09_05:55:04.23124 Process DagFileProcessor9-Process:

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #28797: Update dynamic-task-mapping.rst

2023-01-08 Thread GitBox
boring-cyborg[bot] commented on PR #28797: URL: https://github.com/apache/airflow/pull/28797#issuecomment-1375169822 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution

[GitHub] [airflow] sadler-elena opened a new pull request, #28797: Update dynamic-task-mapping.rst

2023-01-08 Thread GitBox
sadler-elena opened a new pull request, #28797: URL: https://github.com/apache/airflow/pull/28797 Added missing word from sentence, using context from previous section. **^ Add meaningful description above** Read the **[Pull Request

[GitHub] [airflow] mlnsharma commented on pull request #27802: Read logs from all containers in pod for KubernetesPodOperator

2023-01-08 Thread GitBox
mlnsharma commented on PR #27802: URL: https://github.com/apache/airflow/pull/27802#issuecomment-1375096177 Hi @potiuk @jedcunningham @XD-DENG @bdsoha Can you please review my code changes? -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [airflow] amoghrajesh commented on issue #28737: Add legend to grid view

2023-01-08 Thread GitBox
amoghrajesh commented on issue #28737: URL: https://github.com/apache/airflow/issues/28737#issuecomment-1375095038 Hi, @bbovenzi I am new to Airflow and would likewith to get a few things clarified before starting. - Do we have some kind of legends in the airflow codebase that i can

[GitHub] [airflow] uranusjr commented on issue #24547: DAGs with Dynamic Task Mapping are not rendered when cluster policies overwrite a parameter

2023-01-08 Thread GitBox
uranusjr commented on issue #24547: URL: https://github.com/apache/airflow/issues/24547#issuecomment-1375044561 Fixed in #28313 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [airflow] uranusjr closed issue #24547: DAGs with Dynamic Task Mapping are not rendered when cluster policies overwrite a parameter

2023-01-08 Thread GitBox
uranusjr closed issue #24547: DAGs with Dynamic Task Mapping are not rendered when cluster policies overwrite a parameter URL: https://github.com/apache/airflow/issues/24547 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [airflow] uranusjr commented on a diff in pull request #28781: Fix UIAlert should_show when AUTH_ROLE_PUBLIC set

2023-01-08 Thread GitBox
uranusjr commented on code in PR #28781: URL: https://github.com/apache/airflow/pull/28781#discussion_r1064255602 ## airflow/www/utils.py: ## @@ -820,10 +822,24 @@ def __init__( self.html = html self.message = Markup(message) if html else message -def

[GitHub] [airflow] uranusjr commented on a diff in pull request #28781: Fix UIAlert should_show when AUTH_ROLE_PUBLIC set

2023-01-08 Thread GitBox
uranusjr commented on code in PR #28781: URL: https://github.com/apache/airflow/pull/28781#discussion_r1064255376 ## airflow/www/utils.py: ## @@ -820,10 +822,26 @@ def __init__( self.html = html self.message = Markup(message) if html else message -def

[GitHub] [airflow] uranusjr commented on a diff in pull request #28781: Fix UIAlert should_show when AUTH_ROLE_PUBLIC set

2023-01-08 Thread GitBox
uranusjr commented on code in PR #28781: URL: https://github.com/apache/airflow/pull/28781#discussion_r1064255188 ## airflow/www/utils.py: ## @@ -820,10 +822,25 @@ def __init__( self.html = html self.message = Markup(message) if html else message -def

[GitHub] [airflow] uranusjr commented on a diff in pull request #28781: Fix UIAlert should_show when AUTH_ROLE_PUBLIC set

2023-01-08 Thread GitBox
uranusjr commented on code in PR #28781: URL: https://github.com/apache/airflow/pull/28781#discussion_r1064255188 ## airflow/www/utils.py: ## @@ -820,10 +822,25 @@ def __init__( self.html = html self.message = Markup(message) if html else message -def

[GitHub] [airflow] uranusjr commented on a diff in pull request #28777: Fix dag run conf encoding with non-JSON serializable values

2023-01-08 Thread GitBox
uranusjr commented on code in PR #28777: URL: https://github.com/apache/airflow/pull/28777#discussion_r1064248601 ## airflow/www/utils.py: ## @@ -133,13 +133,25 @@ def get_mapped_summary(parent_instance, task_instances): def get_dag_run_conf(dag_run_conf: Any) -> tuple[str

[GitHub] [airflow] uranusjr commented on a diff in pull request #28777: Fix dag run conf encoding with non-JSON serializable values

2023-01-08 Thread GitBox
uranusjr commented on code in PR #28777: URL: https://github.com/apache/airflow/pull/28777#discussion_r1064247537 ## airflow/www/utils.py: ## @@ -133,13 +133,25 @@ def get_mapped_summary(parent_instance, task_instances): def get_dag_run_conf(dag_run_conf: Any) -> tuple[str

[GitHub] [airflow] vchiapaikeo commented on issue #28343: BigQueryColumnCheckOperator doesn't actually implement use_legacy_sql kwarg

2023-01-08 Thread GitBox
vchiapaikeo commented on issue #28343: URL: https://github.com/apache/airflow/issues/28343#issuecomment-1375006740 Created this PR to fix the runtime / type error: https://github.com/apache/airflow/pull/28796 -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] vchiapaikeo opened a new pull request, #28796: Fix BigQueryColumnCheckOperator runtime error

2023-01-08 Thread GitBox
vchiapaikeo opened a new pull request, #28796: URL: https://github.com/apache/airflow/pull/28796 A TypeError which causes tasks to fail at runtime exists in the BigQueryColumnCheckOperator. This was initially uncovered while looking at another issue for this operator here:

[GitHub] [airflow] bbovenzi commented on issue #28737: Add legend to grid view

2023-01-08 Thread GitBox
bbovenzi commented on issue #28737: URL: https://github.com/apache/airflow/issues/28737#issuecomment-1375004932 > @bbovenzi is this issue taken? I would like to try my hands here given some more reference material here. Nope! It's all yours. Feel free to ask any questions here.

[GitHub] [airflow] bugraoz93 opened a new pull request, #28795: Migrate Models Variable to Internal API

2023-01-08 Thread GitBox
bugraoz93 opened a new pull request, #28795: URL: https://github.com/apache/airflow/pull/28795 related: #28271 Migration of `Models.Variable` to Internal API. - set - update - delete -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [airflow] Taragolis commented on a diff in pull request #28777: Fix dag run conf encoding with non-JSON serializable values

2023-01-08 Thread GitBox
Taragolis commented on code in PR #28777: URL: https://github.com/apache/airflow/pull/28777#discussion_r1064227851 ## airflow/www/utils.py: ## @@ -133,13 +133,25 @@ def get_mapped_summary(parent_instance, task_instances): def get_dag_run_conf(dag_run_conf: Any) ->

[GitHub] [airflow] github-actions[bot] commented on pull request #22976: Make sure jobs triggered by airflow web are not identified as orphaned.

2023-01-08 Thread GitBox
github-actions[bot] commented on PR #22976: URL: https://github.com/apache/airflow/pull/22976#issuecomment-1374969395 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for

[GitHub] [airflow] github-actions[bot] commented on pull request #23079: Don't clear downstream tasks when marked status is failure and downstream is True

2023-01-08 Thread GitBox
github-actions[bot] commented on PR #23079: URL: https://github.com/apache/airflow/pull/23079#issuecomment-1374969374 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for

[GitHub] [airflow] github-actions[bot] commented on issue #27589: Scheduler pods memory leak on airflow 2.3.2

2023-01-08 Thread GitBox
github-actions[bot] commented on issue #27589: URL: https://github.com/apache/airflow/issues/27589#issuecomment-1374969321 This issue has been closed because it has not received response from the issue author. -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] github-actions[bot] closed issue #27589: Scheduler pods memory leak on airflow 2.3.2

2023-01-08 Thread GitBox
github-actions[bot] closed issue #27589: Scheduler pods memory leak on airflow 2.3.2 URL: https://github.com/apache/airflow/issues/27589 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [airflow] github-actions[bot] commented on issue #27078: Connection time out when doing sts_assume_role with AwsBaseHook

2023-01-08 Thread GitBox
github-actions[bot] commented on issue #27078: URL: https://github.com/apache/airflow/issues/27078#issuecomment-1374969340 This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further

[GitHub] [airflow] github-actions[bot] closed issue #27883: Getting 404 when clicking on download logs button

2023-01-08 Thread GitBox
github-actions[bot] closed issue #27883: Getting 404 when clicking on download logs button URL: https://github.com/apache/airflow/issues/27883 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] github-actions[bot] commented on issue #27883: Getting 404 when clicking on download logs button

2023-01-08 Thread GitBox
github-actions[bot] commented on issue #27883: URL: https://github.com/apache/airflow/issues/27883#issuecomment-1374969290 This issue has been closed because it has not received response from the issue author. -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] hussein-awala commented on a diff in pull request #28777: Fix dag run conf encoding with non-JSON serializable values

2023-01-08 Thread GitBox
hussein-awala commented on code in PR #28777: URL: https://github.com/apache/airflow/pull/28777#discussion_r1064216151 ## airflow/www/utils.py: ## @@ -133,13 +133,25 @@ def get_mapped_summary(parent_instance, task_instances): def get_dag_run_conf(dag_run_conf: Any) ->

[GitHub] [airflow] potiuk closed issue #28793: Airflow 2.3.3 doesn't retry a task if it externally killed

2023-01-08 Thread GitBox
potiuk closed issue #28793: Airflow 2.3.3 doesn't retry a task if it externally killed URL: https://github.com/apache/airflow/issues/28793 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] o-nikolas commented on a diff in pull request #28161: AIP-51 - Executor Coupling in Logging

2023-01-08 Thread GitBox
o-nikolas commented on code in PR #28161: URL: https://github.com/apache/airflow/pull/28161#discussion_r1064208434 ## airflow/executors/kubernetes_executor.py: ## @@ -749,6 +751,57 @@ def _change_state(self, key: TaskInstanceKey, state: str | None, pod_id: str, na

[GitHub] [airflow] potiuk commented on issue #28793: Airflow 2.3.3 doesn't retry a task if it externally killed

2023-01-08 Thread GitBox
potiuk commented on issue #28793: URL: https://github.com/apache/airflow/issues/28793#issuecomment-1374937415 Any logs showing this situation ? Can you please be more specific what happens and when ? -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [airflow] jens-scheffler-bosch commented on pull request #27063: Feature/26215 AIP-50 Trigger UI based on FAB

2023-01-08 Thread GitBox
jens-scheffler-bosch commented on PR #27063: URL: https://github.com/apache/airflow/pull/27063#issuecomment-1374909603 > > Now time for a final review, then (in my view) ready to be merged. > > I don't see vote thread on the mailing list @eladkal thanks for helping out in

[GitHub] [airflow] yxiao1996 commented on issue #17487: Make gantt view to show also retries

2023-01-08 Thread GitBox
yxiao1996 commented on issue #17487: URL: https://github.com/apache/airflow/issues/17487#issuecomment-1374909408 Reproduce the scenario described in this issue with a python operator which causes failed task at 50% chance ``` @task(task_id=f"sleep_and_randomly_fail", retries=1)

[GitHub] [airflow] Taragolis commented on a diff in pull request #28777: Fix dag run conf encoding with non-JSON serializable values

2023-01-08 Thread GitBox
Taragolis commented on code in PR #28777: URL: https://github.com/apache/airflow/pull/28777#discussion_r1064190069 ## airflow/www/utils.py: ## @@ -133,13 +133,25 @@ def get_mapped_summary(parent_instance, task_instances): def get_dag_run_conf(dag_run_conf: Any) ->

[GitHub] [airflow] VladaZakharova commented on pull request #28564: GCSToBigQueryOperator allows autodetect None and infers schema

2023-01-08 Thread GitBox
VladaZakharova commented on PR #28564: URL: https://github.com/apache/airflow/pull/28564#issuecomment-1374904744 Hi Team :) As for the changes in [#28284](https://github.com/apache/airflow/pull/28284) - all the changes for autodetect=True field were also tested for the JSON format

[GitHub] [airflow] pierrejeambrun commented on issue #28556: Make duration on "List Dag Run" sortable

2023-01-08 Thread GitBox
pierrejeambrun commented on issue #28556: URL: https://github.com/apache/airflow/issues/28556#issuecomment-1374903312 Hello @alejandrofm, Just assigned you :) You can start looking at the file `airflow/www/views.py`, the class `DagRunModelView`. Duration is not a db

[GitHub] [airflow] boring-cyborg[bot] commented on issue #28793: Airflow 2.3.3 doesn't retry a task if it externally killed

2023-01-08 Thread GitBox
boring-cyborg[bot] commented on issue #28793: URL: https://github.com/apache/airflow/issues/28793#issuecomment-1374902248 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] nofarm3 opened a new issue, #28793: Airflow 2.3.3 doesn't retry a task if it externally killed

2023-01-08 Thread GitBox
nofarm3 opened a new issue, #28793: URL: https://github.com/apache/airflow/issues/28793 ### Apache Airflow version Other Airflow 2 version (please specify below) ### What happened When a task get externally killed, it is marked as Failed even though it can be retried.

[GitHub] [airflow] potiuk commented on issue #28789: Add colors in help outputs of Airfow CLI commands

2023-01-08 Thread GitBox
potiuk commented on issue #28789: URL: https://github.com/apache/airflow/issues/28789#issuecomment-1374901005 Sure. But I guess it will be "all-or-nothing" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [airflow] VladaZakharova commented on issue #28343: BigQueryColumnCheckOperator doesn't actually implement use_legacy_sql kwarg

2023-01-08 Thread GitBox
VladaZakharova commented on issue #28343: URL: https://github.com/apache/airflow/issues/28343#issuecomment-1374900424 Hi Team :) Sorry for late answer I was trying to reproduce the error from the bug using fields with DATETIME datatypes, but it seems that i have faced the same problem

[GitHub] [airflow] hussein-awala commented on a diff in pull request #28777: Fix dag run conf encoding with non-JSON serializable values

2023-01-08 Thread GitBox
hussein-awala commented on code in PR #28777: URL: https://github.com/apache/airflow/pull/28777#discussion_r1064173784 ## airflow/www/utils.py: ## @@ -133,13 +133,25 @@ def get_mapped_summary(parent_instance, task_instances): def get_dag_run_conf(dag_run_conf: Any) ->

[GitHub] [airflow] eladkal closed issue #20829: Update google-cloud-container package to v2.10.1

2023-01-08 Thread GitBox
eladkal closed issue #20829: Update google-cloud-container package to v2.10.1 URL: https://github.com/apache/airflow/issues/20829 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [airflow] eladkal commented on issue #20829: Update google-cloud-container package to v2.10.1

2023-01-08 Thread GitBox
eladkal commented on issue #20829: URL: https://github.com/apache/airflow/issues/20829#issuecomment-1374848627 Closing as completed https://github.com/apache/airflow/blob/5246c009c557b4f6bdf1cd62bf9b89a2da63f630/airflow/providers/google/provider.yaml#L86 -- This is an automated

[GitHub] [airflow] Taragolis commented on pull request #28767: Remove `@pytest.mark.quarantined` from remaining quarantine tests

2023-01-08 Thread GitBox
Taragolis commented on PR #28767: URL: https://github.com/apache/airflow/pull/28767#issuecomment-1374830419 I can't reproduce this error locally but I have an idea what could possible wrong Potentially callback might acquire lock

[GitHub] [airflow] potiuk commented on issue #28740: airflow workers and scheduler memory leak

2023-01-08 Thread GitBox
potiuk commented on issue #28740: URL: https://github.com/apache/airflow/issues/28740#issuecomment-1374795388 > @potiuk , We upgraded to EKS cluster 1.23 but still issue persist memory leak with respect to Workers liveness probe and scheduler pods as well. Which memory is it ? Did

[GitHub] [airflow] Taragolis opened a new issue, #28790: Add missing connection types and documentation for community providers

2023-01-08 Thread GitBox
Taragolis opened a new issue, #28790: URL: https://github.com/apache/airflow/issues/28790 ### Body Some of Hooks provide connection ability, however quite a few of them do not provide any documentation and/or connection type (missing in the UI). It would be nice if we add missing

[GitHub] [airflow] Taragolis commented on pull request #28767: Remove `@pytest.mark.quarantined` from remaining quarantine tests

2023-01-08 Thread GitBox
Taragolis commented on PR #28767: URL: https://github.com/apache/airflow/pull/28767#issuecomment-1374756956 @potiuk I hope it just flaky test and not something wrong with callback mechanism. Let keep it for a while and have a look how often it failed - just for some statistic, I will