[airflow] branch main updated (aefa482051 -> 5fad8a4a45)
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from aefa482051 Refactor serialization (#28067) add 5fad8a4a45 Unquarantine background webserver test (#28249) No new revisions were added by this update. Summary of changes: tests/cli/commands/test_webserver_command.py | 6 +- 1 file changed, 5 insertions(+), 1 deletion(-)
[GitHub] [airflow] Taragolis merged pull request #28249: Unquarantine background webserver test
Taragolis merged PR #28249: URL: https://github.com/apache/airflow/pull/28249 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] malthe commented on pull request #28244: Speed up airlfow roles list command
malthe commented on PR #28244: URL: https://github.com/apache/airflow/pull/28244#issuecomment-1343957876 This cut our runtime from 49s to 9s. But what about `airflow roles create` ... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] malthe commented on pull request #28234: Make arguments 'offset' and 'length' not required
malthe commented on PR #28234: URL: https://github.com/apache/airflow/pull/28234#issuecomment-1343930188 @potiuk seems ready to merge now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] XD-DENG commented on pull request #28047: KubernetesExecutor multi_namespace_mode can use namespace list
XD-DENG commented on PR #28047: URL: https://github.com/apache/airflow/pull/28047#issuecomment-1343915815 Hi @potiuk , actually along preparing this PR I have been using Breeze to run the tests locally, especially the `Core` as it's what this PR mainly (or only) touches. It has been running successfully and pass locally. I have tried to run it locally using Breeze again, and it still can succeed (` breeze testing tests --test-types Core --run-in-parallel --parallelism 32`) https://user-images.githubusercontent.com/11539188/206640682-bf79c97b-110b-4f04-b305-3b78edae943a.png;> But it then always fail in the CI. Any further input you can share to debug on this? Especially if any way to locate which certain test class/unit is causing the failure. Thanks a lot! @dstandish @uranusjr if you have any tips which may potentially help here, kindly let me know. Cheers! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ranjithkumar-glean commented on issue #13142: Error while attempting to disable login (setting AUTH_ROLE_PUBLIC = 'Admin')
ranjithkumar-glean commented on issue #13142: URL: https://github.com/apache/airflow/issues/13142#issuecomment-1343914677 still facing same issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] XD-DENG commented on a diff in pull request #28047: KubernetesExecutor multi_namespace_mode can use namespace list
XD-DENG commented on code in PR #28047: URL: https://github.com/apache/airflow/pull/28047#discussion_r1044096352 ## airflow/config_templates/config.yml: ## @@ -2422,11 +2422,21 @@ - name: multi_namespace_mode description: | Allows users to launch pods in multiple namespaces. -Will require creating a cluster-role for the scheduler +Will require creating a cluster-role for the scheduler, +or use multi_namespace_mode_namespace_list configuration. version_added: 1.10.12 type: boolean example: ~ default: "False" +- name: multi_namespace_mode_namespace_list + description: | +If multi_namespace_mode is True while scheduler does not have a cluster-role, +give the list of namespaces where the scheduler will schedule jobs +Scheduler needs to have the necessary permissions in these namespaces. + version_added: 2.5.1 Review Comment: Updated to 2.6.0 at https://github.com/apache/airflow/pull/28047/commits/974e34521e6d32db7f50ca810c1f602e918f8eec Continuing to check the CI failures. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] XD-DENG commented on a diff in pull request #28047: KubernetesExecutor multi_namespace_mode can use namespace list
XD-DENG commented on code in PR #28047: URL: https://github.com/apache/airflow/pull/28047#discussion_r1044093803 ## airflow/config_templates/config.yml: ## @@ -2422,11 +2422,21 @@ - name: multi_namespace_mode description: | Allows users to launch pods in multiple namespaces. -Will require creating a cluster-role for the scheduler +Will require creating a cluster-role for the scheduler, +or use multi_namespace_mode_namespace_list configuration. version_added: 1.10.12 type: boolean example: ~ default: "False" +- name: multi_namespace_mode_namespace_list + description: | +If multi_namespace_mode is True while scheduler does not have a cluster-role, +give the list of namespaces where the scheduler will schedule jobs +Scheduler needs to have the necessary permissions in these namespaces. + version_added: 2.5.1 Review Comment: Yep I agree. Will update to 2.6.0 (I just found this was missed to be updated so updating it to a "random" value and see if there is any luck it's relevant the weird CI errors. Obviously no luck) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] dstandish commented on a diff in pull request #28047: KubernetesExecutor multi_namespace_mode can use namespace list
dstandish commented on code in PR #28047: URL: https://github.com/apache/airflow/pull/28047#discussion_r1044092780 ## airflow/config_templates/config.yml: ## @@ -2422,11 +2422,21 @@ - name: multi_namespace_mode description: | Allows users to launch pods in multiple namespaces. -Will require creating a cluster-role for the scheduler +Will require creating a cluster-role for the scheduler, +or use multi_namespace_mode_namespace_list configuration. version_added: 1.10.12 type: boolean example: ~ default: "False" +- name: multi_namespace_mode_namespace_list + description: | +If multi_namespace_mode is True while scheduler does not have a cluster-role, +give the list of namespaces where the scheduler will schedule jobs +Scheduler needs to have the necessary permissions in these namespaces. + version_added: 2.5.1 Review Comment: i think it will need to go in 2.6 since new feature -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] Bowrna opened a new pull request, #28251: task docker documentation added
Bowrna opened a new pull request, #28251:
URL: https://github.com/apache/airflow/pull/28251
closes: #25926
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] uranusjr closed pull request #28250: Simplify ResolveMixin interface
uranusjr closed pull request #28250: Simplify ResolveMixin interface URL: https://github.com/apache/airflow/pull/28250 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] uranusjr opened a new pull request, #28250: Simplify ResolveMixin interface
uranusjr opened a new pull request, #28250: URL: https://github.com/apache/airflow/pull/28250 Minor refactoring. The `iter_references()` function only actually works for XComArg, and should therefore be put there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] zsdyx commented on issue #15938: celery_executor becomes stuck if child process receives signal before reset_signals is called
zsdyx commented on issue #15938:
URL: https://github.com/apache/airflow/issues/15938#issuecomment-1343758468
Provide some important logs first. After the service is start, Output after
each interval.
```
Exiting gracefully upon receiving signal 15
Sending Signals.SIGTERM to group 20. PIDs of all processes in the group:
[20]
```
ENV
GCP: k8s
python: 3.10
airflow: 2.4.1
excutor:CeleryExecutor
```
INFO,"in state
STARTED",2022-12-09T01:54:53.346995938Z
INFO,"in state
STARTED",2022-12-09T01:54:53.346992107Z
INFO,"in state
STARTED",2022-12-09T01:54:53.346986570Z
INFO,[2022-12-09T01:54:53.346+] {celery_executor.py:534} INFO - Adopted
the following 128 tasks from a dead executor,2022-12-09T01:54:53.346933893Z
INFO,[2022-12-09T01:54:52.470+] {settings.py:543} INFO - Loaded
airflow_local_settings from /opt/airflow/config/airflow_local_settings.py
.,2022-12-09T01:54:52.470955248Z
INFO,[2022-12-09T01:54:52.468+] {settings.py:58} INFO - Configured
default timezone Timezone('UTC'),2022-12-09T01:54:52.469795791Z
INFO,[2022-12-09T01:54:52.465+] {scheduler_job.py:1374} INFO - Resetting
orphaned tasks for active dag runs,2022-12-09T01:54:52.466143332Z
INFO,[2022-12-09T01:54:52.462+] {manager.py:163} INFO - Launched
DagFileProcessorManager with pid: 20,2022-12-09T01:54:52.465243117Z
INFO,[2022-12-09T01:54:52.450+] {executor_loader.py:107} INFO - Loaded
executor: CeleryExecutor,2022-12-09T01:54:52.451308381Z
INFO,[2022-12-09T01:54:52.258+] {scheduler_job.py:706} INFO - Processing
each file at most -1 times,2022-12-09T01:54:52.259176063Z
INFO,[2022-12-09T01:54:52.258+] {scheduler_job.py:701} INFO - Starting
the scheduler,2022-12-09T01:54:52.258964320Z
INFO, _/_/ |_/_/ /_//_//_/
\//|__/,2022-12-09T01:54:52.258959593Z
INFO,___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/
/,2022-12-09T01:54:52.258955844Z
INFO, /| |_ /__ ___/_ /_ __ /_ __ \_ | /| /
/,2022-12-09T01:54:52.258952230Z
INFO, |__( )_ __/__ /
__,2022-12-09T01:54:52.258946962Z
INFO, _,2022-12-09T01:54:52.258915119Z
INFO,,2022-12-09T01:54:48.293564892Z
INFO,[2022-12-09T01:54:43.752+] {scheduler_job.py:775} INFO - Exited
execute loop,2022-12-09T01:54:43.752660415Z
INFO,[2022-12-09T01:54:43.752+] {process_utils.py:98} INFO - Sending the
signal Signals.SIGTERM to process 20 as process group is
missing.,2022-12-09T01:54:43.752538440Z
INFO,[2022-12-09T01:54:43.752+] {process_utils.py:84} INFO - Sending the
signal Signals.SIGTERM to group 20,2022-12-09T01:54:43.752411066Z
INFO,[2022-12-09T01:54:43.752+] {process_utils.py:129} INFO - Sending
Signals.SIGTERM to group 20. PIDs of all processes in the group:
[],2022-12-09T01:54:43.752361863Z
INFO,"[2022-12-09T01:54:43.701+] {process_utils.py:79} INFO - Process
psutil.Process(pid=20, status='terminated', exitcode=0, started='01:49:51')
(20) terminated with exit code 0",2022-12-09T01:54:43.702040509Z
INFO,[2022-12-09T01:54:43.529+] {process_utils.py:84} INFO - Sending the
signal Signals.SIGTERM to group 20,2022-12-09T01:54:43.529318886Z
INFO,[2022-12-09T01:54:43.528+] {process_utils.py:129} INFO - Sending
Signals.SIGTERM to group 20. PIDs of all processes in the group:
[20],2022-12-09T01:54:43.529225207Z
INFO,[2022-12-09T01:54:42.526+] {scheduler_job.py:173} INFO - Exiting
gracefully upon receiving signal 15,2022-12-09T01:54:42.527221527Z
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] nlnjnj commented on issue #22657: Wrong apiVersion Detected During Ingress Creation
nlnjnj commented on issue #22657: URL: https://github.com/apache/airflow/issues/22657#issuecomment-1343728499 I came across the same problem on hwcloud -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] utkarsharma2 commented on issue #27929: AIP-51 - Single Threaded Executors
utkarsharma2 commented on issue #27929: URL: https://github.com/apache/airflow/issues/27929#issuecomment-1343725165 @o-nikolas What's the intent of `supports_ad_hoc_ti_run` is it directly related to an executor being single-threaded? if so, I can use this field. WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk opened a new pull request, #28249: Unquarantine background webserver test
potiuk opened a new pull request, #28249:
URL: https://github.com/apache/airflow/pull/28249
The test was occasionally failing because gunicorn was not immediately
launched as subprocess.
This change adds extra attempts to wait for gunicorn process and
unquarantines it.
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] potiuk opened a new pull request, #28248: Hopefully stabilize quarantined tests for celery hanging
potiuk opened a new pull request, #28248:
URL: https://github.com/apache/airflow/pull/28248
The test was running too long (> 60s) but likely this was simply not enough
with current initialization. The timeout for this test was increased while the
number of iterations were decreased.
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] potiuk opened a new pull request, #28247: Convert test_task_command to Pytest and unquarantine tests in it
potiuk opened a new pull request, #28247:
URL: https://github.com/apache/airflow/pull/28247
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #26027: import_error_alert_emails
github-actions[bot] commented on PR #26027: URL: https://github.com/apache/airflow/pull/26027#issuecomment-1343601476 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] jedcunningham opened a new pull request, #28246: Better table name validation for db clean
jedcunningham opened a new pull request, #28246:
URL: https://github.com/apache/airflow/pull/28246
We should warn users when they provide table names that are not valid, and
make it clear no cleaning happens when you provide no valid tables.
e.g.
```
$ airflow db clean -t fake,task_instance --clean-before-timestamp 2022-12-08
[2022-12-08 17:03:33,729] {db_cleanup.py:345} WARNING - The following
table(s) are not valid choices and will be skipped: {'fake'}
You have requested that we purge all data prior to 2022-12-08T00:00:00+00:00
for tables {'task_instance'}.
...
```
```
$ airflow db clean -t only,fake,tables --clean-before-timestamp 2022-12-08
[2022-12-08 17:04:20,060] {db_cleanup.py:345} WARNING - The following
table(s) are not valid choices and will be skipped: {'only', 'tables', 'fake'}
No tables selected for db cleanup. Please choose valid table names.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] pierrejeambrun commented on pull request #28245: Add markdown rendering for task notes.
pierrejeambrun commented on PR #28245: URL: https://github.com/apache/airflow/pull/28245#issuecomment-1343534612 Jest is not happy, might need to add the react-markdown module to the babel transformation. I'll take a look -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] pierrejeambrun opened a new pull request, #28245: Add markdown rendering for task notes.
pierrejeambrun opened a new pull request, #28245: URL: https://github.com/apache/airflow/pull/28245 closes: #28167. Allow rendering of markdown for the task and dagrun notes in the grid view. We use a [react-markdown](https://github.com/remarkjs/react-markdown) which is very well maintained and popular and the [remark-gfm](https://github.com/remarkjs/remark-gfm) plugin to add support for table, strikethrough, checklist and other extra tags. We add a custom components mapping to be able to render ChakraUI components. This allows us to the markdown part consistent with the rest of the app. (same components). We also have full control here of what is rendered for each element. A few improvement to consider, that can come as follow up PRs: - As we render ChakraUI component when rendering the markdown, it is so consistent with the rest of the app that it can be difficult to dissociate what is the note, and what is the actual UI. To address this I added a few style unique to the markdown part so we have some differences (especially in font size) but it is not obvious. (maybe splitting to its own tab would help, see below) - Long notes are hard to see in the details tab because there is already a lot of information, maybe separating the note in its own tab would make it easier to visualize. (And also dissociate from the rest of the UI) - During my test, it was working pretty well, and we might sometimes want to put more that 1000 characters in the note. Maybe we could increase this a bit. (A table for instance really fills up the limit quite rapidly, just wondering) - No support yet for syntax highlighting for code block. This is just a normal 'Code' component and no language specific stuff happen. There are plugins for that, we might want to take a look. - I activated the 'skipHtml' props, as I don't know yet how safe it is to allow arbitrary html to run in here or if this is working well. An examples:  Another one with a table and a code block:  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] o-nikolas commented on a diff in pull request #28161: WIP AIP-51 - Executor Coupling in Logging
o-nikolas commented on code in PR #28161:
URL: https://github.com/apache/airflow/pull/28161#discussion_r1043906313
##
airflow/executors/base_executor.py:
##
@@ -373,6 +374,62 @@ def validate_airflow_tasks_run_command(command: list[str])
-> tuple[str | None,
return dag_id, task_id
return None, None
+@staticmethod
+def _get_log_retrieval_url(ti: TaskInstance, log_relative_path: str) ->
str:
+url = urljoin(
+f"http://{ti.hostname}:{conf.get('logging',
'WORKER_LOG_SERVER_PORT')}/log/",
+log_relative_path,
+)
+return url
+
+def get_task_log(self, ti: TaskInstance, log_relative_path: str) -> str |
tuple[str, dict[str, bool]]:
Review Comment:
Perfect, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[airflow] branch main updated (2fa4dd9ea2 -> aefa482051)
This is an automated email from the ASF dual-hosted git repository.
bolke pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
from 2fa4dd9ea2 Removed unused presto integration tests (#28226)
add aefa482051 Refactor serialization (#28067)
No new revisions were added by this update.
Summary of changes:
airflow/models/param.py| 25 +-
airflow/plugins_manager.py | 21 +-
airflow/serialization/serde.py | 306 +
airflow/serialization/serialized_objects.py| 26 +-
.../auth => serialization/serializers}/__init__.py | 0
airflow/serialization/serializers/bignum.py| 55
airflow/serialization/serializers/datetime.py | 73 +
airflow/serialization/serializers/kubernetes.py| 68 +
airflow/serialization/serializers/numpy.py | 104 +++
airflow/serialization/serializers/timezone.py | 69 +
airflow/utils/json.py | 239 +++-
airflow/utils/module_loading.py| 29 +-
airflow/www/views.py | 2 +-
docs/apache-airflow/concepts/index.rst | 1 +
docs/apache-airflow/concepts/serializers.rst | 128 +
docs/apache-airflow/concepts/taskflow.rst | 6 +-
docs/apache-airflow/integration.rst| 1 +
docs/spelling_wordlist.txt | 3 +
.../serialization/serializers}/__init__.py | 0
.../serialization/serializers/test_serializers.py | 93 +++
tests/serialization/test_serde.py | 223 +++
tests/utils/test_json.py | 177 +---
22 files changed, 1259 insertions(+), 390 deletions(-)
create mode 100644 airflow/serialization/serde.py
copy airflow/{api/auth => serialization/serializers}/__init__.py (100%)
create mode 100644 airflow/serialization/serializers/bignum.py
create mode 100644 airflow/serialization/serializers/datetime.py
create mode 100644 airflow/serialization/serializers/kubernetes.py
create mode 100644 airflow/serialization/serializers/numpy.py
create mode 100644 airflow/serialization/serializers/timezone.py
create mode 100644 docs/apache-airflow/concepts/serializers.rst
copy {airflow/api/auth => tests/serialization/serializers}/__init__.py (100%)
create mode 100644 tests/serialization/serializers/test_serializers.py
create mode 100644 tests/serialization/test_serde.py
[GitHub] [airflow] grjones commented on issue #21943: Changing Image tags causes Error: UPGRADE FAILED: cannot patch "-create-user" with kind Job: Job.batch "-create-user"
grjones commented on issue #21943: URL: https://github.com/apache/airflow/issues/21943#issuecomment-1343481171 @mconigliaro Thank you kindly -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] bolkedebruin merged pull request #28067: Refactor serialization
bolkedebruin merged PR #28067: URL: https://github.com/apache/airflow/pull/28067 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #28244: Speed up airlfow role list command
potiuk commented on PR #28244: URL: https://github.com/apache/airflow/pull/28244#issuecomment-1343476035 Maybe not the "FINAL" fix but on my slow local DB: ``` time airflow roles list ``` Before: ``` name == Admin Op Public User Viewer airflow roles list 5.49s user 1.79s system 136% cpu 5.339 total ``` After: ``` name == Admin Op Public User Viewer airflow roles list 2.43s user 1.75s system 198% cpu 2.103 total ``` @malthe - can you check it on your system please ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk opened a new pull request, #28244: Speed up airlfow role list command
potiuk opened a new pull request, #28244:
URL: https://github.com/apache/airflow/pull/28244
The command initialized whole flask_app of ours, but what we really needed
was FAB security manager only.
Fixes: #28242
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] ferruzzi commented on a diff in pull request #28236: Better support for Boto Waiters
ferruzzi commented on code in PR #28236:
URL: https://github.com/apache/airflow/pull/28236#discussion_r1043895020
##
airflow/providers/amazon/aws/hooks/eks.py:
##
@@ -596,3 +597,6 @@ def fetch_access_token_for_cluster(self, eks_cluster_name:
str) -> str:
# remove any base64 encoding padding:
return "k8s-aws-v1." + base64_url.rstrip("=")
+
+def get_waiter(self, waiter_name):
+return EksBotoWaiter(client=self.conn).waiter(waiter_name)
Review Comment:
I think I got this figured out and it's even cleaner. I am also adding some
unit tests and need to update the README to match so I'll have an update maybe
tonight or tomorrow.
get_waiter now lives in
`airflow.providers.amazon.aws.hooks.base_aws.AwsGenericHook.get_waiter` and no
change needs to be made tot he service hook to implement it. :+1: The code
is a little more complicated than before, but new additions are easier so
that's a net plus.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[airflow] branch constraints-main updated: Updating constraints. Build id:
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 9069439ab1 Updating constraints. Build id: 9069439ab1 is described below commit 9069439ab1e097e21f316f65d2fd2f729aadc5a4 Author: Automated GitHub Actions commit AuthorDate: Thu Dec 8 22:46:53 2022 + Updating constraints. Build id: This update in constraints is automatically committed by the CI 'constraints-push' step based on HEAD of '' in '' with commit sha . All tests passed in this build so we determined we can push the updated constraints. See https://github.com/apache/airflow/blob/main/README.md#installing-from-pypi for details. --- constraints-3.10.txt | 10 +- constraints-3.7.txt | 10 +- constraints-3.8.txt | 10 +- constraints-3.9.txt | 10 +- constraints-no-providers-3.10.txt | 2 +- constraints-no-providers-3.7.txt | 2 +- constraints-no-providers-3.8.txt | 2 +- constraints-no-providers-3.9.txt | 2 +- constraints-source-providers-3.10.txt | 10 +- constraints-source-providers-3.7.txt | 10 +- constraints-source-providers-3.8.txt | 10 +- constraints-source-providers-3.9.txt | 10 +- 12 files changed, 44 insertions(+), 44 deletions(-) diff --git a/constraints-3.10.txt b/constraints-3.10.txt index 356a3ec474..c9f89aeb2e 100644 --- a/constraints-3.10.txt +++ b/constraints-3.10.txt @@ -1,5 +1,5 @@ # -# This constraints file was automatically generated on 2022-12-08T14:52:48Z +# This constraints file was automatically generated on 2022-12-08T22:46:35Z # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow. # This variant of constraints install uses the HEAD of the branch version for 'apache-airflow' but installs # the providers from PIP-released packages at the moment of the constraint generation. @@ -173,9 +173,9 @@ billiard==3.6.4.0 black==22.10.0 bleach==5.0.1 blinker==1.5 -boto3==1.26.25 +boto3==1.26.26 boto==2.49.0 -botocore==1.29.25 +botocore==1.29.26 bowler==0.9.0 cachelib==0.9.0 cachetools==4.2.2 @@ -553,7 +553,7 @@ sphinxcontrib-spelling==7.7.0 spython==0.3.0 sqlalchemy-bigquery==1.5.0 sqlalchemy-drill==1.1.2 -sqlalchemy-redshift==0.8.11 +sqlalchemy-redshift==0.8.12 sqlparse==0.4.3 sshpubkeys==3.3.1 sshtunnel==0.4.0 @@ -576,7 +576,7 @@ toolz==0.12.0 tornado==6.1 towncrier==22.8.0 tqdm==4.64.1 -traitlets==5.6.0 +traitlets==5.7.0 trino==0.319.0 twine==4.0.2 types-Deprecated==1.2.9 diff --git a/constraints-3.7.txt b/constraints-3.7.txt index 2555680a25..042903c0bb 100644 --- a/constraints-3.7.txt +++ b/constraints-3.7.txt @@ -1,5 +1,5 @@ # -# This constraints file was automatically generated on 2022-12-08T14:53:07Z +# This constraints file was automatically generated on 2022-12-08T22:46:51Z # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow. # This variant of constraints install uses the HEAD of the branch version for 'apache-airflow' but installs # the providers from PIP-released packages at the moment of the constraint generation. @@ -173,9 +173,9 @@ billiard==3.6.4.0 black==22.10.0 bleach==5.0.1 blinker==1.5 -boto3==1.26.25 +boto3==1.26.26 boto==2.49.0 -botocore==1.29.25 +botocore==1.29.26 bowler==0.9.0 cached-property==1.5.2 cachelib==0.9.0 @@ -554,7 +554,7 @@ sphinxcontrib-spelling==7.7.0 spython==0.3.0 sqlalchemy-bigquery==1.5.0 sqlalchemy-drill==1.1.2 -sqlalchemy-redshift==0.8.11 +sqlalchemy-redshift==0.8.12 sqlparse==0.4.3 sshpubkeys==3.3.1 sshtunnel==0.4.0 @@ -576,7 +576,7 @@ toolz==0.12.0 tornado==6.1 towncrier==22.8.0 tqdm==4.64.1 -traitlets==5.6.0 +traitlets==5.7.0 trino==0.319.0 twine==4.0.2 typed-ast==1.5.4 diff --git a/constraints-3.8.txt b/constraints-3.8.txt index 596093ebf1..d09fb9e596 100644 --- a/constraints-3.8.txt +++ b/constraints-3.8.txt @@ -1,5 +1,5 @@ # -# This constraints file was automatically generated on 2022-12-08T14:53:01Z +# This constraints file was automatically generated on 2022-12-08T22:46:46Z # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow. # This variant of constraints install uses the HEAD of the branch version for 'apache-airflow' but installs # the providers from PIP-released packages at the moment of the constraint generation. @@ -174,9 +174,9 @@ billiard==3.6.4.0 black==22.10.0 bleach==5.0.1 blinker==1.5 -boto3==1.26.25 +boto3==1.26.26 boto==2.49.0 -botocore==1.29.25 +botocore==1.29.26 bowler==0.9.0 cachelib==0.9.0 cachetools==4.2.2 @@ -556,7 +556,7 @@ sphinxcontrib-spelling==7.7.0 spython==0.3.0 sqlalchemy-bigquery==1.5.0 sqlalchemy-drill==1.1.2 -sqlalchemy-redshift==0.8.11 +sqlalchemy-redshift==0.8.12 sqlparse==0.4.3 sshpubkeys==3.3.1
[GitHub] [airflow] jedcunningham opened a new pull request, #28243: Remove RTIF from db clean
jedcunningham opened a new pull request, #28243: URL: https://github.com/apache/airflow/pull/28243 RTIF is already cleaned by TI's FK, and doesn't have a column to use as a recency column on it's own. db clean was added for 2.3, but cleaning RTIF was broken before 2.3 was released, meaning this never actually functioned anyways. In addition, if users explicitly try and clear this table with `--tables`, it gets ignored, making this a safe change to make. Closes #26581 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #28242: Airflow CLI to list roles is slow
potiuk commented on issue #28242: URL: https://github.com/apache/airflow/issues/28242#issuecomment-1343458678 Or maybe we could actually not initialize views ... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #28242: Airflow CLI to list roles is slow
potiuk commented on issue #28242: URL: https://github.com/apache/airflow/issues/28242#issuecomment-1343455451 Correction: We could likely decrease it a bit by removing connexion (REST API) intialization- it seems it contributes to > 50% of the time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #28242: Airflow CLI to list roles is slow
potiuk commented on issue #28242: URL: https://github.com/apache/airflow/issues/28242#issuecomment-1343454139 I believe there is not much we can do - when you look at the flamegraph it initializes flask and flask-app-builder. Until we replace the user model with somethiing else thanf Flass App Builder, there is not much we can do I am afraid. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] malthe commented on issue #28242: Airflow CLI to list roles is slow
malthe commented on issue #28242: URL: https://github.com/apache/airflow/issues/28242#issuecomment-1343448969 Flamegraph generated using [py-spy](https://github.com/benfred/py-spy).  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] malthe opened a new issue, #28242: Airflow CLI to list roles is slow
malthe opened a new issue, #28242: URL: https://github.com/apache/airflow/issues/28242 ### Apache Airflow version 2.5.0 ### What happened We're currently running a suboptimal setup where database connectivity is laggy, 125ms roundtrip. This has interesting consequences. For example, `airflow roles list` is really slow. Turns out that it's doing a lot of individual queries. ### What you think should happen instead Ideally, listing roles should be a single (perhaps complex) query. ### How to reproduce We're using py-spy to sample program execution: ```bash $ py-spy record -o spy.svg -i --rate 250 --nonblocking airflow roles list ``` Now, to see the bad behavior, the database should incur significant latency. ### Operating System Linux ### Versions of Apache Airflow Providers _No response_ ### Deployment Official Apache Airflow Helm Chart ### Deployment details _No response_ ### Anything else _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] benrifkind commented on issue #28206: Tasks stuck in queued state
benrifkind commented on issue #28206: URL: https://github.com/apache/airflow/issues/28206#issuecomment-1343447057 Hi @uranusjr. Thanks for your response. I believe this is an issue with the CeleryExecutor so I have not tested it with an other Executors. I checked and this doesn't seem to be an issue specifically with KubernetesPodOperator. I was able to replicate it with the BashOperator. In terms of number of DAGs and tasks. I was able to replicate this with one DAG with many tasks. I think the issue occurs when a celery worker goes down unexpectedly while it is still responsible for running tasks. So with one celery worker, running one DAG with a lot of tasks and high concurrency creates the problem. Basically the celery worker dies or is killed and once it comes back up the scheduler thinks the tasks are being run so it can't rerun them on this restarted worker. Of course I'm not sure that is what is happening but it's my best guess. I am not sure why restarting the scheduler after clearing the tasks fixes the issue. Thanks for your help. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] vincbeck opened a new pull request, #28241: Create Lambda create operator and sensor
vincbeck opened a new pull request, #28241: URL: https://github.com/apache/airflow/pull/28241 Create Lambda create operator and sensor -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #28240: Improve caching for pre-commits in CI
potiuk commented on PR #28240: URL: https://github.com/apache/airflow/pull/28240#issuecomment-1343443384 This should save ~ 3m30 s. for each static checks build. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk opened a new pull request, #28240: Improve caching for pre-commits in CI
potiuk opened a new pull request, #28240:
URL: https://github.com/apache/airflow/pull/28240
Configuration of caching for pre-commits in CI has been broken:
* full pre-commit cache had `pre-commit-` instead of `pre-commit-full-`
* basic checks never run in "main" so the cache had not been stored in the
main branch - thus pre-commits for the basic checks were never cached. However
it is quite OK for pre-commit basic to use the full pre-commit package cache
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] malthe commented on a diff in pull request #28234: Make arguments 'offset' and 'length' not required
malthe commented on code in PR #28234: URL: https://github.com/apache/airflow/pull/28234#discussion_r1043854197 ## airflow/providers/microsoft/azure/hooks/wasb.py: ## @@ -385,7 +385,7 @@ def upload( return blob_client.upload_blob(data, blob_type, length=length, **kwargs) def download( -self, container_name, blob_name, offset: int, length: int, **kwargs +self, container_name, blob_name, offset: int | None = None, length: int | None = None, **kwargs Review Comment: Fixed in d77e6815b9c586b6d8aa4cc72120727854a77d6e. I ended up trading one line for another, perhaps not the cleanest commit ever. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] mconigliaro commented on issue #27561: Helm chart tries to patch immutable Job resources on helm upgrade
mconigliaro commented on issue #27561: URL: https://github.com/apache/airflow/issues/27561#issuecomment-1343412885 I think https://github.com/apache/airflow/pull/27148 might be the fix for this. It adds `createUserJob.applyCustomEnv` and `migrateDatabaseJob.applyCustomEnv` options which we should set to `false`. We're just waiting on a new release of the helm chart. Related: https://github.com/apache/airflow/issues/21943 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ferruzzi commented on a diff in pull request #28236: Better support for Boto Waiters
ferruzzi commented on code in PR #28236: URL: https://github.com/apache/airflow/pull/28236#discussion_r1043849069 ## airflow/providers/amazon/aws/waiters/base_waiter.py: ## @@ -0,0 +1,37 @@ +# Licensed to the Apache Software Foundation (ASF) under one Review Comment: Yes? Looks like it. I've never used that one and it's not something you could inherit and use for other services. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ferruzzi commented on a diff in pull request #28236: Better support for Boto Waiters
ferruzzi commented on code in PR #28236:
URL: https://github.com/apache/airflow/pull/28236#discussion_r1043848248
##
airflow/providers/amazon/aws/hooks/eks.py:
##
@@ -596,3 +597,6 @@ def fetch_access_token_for_cluster(self, eks_cluster_name:
str) -> str:
# remove any base64 encoding padding:
return "k8s-aws-v1." + base64_url.rstrip("=")
+
+def get_waiter(self, waiter_name):
+return EksBotoWaiter(client=self.conn).waiter(waiter_name)
Review Comment:
so a basehook solution might look like
```
def get_waiter(self):
client_name = self,conn.something
config_path = Path(__file__).with_name(f"{client_name}.json").resolve()
with open(config_path) as config_file:
self.default_config = json.load(config_file)
return BaseBotoWaiter(client=client_name,
model_config=self.default_config)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] ferruzzi commented on a diff in pull request #28236: Better support for Boto Waiters
ferruzzi commented on code in PR #28236:
URL: https://github.com/apache/airflow/pull/28236#discussion_r1043844601
##
airflow/providers/amazon/aws/hooks/eks.py:
##
@@ -596,3 +597,6 @@ def fetch_access_token_for_cluster(self, eks_cluster_name:
str) -> str:
# remove any base64 encoding padding:
return "k8s-aws-v1." + base64_url.rstrip("=")
+
+def get_waiter(self, waiter_name):
+return EksBotoWaiter(client=self.conn).waiter(waiter_name)
Review Comment:
Maybe possible to work some magic if every service's custom waiter model
config is in ~/aws/waiters/{service}.json then the service name could be
extracted from the hook
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] csm10495 commented on a diff in pull request #28239: Add a new config for adapting masked secrets to make it easier to prevent secret leakage in logs
csm10495 commented on code in PR #28239: URL: https://github.com/apache/airflow/pull/28239#discussion_r1043842130 ## airflow/config_templates/config.yml: ## @@ -713,6 +713,17 @@ type: string example: ~ default: "airflow.utils.log.timezone_aware.TimezoneAware" +- name: secret_mask_adapter + description: | +An import path to a function to add adaptations of each secret added with +`airflow.utils.log.secrets_masker.mask_secret` to the log message. The given function +is expected to require a single parameter: the secret to be adapted. It may return a +single adaptation of the secret or an iterable of adaptations to each be masked as secrets. +The original secret will be masked as well as any adaptations returned. + version_added: 2.5.0 Review Comment: Dang. I'm behind the times. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #28239: Add a new config for adapting masked secrets to make it easier to prevent secret leakage in logs
potiuk commented on code in PR #28239: URL: https://github.com/apache/airflow/pull/28239#discussion_r1043841553 ## airflow/config_templates/config.yml: ## @@ -713,6 +713,17 @@ type: string example: ~ default: "airflow.utils.log.timezone_aware.TimezoneAware" +- name: secret_mask_adapter + description: | +An import path to a function to add adaptations of each secret added with +`airflow.utils.log.secrets_masker.mask_secret` to the log message. The given function +is expected to require a single parameter: the secret to be adapted. It may return a +single adaptation of the secret or an iterable of adaptations to each be masked as secrets. +The original secret will be masked as well as any adaptations returned. + version_added: 2.5.0 Review Comment: ```suggestion version_added: 2.6.0 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] dstandish commented on a diff in pull request #28238: Run kubernetes executor tasks in interactive mode for live logs
dstandish commented on code in PR #28238: URL: https://github.com/apache/airflow/pull/28238#discussion_r1043841491 ## airflow/executors/base_executor.py: ## @@ -128,6 +128,7 @@ def queue_task_instance( pool=pool, pickle_id=pickle_id, cfg_path=cfg_path, +interactive=self.__class__.__name__ == "KubernetesExecutor", Review Comment: or... "requires interactive" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #28236: Better support for Boto Waiters
Taragolis commented on code in PR #28236:
URL: https://github.com/apache/airflow/pull/28236#discussion_r1043841332
##
airflow/providers/amazon/aws/hooks/eks.py:
##
@@ -596,3 +597,6 @@ def fetch_access_token_for_cluster(self, eks_cluster_name:
str) -> str:
# remove any base64 encoding padding:
return "k8s-aws-v1." + base64_url.rstrip("=")
+
+def get_waiter(self, waiter_name):
+return EksBotoWaiter(client=self.conn).waiter(waiter_name)
Review Comment:
On the other hand a lot of boto3 clients doesn't have waiters so it might be
add additional complexity if it included in base hook
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #28234: Make arguments 'offset' and 'length' not required
potiuk commented on code in PR #28234: URL: https://github.com/apache/airflow/pull/28234#discussion_r1043839483 ## airflow/providers/microsoft/azure/hooks/wasb.py: ## @@ -385,7 +385,7 @@ def upload( return blob_client.upload_blob(data, blob_type, length=length, **kwargs) def download( -self, container_name, blob_name, offset: int, length: int, **kwargs +self, container_name, blob_name, offset: int | None = None, length: int | None = None, **kwargs Review Comment: (and yes we can complain after we do). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] csm10495 opened a new pull request, #28239: Add a new config for adapting masked secrets to make it easier to prevent secret leakage in logs
csm10495 opened a new pull request, #28239: URL: https://github.com/apache/airflow/pull/28239 We've hit some cases where secrets accidentally leak because they take slightly different forms than expected in logs. One of example of this was a password was urlquoted then logged as part of a url. Using this new config, we can specify that we want to mask secrets and adaptations of them (like urlquoted) versions. A custom function can also be written that returns a list of adaptations to filter in logs as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #28234: Make arguments 'offset' and 'length' not required
potiuk commented on code in PR #28234: URL: https://github.com/apache/airflow/pull/28234#discussion_r1043839051 ## airflow/providers/microsoft/azure/hooks/wasb.py: ## @@ -385,7 +385,7 @@ def upload( return blob_client.upload_blob(data, blob_type, length=length, **kwargs) def download( -self, container_name, blob_name, offset: int, length: int, **kwargs +self, container_name, blob_name, offset: int | None = None, length: int | None = None, **kwargs Review Comment: Unfortunately, none of us can be made responsible for clarity of docs that we have little control over, but we have control on verifying our assumptions in case they are unclear. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #28236: Better support for Boto Waiters
Taragolis commented on code in PR #28236:
URL: https://github.com/apache/airflow/pull/28236#discussion_r1043838145
##
airflow/providers/amazon/aws/hooks/eks.py:
##
@@ -596,3 +597,6 @@ def fetch_access_token_for_cluster(self, eks_cluster_name:
str) -> str:
# remove any base64 encoding padding:
return "k8s-aws-v1." + base64_url.rstrip("=")
+
+def get_waiter(self, waiter_name):
+return EksBotoWaiter(client=self.conn).waiter(waiter_name)
Review Comment:
It just an idea.
It could be some optional argument of hook and if it None than use
predefined model for hook (e.g. from waiters directory) if defined than use
this model.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #28236: Better support for Boto Waiters
Taragolis commented on code in PR #28236:
URL: https://github.com/apache/airflow/pull/28236#discussion_r1043838145
##
airflow/providers/amazon/aws/hooks/eks.py:
##
@@ -596,3 +597,6 @@ def fetch_access_token_for_cluster(self, eks_cluster_name:
str) -> str:
# remove any base64 encoding padding:
return "k8s-aws-v1." + base64_url.rstrip("=")
+
+def get_waiter(self, waiter_name):
+return EksBotoWaiter(client=self.conn).waiter(waiter_name)
Review Comment:
It just an idea.
It could be some parameter of hook and if it None than use predefined model
for hook (e.g. from waiters directory).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #28234: Make arguments 'offset' and 'length' not required
potiuk commented on code in PR #28234: URL: https://github.com/apache/airflow/pull/28234#discussion_r1043837071 ## airflow/providers/microsoft/azure/hooks/wasb.py: ## @@ -385,7 +385,7 @@ def upload( return blob_client.upload_blob(data, blob_type, length=length, **kwargs) def download( -self, container_name, blob_name, offset: int, length: int, **kwargs +self, container_name, blob_name, offset: int | None = None, length: int | None = None, **kwargs Review Comment: Try to remove and see if it works with `--all-files` :) If it's not clear, just testing hypothesis is a good idea. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] malthe commented on a diff in pull request #28234: Make arguments 'offset' and 'length' not required
malthe commented on code in PR #28234: URL: https://github.com/apache/airflow/pull/28234#discussion_r1043836075 ## airflow/providers/microsoft/azure/hooks/wasb.py: ## @@ -385,7 +385,7 @@ def upload( return blob_client.upload_blob(data, blob_type, length=length, **kwargs) def download( -self, container_name, blob_name, offset: int, length: int, **kwargs +self, container_name, blob_name, offset: int | None = None, length: int | None = None, **kwargs Review Comment: From the docs it seems like a mistake to have the same rule for both "azure.batch" and "azure.batch.models". I am not totally sure what ".*" adds to the picture. The docs aren't really clear (to me at least). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (bdc3d2e647 -> 2fa4dd9ea2)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from bdc3d2e647 Keep pod name for k8s executor under 63 characters (#28237) add 2fa4dd9ea2 Removed unused presto integration tests (#28226) No new revisions were added by this update. Summary of changes: .../google/cloud/transfers/test_presto_to_gcs.py | 332 - .../cloud/transfers/test_presto_to_gcs_system.py | 160 -- tests/providers/common/sql/sensors/test_sql.py | 13 - .../mysql/transfers/test_presto_to_mysql.py| 20 -- 4 files changed, 525 deletions(-) delete mode 100644 tests/integration/providers/google/cloud/transfers/test_presto_to_gcs.py delete mode 100644 tests/integration/providers/google/cloud/transfers/test_presto_to_gcs_system.py
[GitHub] [airflow] potiuk merged pull request #28226: Removed unused presto integration tests
potiuk merged PR #28226: URL: https://github.com/apache/airflow/pull/28226 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #28234: Make arguments 'offset' and 'length' not required
potiuk commented on code in PR #28234: URL: https://github.com/apache/airflow/pull/28234#discussion_r1043833245 ## airflow/providers/microsoft/azure/hooks/wasb.py: ## @@ -385,7 +385,7 @@ def upload( return blob_client.upload_blob(data, blob_type, length=length, **kwargs) def download( -self, container_name, blob_name, offset: int, length: int, **kwargs +self, container_name, blob_name, offset: int | None = None, length: int | None = None, **kwargs Review Comment: > This may be the solution for other azure packages #27199 Likely yes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #28234: Make arguments 'offset' and 'length' not required
potiuk commented on code in PR #28234: URL: https://github.com/apache/airflow/pull/28234#discussion_r1043832758 ## airflow/providers/microsoft/azure/hooks/wasb.py: ## @@ -385,7 +385,7 @@ def upload( return blob_client.upload_blob(data, blob_type, length=length, **kwargs) def download( -self, container_name, blob_name, offset: int, length: int, **kwargs +self, container_name, blob_name, offset: int | None = None, length: int | None = None, **kwargs Review Comment: But ultimately what I usually do is to add what I think works and check (run mypy static checks in this case :) ) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #28234: Make arguments 'offset' and 'length' not required
potiuk commented on code in PR #28234: URL: https://github.com/apache/airflow/pull/28234#discussion_r1043831892 ## airflow/providers/microsoft/azure/hooks/wasb.py: ## @@ -385,7 +385,7 @@ def upload( return blob_client.upload_blob(data, blob_type, length=length, **kwargs) def download( -self, container_name, blob_name, offset: int, length: int, **kwargs +self, container_name, blob_name, offset: int | None = None, length: int | None = None, **kwargs Review Comment: The docs say: > Some of the config options may be set either globally (in the [mypy] section) or on a per-module basis (in sections like [mypy-foo.bar]). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #28234: Make arguments 'offset' and 'length' not required
potiuk commented on code in PR #28234: URL: https://github.com/apache/airflow/pull/28234#discussion_r1043831399 ## airflow/providers/microsoft/azure/hooks/wasb.py: ## @@ -385,7 +385,7 @@ def upload( return blob_client.upload_blob(data, blob_type, length=length, **kwargs) def download( -self, container_name, blob_name, offset: int, length: int, **kwargs +self, container_name, blob_name, offset: int | None = None, length: int | None = None, **kwargs Review Comment: I think it's best to consult the docs to be sure: https://mypy.readthedocs.io/en/stable/config_file.html#examples . I can't remember all nuances, but I beleve this is the package whic azure storage provides. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ferruzzi commented on a diff in pull request #28236: Better support for Boto Waiters
ferruzzi commented on code in PR #28236:
URL: https://github.com/apache/airflow/pull/28236#discussion_r1043831359
##
airflow/providers/amazon/aws/hooks/eks.py:
##
@@ -596,3 +597,6 @@ def fetch_access_token_for_cluster(self, eks_cluster_name:
str) -> str:
# remove any base64 encoding padding:
return "k8s-aws-v1." + base64_url.rstrip("=")
+
+def get_waiter(self, waiter_name):
+return EksBotoWaiter(client=self.conn).waiter(waiter_name)
Review Comment:
Maybe, but then you'd have to pass it the waiter model every time too, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] kaxil commented on a diff in pull request #28238: Run kubernetes executor tasks in interactive mode for live logs
kaxil commented on code in PR #28238: URL: https://github.com/apache/airflow/pull/28238#discussion_r1043831334 ## airflow/executors/base_executor.py: ## @@ -128,6 +128,7 @@ def queue_task_instance( pool=pool, pickle_id=pickle_id, cfg_path=cfg_path, +interactive=self.__class__.__name__ == "KubernetesExecutor", Review Comment: +1 to @blag 's suggestion -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] blag commented on a diff in pull request #28238: Run kubernetes executor tasks in interactive mode for live logs
blag commented on code in PR #28238: URL: https://github.com/apache/airflow/pull/28238#discussion_r1043828568 ## airflow/executors/base_executor.py: ## @@ -128,6 +128,7 @@ def queue_task_instance( pool=pool, pickle_id=pickle_id, cfg_path=cfg_path, +interactive=self.__class__.__name__ == "KubernetesExecutor", Review Comment: Given [AIP-51](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-51+Removing+Executor+Coupling+from+Core+Airflow), can we tweak how this check is implemented? Maybe have a `supports_interactive` attribute on BaseExecutor? It just seems weird to require a base class to have knowledge about subclasses. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (cbfbf8b843 -> bdc3d2e647)
This is an automated email from the ASF dual-hosted git repository. dstandish pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from cbfbf8b843 Make live logs reading work for "other" k8s executors (#28213) add bdc3d2e647 Keep pod name for k8s executor under 63 characters (#28237) No new revisions were added by this update. Summary of changes: airflow/kubernetes/kubernetes_helper_functions.py | 7 ++- airflow/kubernetes/pod_generator.py| 9 + .../providers/cncf/kubernetes/operators/kubernetes_pod.py | 6 -- airflow/utils/log/file_task_handler.py | 14 -- tests/kubernetes/test_kubernetes_helper_functions.py | 6 -- 5 files changed, 23 insertions(+), 19 deletions(-)
[GitHub] [airflow] dstandish merged pull request #28237: Keep pod name for k8s executor under 63 characters
dstandish merged PR #28237: URL: https://github.com/apache/airflow/pull/28237 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] IAL32 commented on pull request #28187: Add IAM authentication to Amazon Redshift Connection by AWS Connection
IAL32 commented on PR #28187: URL: https://github.com/apache/airflow/pull/28187#issuecomment-1343343884 I would also like some feedback from someone with more experience with RedShift. I don't use it myself very often either -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] dstandish opened a new pull request, #28238: Run kubernetes executor tasks in interactive mode for live logs
dstandish opened a new pull request, #28238: URL: https://github.com/apache/airflow/pull/28238 For k8s executor, task log handler reads from pod logs while task is running. By running in "interactive" mode we avoid redirecting stdout, which keeps the logs visible when reading from pod. Otherwise, we can't get any logs til task is complete and logs are uploaded to cloud storage (assuming user enables remote logging). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #28236: Better support for Boto Waiters
Taragolis commented on code in PR #28236:
URL: https://github.com/apache/airflow/pull/28236#discussion_r1043809144
##
airflow/providers/amazon/aws/waiters/base_waiter.py:
##
@@ -0,0 +1,37 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from botocore.waiter import Waiter, WaiterModel, create_waiter_with_client
+
+from airflow.providers.amazon.aws.hooks.base_aws import BaseAwsConnection
+
+
+class BaseBotoWaiter:
+"""
+Used to create custom Boto3 Waiters.
+
+For more details, see airflow/providers/amazon/aws/waiters/README.md
+"""
+
+def __init__(self, client: BaseAwsConnection, model_config: dict) -> None:
Review Comment:
`BaseAwsConnection` use mostly for type annotations (which not fully
implemented yet) and it is union type of `boro3.client` and `boto3.resource`
`boto3.client` is subclass of `botocore.client.BaseClient` (plus some magic
and factories) and waiters if it exists is a part of this client. However
`boto3.resource` as far as I know doesn't have waiters access directly but you
always could get actual client of resource by call property
`resource.meta.client`.
Sooo.. I think annotations as `boto3.client` would be enough.
##
airflow/providers/amazon/aws/hooks/eks.py:
##
@@ -596,3 +597,6 @@ def fetch_access_token_for_cluster(self, eks_cluster_name:
str) -> str:
# remove any base64 encoding padding:
return "k8s-aws-v1." + base64_url.rstrip("=")
+
+def get_waiter(self, waiter_name):
+return EksBotoWaiter(client=self.conn).waiter(waiter_name)
Review Comment:
Just a thought: "Is it possible to make it part of base AWS hook?"
##
airflow/providers/amazon/aws/waiters/base_waiter.py:
##
@@ -0,0 +1,37 @@
+# Licensed to the Apache Software Foundation (ASF) under one
Review Comment:
I thought BatchClient already implements some kind of waiters (I personally
never use it in this hook)
- `airflow/providers/amazon/aws/hooks/batch_waiters.json`
- `airflow/providers/amazon/aws/hooks/batch_waiters.py`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] malthe commented on a diff in pull request #28234: Make arguments 'offset' and 'length' not required
malthe commented on code in PR #28234: URL: https://github.com/apache/airflow/pull/28234#discussion_r1043810079 ## airflow/providers/microsoft/azure/hooks/wasb.py: ## @@ -385,7 +385,7 @@ def upload( return blob_client.upload_blob(data, blob_type, length=length, **kwargs) def download( -self, container_name, blob_name, offset: int, length: int, **kwargs +self, container_name, blob_name, offset: int | None = None, length: int | None = None, **kwargs Review Comment: What's the deal with those exclusions: ``` [mypy-azure.batch.*] no_implicit_optional = False [mypy-azure.batch.models.*] no_implicit_optional = False ``` Does it pattern-match on the module where the problem is? I am not totally sure what to put here for azure-storage, @potiuk. ```cfg [mypy-azure.storage.*] no_implicit_optional = False ``` ^^^ something like that? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] mconigliaro commented on issue #21943: Changing Image tags causes Error: UPGRADE FAILED: cannot patch "-create-user" with kind Job: Job.batch "-create-user"
mconigliaro commented on issue #21943: URL: https://github.com/apache/airflow/issues/21943#issuecomment-1343331982 I think https://github.com/apache/airflow/pull/27148 might be the fix for this. It adds a `createUserJob.applyCustomEnv` option which we should probably set to `false`. We're just waiting on a new release of the helm chart. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] eladkal commented on a diff in pull request #28234: Make arguments 'offset' and 'length' not required
eladkal commented on code in PR #28234: URL: https://github.com/apache/airflow/pull/28234#discussion_r1043804562 ## airflow/providers/microsoft/azure/hooks/wasb.py: ## @@ -385,7 +385,7 @@ def upload( return blob_client.upload_blob(data, blob_type, length=length, **kwargs) def download( -self, container_name, blob_name, offset: int, length: int, **kwargs +self, container_name, blob_name, offset: int | None = None, length: int | None = None, **kwargs Review Comment: This may be the solution for other azure packages https://github.com/apache/airflow/pull/27199 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #28234: Make arguments 'offset' and 'length' not required
potiuk commented on code in PR #28234: URL: https://github.com/apache/airflow/pull/28234#discussion_r1043801723 ## airflow/providers/microsoft/azure/hooks/wasb.py: ## @@ -385,7 +385,7 @@ def upload( return blob_client.upload_blob(data, blob_type, length=length, **kwargs) def download( -self, container_name, blob_name, offset: int, length: int, **kwargs +self, container_name, blob_name, offset: int | None = None, length: int | None = None, **kwargs Review Comment: Still @malthe you will need to add azure-storage to setup.cfg -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] dstandish opened a new pull request, #28237: Keep pod name for k8s executor under 63 characters
dstandish opened a new pull request, #28237: URL: https://github.com/apache/airflow/pull/28237 Because of the way that the task log handler reads from running k8s executor pods, we must keep pod name <= 63 characters. The handler gets pod name from ti.hostname. TI hostname is derived from the container hostname, which is truncated to 63 characters. We could lift this limit by using label selectors instead of pod name to find the pod. But for now, easy enough to keep limited to 63. Since we limit to 63 in the code, we can remove the logic to find the matching pod when length is >= 63. --- Note: Recently I made changes that allowed for longer pod name, and take up less space on the random part, and allow hyphens. At the time it wasn't clear why the pod id length had to be limited. But I think the reason is this task log issue. So we restore the length limit here, while still retaining the other improvements. And we make clear in the place where we limit what is the reason for the limit. And we remove the code to handle longer limit since it's not needed. For KPO, we can keep the longer name since hostname is not used for pod id. (there we use labels) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (1eaedc8ae8 -> cbfbf8b843)
This is an automated email from the ASF dual-hosted git repository. jedcunningham pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 1eaedc8ae8 Improve docstrings for `AwsLambdaInvokeFunctionOperator` (#28233) add cbfbf8b843 Make live logs reading work for "other" k8s executors (#28213) No new revisions were added by this update. Summary of changes: airflow/utils/log/file_task_handler.py | 21 ++-- tests/utils/test_log_handlers.py | 36 ++ 2 files changed, 55 insertions(+), 2 deletions(-)
[GitHub] [airflow] jedcunningham merged pull request #28213: Make live logs reading work for "other" k8s executors
jedcunningham merged PR #28213: URL: https://github.com/apache/airflow/pull/28213 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] eladkal commented on a diff in pull request #28234: Make arguments 'offset' and 'length' not required
eladkal commented on code in PR #28234: URL: https://github.com/apache/airflow/pull/28234#discussion_r1043792417 ## airflow/providers/microsoft/azure/hooks/wasb.py: ## @@ -385,7 +385,7 @@ def upload( return blob_client.upload_blob(data, blob_type, length=length, **kwargs) def download( -self, container_name, blob_name, offset: int, length: int, **kwargs +self, container_name, blob_name, offset: int | None = None, length: int | None = None, **kwargs Review Comment: ah! I didn't know about this setting -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] dstandish closed pull request #28215: WIP shorten k8s exec pod id for task logging reasons
dstandish closed pull request #28215: WIP shorten k8s exec pod id for task logging reasons URL: https://github.com/apache/airflow/pull/28215 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] dstandish commented on pull request #28215: WIP shorten k8s exec pod id for task logging reasons
dstandish commented on PR #28215: URL: https://github.com/apache/airflow/pull/28215#issuecomment-1343306885 closing for now -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #28234: Make arguments 'offset' and 'length' not required
potiuk commented on code in PR #28234: URL: https://github.com/apache/airflow/pull/28234#discussion_r1043789573 ## airflow/providers/microsoft/azure/hooks/wasb.py: ## @@ -385,7 +385,7 @@ def upload( return blob_client.upload_blob(data, blob_type, length=length, **kwargs) def download( -self, container_name, blob_name, offset: int, length: int, **kwargs +self, container_name, blob_name, offset: int | None = None, length: int | None = None, **kwargs Review Comment: I am afraid #25426 was a mistake. They simply added type-hints but left None as default. They simply use "implict optional" and we don't use it. Implicit optional is a setting we have to add in https://github.com/apache/airflow/blob/1eaedc8ae85dbbfaaa402f9936304be51de98f88/setup.cfg#L191 to add azure-storage to list of other azure libraries. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #28234: Make arguments 'offset' and 'length' not required
potiuk commented on code in PR #28234: URL: https://github.com/apache/airflow/pull/28234#discussion_r1043789573 ## airflow/providers/microsoft/azure/hooks/wasb.py: ## @@ -385,7 +385,7 @@ def upload( return blob_client.upload_blob(data, blob_type, length=length, **kwargs) def download( -self, container_name, blob_name, offset: int, length: int, **kwargs +self, container_name, blob_name, offset: int | None = None, length: int | None = None, **kwargs Review Comment: I am afraid #25426 was a mistake. They simply added type-hints but left None as default. They simply use "implict optional" and we don't use it. Implicit optional is a setting in https://github.com/apache/airflow/blob/1eaedc8ae85dbbfaaa402f9936304be51de98f88/setup.cfg#L191 to add azure-storage to list of other azure libraries. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (3f06efa5ca -> 1eaedc8ae8)
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 3f06efa5ca Add webserverConfigConfigMapName (#27419) add 1eaedc8ae8 Improve docstrings for `AwsLambdaInvokeFunctionOperator` (#28233) No new revisions were added by this update. Summary of changes: airflow/providers/amazon/aws/hooks/lambda_function.py | 6 +- airflow/providers/amazon/aws/operators/lambda_function.py | 5 - 2 files changed, 9 insertions(+), 2 deletions(-)
[GitHub] [airflow] eladkal merged pull request #28233: Improve docstring for `AwsLambdaInvokeFunctionOperator`
eladkal merged PR #28233: URL: https://github.com/apache/airflow/pull/28233 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ferruzzi opened a new pull request, #28236: Better support for Boto Waiters
ferruzzi opened a new pull request, #28236: URL: https://github.com/apache/airflow/pull/28236 This PR cleans up a bit of tech debt in the EKS operators. When I wrote them I didn't know what boto waiters were. A Boto waiter is similar to an Ariflow Sensor but the delay, backoff, and max-retries is all handled server-side by the service. I've replaced a bunch of my custom waiter code with official boto waiters, added the infrastructure to allow us to write custom waiters, along with documentation and an example. This should make future AWS sensors much easier and cleaner to write. We may want to go through the existing sensors and convert them over at some point. This also opens up an interesting idea of using a custom waiter in a TaskFlow method in place of a sensor. Who knows. Rendered version of the README is [here](https://github.com/aws-mwaa/upstream-to-airflow/blob/a5ff1624eae1570ad9b885fdbc8ec4c1a3b65ea7/airflow/providers/amazon/aws/waiters/README.md) for easier review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] eladkal commented on a diff in pull request #28234: Make arguments 'offset' and 'length' not required
eladkal commented on code in PR #28234: URL: https://github.com/apache/airflow/pull/28234#discussion_r1043784072 ## airflow/providers/microsoft/azure/hooks/wasb.py: ## @@ -385,7 +385,7 @@ def upload( return blob_client.upload_blob(data, blob_type, length=length, **kwargs) def download( -self, container_name, blob_name, offset: int, length: int, **kwargs +self, container_name, blob_name, offset: int | None = None, length: int | None = None, **kwargs Review Comment: This is what we had before https://github.com/apache/airflow/pull/25426 we needed to make the change to update to newer sdk version see https://github.com/apache/airflow/pull/25426#issuecomment-1233152203 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #28235: Add table_resource to template fields for BigQueryCreateEmptyTableOperator
boring-cyborg[bot] commented on PR #28235: URL: https://github.com/apache/airflow/pull/28235#issuecomment-1343283911 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst) Here are some useful points: - Pay attention to the quality of your code (flake8, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that. - In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it. - Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations. - Be patient and persistent. It might take some time to get a review or get the final approval from Committers. - Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack. - Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices). Apache Airflow is a community-driven project and together we are making it better . In case of doubts contact the developers at: Mailing List: d...@airflow.apache.org Slack: https://s.apache.org/airflow-slack -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] jbandoro opened a new pull request, #28235: Add table_resource to template fields for BigQueryCreateEmptyTableOperator
jbandoro opened a new pull request, #28235:
URL: https://github.com/apache/airflow/pull/28235
Allows `table_resource` to be templated since if it is provided, all other
parameters in the operator are ignored.
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] dstandish commented on a diff in pull request #28213: Make live logs reading work for "other" k8s executors
dstandish commented on code in PR #28213:
URL: https://github.com/apache/airflow/pull/28213#discussion_r1043760406
##
airflow/utils/log/file_task_handler.py:
##
@@ -173,7 +190,8 @@ def _read(self, ti: TaskInstance, try_number: int,
metadata: dict[str, Any] | No
log = f"*** Failed to load local log file: {location}\n"
log += f"*** {str(e)}\n"
return log, {"end_of_log": True}
-elif conf.get("core", "executor") == "KubernetesExecutor":
+
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] dstandish commented on a diff in pull request #28213: Make live logs reading work for "other" k8s executors
dstandish commented on code in PR #28213: URL: https://github.com/apache/airflow/pull/28213#discussion_r1043758356 ## airflow/utils/log/file_task_handler.py: ## @@ -132,6 +132,19 @@ def _render_filename(self, ti: TaskInstance, try_number: int) -> str: def _read_grouped_logs(self): return False +@staticmethod +def _try_read_from_k8s(queue): Review Comment: ok @uranusjr ready for another look... and added test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated: Add webserverConfigConfigMapName (#27419)
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 3f06efa5ca Add webserverConfigConfigMapName (#27419)
3f06efa5ca is described below
commit 3f06efa5ca2d061c5a2df7b08d53e09a552bdd72
Author: JulesTriomphe <35426973+julestriom...@users.noreply.github.com>
AuthorDate: Thu Dec 8 20:37:15 2022 +0100
Add webserverConfigConfigMapName (#27419)
---
chart/templates/_helpers.yaml | 11 +++
.../templates/configmaps/webserver-configmap.yaml | 4 +--
.../templates/scheduler/scheduler-deployment.yaml | 14 +
.../templates/triggerer/triggerer-deployment.yaml | 11 +++
.../templates/webserver/webserver-deployment.yaml | 14 -
chart/templates/workers/worker-deployment.yaml | 17 ++
chart/values.schema.json | 14 -
chart/values.yaml | 7 +++--
tests/charts/test_airflow_common.py| 36 ++
tests/charts/test_webserver.py | 12
10 files changed, 128 insertions(+), 12 deletions(-)
diff --git a/chart/templates/_helpers.yaml b/chart/templates/_helpers.yaml
index f3cafc0373..074ecc7d49 100644
--- a/chart/templates/_helpers.yaml
+++ b/chart/templates/_helpers.yaml
@@ -443,6 +443,17 @@ server_tls_key_file = /etc/pgbouncer/server.key
{{ (printf "%s/webserver_config.py" .Values.airflowHome) | quote }}
{{- end }}
+{{ define "airflow_webserver_config_configmap_name" -}}
+{{ default (printf "%s-webserver-config" .Release.Name)
.Values.webserver.webserverConfigConfigMapName }}
+{{- end }}
+
+{{ define "airflow_webserver_config_mount" -}}
+- name: webserver-config
+ mountPath: {{ template "airflow_webserver_config_path" . }}
+ subPath: webserver_config.py
+ readOnly: True
+{{- end -}}
+
{{ define "airflow_local_setting_path" -}}
{{ (printf "%s/config/airflow_local_settings.py" .Values.airflowHome) | quote
}}
{{- end }}
diff --git a/chart/templates/configmaps/webserver-configmap.yaml
b/chart/templates/configmaps/webserver-configmap.yaml
index e8d5ca016e..26fb5094d9 100644
--- a/chart/templates/configmaps/webserver-configmap.yaml
+++ b/chart/templates/configmaps/webserver-configmap.yaml
@@ -18,11 +18,11 @@
## Airflow ConfigMap
#
-{{- if .Values.webserver.webserverConfig }}
+{{- if and .Values.webserver.webserverConfig (not
.Values.webserver.webserverConfigConfigMapName) }}
apiVersion: v1
kind: ConfigMap
metadata:
- name: {{ .Release.Name }}-webserver-config
+ name: {{ template "airflow_webserver_config_configmap_name" . }}
labels:
tier: airflow
component: config
diff --git a/chart/templates/scheduler/scheduler-deployment.yaml
b/chart/templates/scheduler/scheduler-deployment.yaml
index 89e5ab3d8c..9f8411c716 100644
--- a/chart/templates/scheduler/scheduler-deployment.yaml
+++ b/chart/templates/scheduler/scheduler-deployment.yaml
@@ -139,6 +139,9 @@ spec:
{{- include "airflow_config_mount" . | nindent 12 }}
{{- if .Values.scheduler.extraVolumeMounts }}
{{ toYaml .Values.scheduler.extraVolumeMounts | indent 12 }}
+{{- end }}
+{{- if or .Values.webserver.webserverConfig
.Values.webserver.webserverConfigConfigMapName }}
+{{ include "airflow_webserver_config_mount" . | indent 12 }}
{{- end }}
args:
{{- include "wait-for-migrations-command" . | indent 10 }}
@@ -204,6 +207,9 @@ spec:
- name: logs
mountPath: {{ template "airflow_logs" . }}
{{- include "airflow_config_mount" . | nindent 12 }}
+{{- if or .Values.webserver.webserverConfig
.Values.webserver.webserverConfigConfigMapName }}
+{{ include "airflow_webserver_config_mount" . | indent 12 }}
+{{- end }}
{{- if and $localOrDagProcessorDisabled (or .Values.dags.persistence.enabled
.Values.dags.gitSync.enabled) }}
{{- include "airflow_dags_mount" . | nindent 12 }}
{{- end }}
@@ -235,6 +241,9 @@ spec:
mountPath: {{ template "airflow_logs" . }}
{{- if .Values.scheduler.extraVolumeMounts }}
{{ toYaml .Values.scheduler.extraVolumeMounts | indent 12 }}
+{{- end }}
+{{- if or .Values.webserver.webserverConfig
.Values.webserver.webserverConfigConfigMapName }}
+{{ include "airflow_webserver_config_mount" . | indent 12 }}
{{- end }}
{{- end }}
{{- if .Values.scheduler.extraContainers }}
@@ -244,6 +253,11 @@ spec:
- name: config
configMap:
name: {{ template "airflow_config" . }}
+{{- if or .Values.webserver.webserverConfig
.Values.webserver.webserverConfigConfigMapName }}
+- name: webserver-config
+ configMap:
+name: {{ template "airflow_webserver_config_configmap_name" . }}
+{{- end }}
{{- if $localOrDagProcessorDisabled }}
{{- if
[GitHub] [airflow] jedcunningham merged pull request #27419: Add webserverConfigConfigMapName
jedcunningham merged PR #27419: URL: https://github.com/apache/airflow/pull/27419 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] malthe opened a new pull request, #28234: Make arguments 'offset' and 'length' not required
malthe opened a new pull request, #28234: URL: https://github.com/apache/airflow/pull/28234 In https://github.com/apache/airflow/pull/25426, the `offset` and `length` arguments were made required. But this is not in line with the Azure blob client interface. The `length` argument is a hint to optimize performance for larger reads, while `offset` is used only for more advanced reading. We're currently using neither for log reading. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] KulykDmytro commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
KulykDmytro commented on issue #23727: URL: https://github.com/apache/airflow/issues/23727#issuecomment-1343251798 in 2.5.0 #28227 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] KulykDmytro commented on issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack
KulykDmytro commented on issue #27593: URL: https://github.com/apache/airflow/issues/27593#issuecomment-1343250768 linking #28227 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] eladkal commented on pull request #28052: Fix EmrAddStepsOperature wait_for_completion parameter is not working
eladkal commented on PR #28052: URL: https://github.com/apache/airflow/pull/28052#issuecomment-1343241181 Rebased the PR as it was too far behind main. will merge once CI is green -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] eladkal commented on pull request #28192: Fix AWS EMR EmrAddStepsOperator to wait_for_completion parameter
eladkal commented on PR #28192: URL: https://github.com/apache/airflow/pull/28192#issuecomment-1343229119 closing in favor of https://github.com/apache/airflow/pull/28052 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] eladkal closed pull request #28192: Fix AWS EMR EmrAddStepsOperator to wait_for_completion parameter
eladkal closed pull request #28192: Fix AWS EMR EmrAddStepsOperator to wait_for_completion parameter URL: https://github.com/apache/airflow/pull/28192 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] eladkal opened a new pull request, #28233: Improve docstring for `AwsLambdaInvokeFunctionOperator`
eladkal opened a new pull request, #28233:
URL: https://github.com/apache/airflow/pull/28233
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #28232: Allow changing the Task comments via a task
boring-cyborg[bot] commented on issue #28232: URL: https://github.com/apache/airflow/issues/28232#issuecomment-1343204264 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
