jscheffl commented on code in PR #43374:
URL: https://github.com/apache/airflow/pull/43374#discussion_r1816090044
##########
providers/src/airflow/providers/edge/cli/edge_command.py:
##########
@@ -238,14 +238,18 @@ def check_running_jobs(self) -> None:
EdgeJob.set_state(job.edge_job.key,
TaskInstanceState.FAILED)
if job.logfile.exists() and job.logfile.stat().st_size >
job.logsize:
with job.logfile.open("r") as logfile:
+ push_log_chunk_size = conf.getint("edge",
"push_log_chunk_size", fallback=1024)
Review Comment:
Fallback is not needed if the config entry is in provider YAML defined.
```suggestion
push_log_chunk_size = conf.getint("edge",
"push_log_chunk_size")
```
##########
providers/src/airflow/providers/edge/provider.yaml:
##########
@@ -102,3 +102,11 @@ config:
type: integer
example: ~
default: "60"
+ push_log_chunk_size:
+ description: |
+ Edge Worker uploads log files in chunks. If the log file part which
is uploaded exceeds the chunk size it creates a new request. This solves issue
if firewall defines max size of request.
+ This value must be defined in Bytes.
+ version_added: ~
+ type: integer
+ example: ~
+ default: "1024"
Review Comment:
Following the NGinx documentation in
https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size
the default is 1MB.
I'd propose to set the default to 512KB
##########
providers/src/airflow/providers/edge/provider.yaml:
##########
@@ -102,3 +102,11 @@ config:
type: integer
example: ~
default: "60"
+ push_log_chunk_size:
+ description: |
+ Edge Worker uploads log files in chunks. If the log file part which
is uploaded exceeds the chunk size it creates a new request. This solves issue
if firewall defines max size of request.
+ This value must be defined in Bytes.
Review Comment:
The rationale is usually not a firewall but limits on the application
gateway.
Can you point as example to nginx documentation in
https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size
as nginx is used in most K8s ingress deployments, but also give a hint that if
HTTP 413 is seen that this needs to be adjusted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]