jscheffl commented on code in PR #43374:
URL: https://github.com/apache/airflow/pull/43374#discussion_r1816090044


##########
providers/src/airflow/providers/edge/cli/edge_command.py:
##########
@@ -238,14 +238,18 @@ def check_running_jobs(self) -> None:
                     EdgeJob.set_state(job.edge_job.key, 
TaskInstanceState.FAILED)
             if job.logfile.exists() and job.logfile.stat().st_size > 
job.logsize:
                 with job.logfile.open("r") as logfile:
+                    push_log_chunk_size = conf.getint("edge", 
"push_log_chunk_size", fallback=1024)

Review Comment:
   Fallback is not needed if the config entry is in provider YAML defined.
   ```suggestion
                       push_log_chunk_size = conf.getint("edge", 
"push_log_chunk_size")
   ```



##########
providers/src/airflow/providers/edge/provider.yaml:
##########
@@ -102,3 +102,11 @@ config:
         type: integer
         example: ~
         default: "60"
+      push_log_chunk_size:
+        description: |
+          Edge Worker uploads log files in chunks. If the log file part which 
is uploaded exceeds the chunk size it creates a new request. This solves issue 
if firewall defines max size of request.
+          This value must be defined in Bytes.
+        version_added: ~
+        type: integer
+        example: ~
+        default: "1024"

Review Comment:
   Following the NGinx documentation in 
https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size 
the default is 1MB.
   
   I'd propose to set the default to 512KB



##########
providers/src/airflow/providers/edge/provider.yaml:
##########
@@ -102,3 +102,11 @@ config:
         type: integer
         example: ~
         default: "60"
+      push_log_chunk_size:
+        description: |
+          Edge Worker uploads log files in chunks. If the log file part which 
is uploaded exceeds the chunk size it creates a new request. This solves issue 
if firewall defines max size of request.
+          This value must be defined in Bytes.

Review Comment:
   The rationale is usually not a firewall but limits on the application 
gateway.
   
   Can you point as example to nginx documentation in 
https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size 
as nginx is used in most K8s ingress deployments, but also give a hint that if 
HTTP 413 is seen that this needs to be adjusted.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to