[
https://issues.apache.org/jira/browse/HDDS-16156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16156:
------------------------------
Attachment: HDDS-16156.001.patch
> Flaky kubernetes check: s3g CrashLoopBackOff with "Access denied:
> ozone_http_tmp_base_dir" (HTTP base dir created under non-writable CWD)
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16156
> URL: https://issues.apache.org/jira/browse/HDDS-16156
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: kubernetes, s3gateway
> Reporter: Siyao Meng
> Priority: Major
> Attachments: HDDS-16156.001.patch
>
>
> h3. Symptom
> In the {{kubernetes}} acceptance check, the {{s3g}} container
> CrashLoopBackOffs: it exits immediately at startup, before serving anything,
> so cluster bring-up for the {{ozone}} example (and others) fails ("Test
> execution of ozone is FAILED"). Every {{s3g}} pod across all k8s examples in
> the run (getting-started, ozone, ozone-dev, ozone-ha, minikube) fails
> identically; {{scm}}/{{om}}/{{recon}}/{{httpfs}} start fine.
> {code}
> 2026-08-11 22:11:26 INFO TracingUtil:87 - Initialized tracing service:
> S3gateway (enabled=false, applicationAware=true)
> Error: Access denied: ozone_http_tmp_base_dir3661902540861472713
> {code}
> (from {{kubernetes/.../s3g-0}} pod log)
> h3. Root cause
> {{HttpServer2.setHttpBaseDir()}} creates the HTTP temp base directory under
> the process *current working directory* when {{ozone.http.basedir}} is unset:
> {code}
> // hadoop-hdds/framework/.../server/http/HttpServer2.java:1808-1815
> File tmpMetaDir = Files.createTempDirectory(Paths.get(""),
> "ozone_http_tmp_base_dir").toFile();
> {code}
> When the container's CWD is not writable, {{Files.createTempDirectory}}
> throws {{AccessDeniedException}}, which {{GenericCli}} reports as {{"Access
> denied: " + message}} (GenericCli.java:157) and the process exits. Only
> {{s3g}} is affected: it is stateless and has no metadata dir, so
> {{ozone.http.basedir}} stays empty and it falls into this CWD path, whereas
> {{scm}}/{{om}}/{{recon}} resolve {{ozone.http.basedir}} to their writable
> metadata dir and never enter it (their logs never reference
> {{ozone_http_tmp_base_dir}}). The s3g statefulset sets no {{workingDir}}, so
> the CWD is the image default. This makes s3g startup fragile to the runtime
> environment's CWD writability, so the {{kubernetes}} check fails
> intermittently (it is green on master for the same code/image). The
> CWD-relative temp dir was introduced in HDDS-9483 (see also HDDS-5239).
> h3. Fix direction
> Create the HTTP base dir under a guaranteed-writable location
> ({{java.io.tmpdir}}, or the service metadata dir) instead of the process CWD
> ({{Paths.get("")}}); alternatively set {{ozone.http.basedir}} for s3g. Also
> consider surfacing the offending path in the error so this is not just
> "Access denied".
> h3. Notes
> Observed on a PR build (run below) but *not* caused by that PR: the failing
> code path predates it and runs during early s3g startup, before any SCM
> interaction, so it is unrelated to the change under test. PR-build
> logs/artifacts expire; the s3g pod log snippet above is the evidence.
> - https://github.com/apache/ozone/actions/runs/31539983527/job/93943601739
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
