Meng-Shuan Tsai created HDDS-16124:
--------------------------------------
Summary: Revisit persistence.path handling in ozone helm charts
Key: HDDS-16124
URL: https://issues.apache.org/jira/browse/HDDS-16124
Project: Apache Ozone
Issue Type: Sub-task
Components: Helm, kubernetes
Reporter: Meng-Shuan Tsai
Assignee: Meng-Shuan Tsai
The ozone Helm chart exposes {{persistence.path}} values for each component
({{{}om.persistence.path{}}}, {{{}scm.persistence.path{}}},
{{{}datanode.persistence.path{}}}, {{{}s3g.persistence.path{}}}, all defaulting
to {{{}/data{}}}), which suggests the volume mount path is user-configurable.
However, the actual paths that Ozone processes read/write are generated in
{{templates/_helpers.tpl}} ({{{}ozone.configuration.env.common{}}}) with
hardcoded {{/data}} values, shared by om, scm, datanode, s3g and recon:
{code:yaml}
- name: OZONE-SITE.XML_hdds.datanode.dir
value: /data/storage
- name: OZONE-SITE.XML_ozone.scm.datanode.id.dir
value: /data/metadata
- name: OZONE-SITE.XML_ozone.metadata.dirs
value: /data/metadata
{code}
In addition, {{templates/om/om-statefulset.yaml}} hardcodes the OM
initialization check:
{code:yaml}
- name: ENSURE_OM_INITIALIZED
value: /data/metadata/om/current/VERSION
{code}
Meanwhile, the PVC mount points *do* consume the values, e.g. in
{{{}om-statefulset.yaml{}}}:
{code:yaml}
volumeMounts: - name: {{ .Release.Name }}-om
mountPath: {{ .Values.om.persistence.path }}
{code}
The same pattern exists in the datanode, scm and s3g StatefulSets.
There is no single source of truth: the chart only works because the default
value of {{persistence.path}} happens to match the hardcoded {{{}/data{}}}.
*Failure scenario* — if a user sets {{{}om.persistence.path: /ozone-data{}}}:
# The PVC is mounted at {{{}/ozone-data{}}}, but OM still writes metadata to
{{/data/metadata}} (per the hardcoded ozone-site config), i.e. onto the
container's ephemeral filesystem. All data is silently lost on pod restart.
# {{ENSURE_OM_INITIALIZED}} still points at
{{{}/data/metadata/om/current/VERSION{}}}, which now lives on the ephemeral
layer, so the {{VERSION}} file is missing after every restart and the
entrypoint re-runs {{ozone om --init}} on each pod restart.
# The OM decommission hook job
({{{}templates/helm/om-decommission-job.yaml{}}}) cleans up the decommissioned
node's PVC via {{{}rm -rf /old\{{ .Values.om.persistence.path }}/*{}}}; in this
scenario the PVC never contained any OM data, so the cleanup no longer operates
on the actual data location.
This affects every component wired to the shared config block (datanode, scm,
om, s3g, recon, and the helm hook jobs). The {{persistence.path}} values are
effectively not safe to change, but nothing in the chart documentation or
values comments indicates this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
