CodeBleu opened a new issue, #13092:
URL: https://github.com/apache/cloudstack/issues/13092
### problem
When a template has a linked userdata with policy `APPENDONLY`/`APPEND`, and
a user deploys a VM with additional cloud-config userdata, CloudStack
concatenates the two cloud-config documents as raw text rather than producing a
proper multipart MIME message with two distinct parts.
The resulting payload is a single MIME part containing two `#cloud-config`
documents glued together. Because cloud-init parses this as a single YAML
document, any duplicate top-level keys (e.g., `runcmd`, `write_files`) cause
the second occurrence to silently override the first per YAML semantics. The
template-linked userdata is therefore not actually enforced — a user who
supplies a cloud-config with the same top-level keys as the template silently
destroys the template's directives.
This breaks the security guarantee that `APPENDONLY` is documented to
provide. The documentation states:
> Don't allow users to override linked UserData but allow users to pass
userdata content … which is appended to the linked UserData of the Template.
In practice, override is exactly what occurs for any colliding top-level key.
## Expected Behavior
Both the template's directives **and** the user's directives should execute.
With the reproduction steps in the next section, all four files should exist
after boot:
- `/tmp/template-write-files.txt`
- `/tmp/template-runcmd.txt`
- `/tmp/user-write-files.txt`
- `/tmp/user-runcmd.txt`
This requires CloudStack to deliver the appended userdata as a proper
multipart MIME message with two distinct `text/cloud-config` parts. cloud-init
would then merge them correctly using its standard merge handlers (or via
`merge_how` directives if specified).
## Actual Behavior
Only the user-supplied directives execute. The template's directives are
silently lost:
| File | Status |
|---|---|
| `/tmp/template-write-files.txt` | **MISSING** |
| `/tmp/template-runcmd.txt` | **MISSING** |
| `/tmp/user-write-files.txt` | exists |
| `/tmp/user-runcmd.txt` | exists |
The cloud-init metadata files reveal why.
`/var/lib/cloud/instance/user-data.txt.i` shows:
```
Content-Type: multipart/mixed; boundary="===============XXXXXXX=="
MIME-Version: 1.0
Number-Attachments: 1
--===============XXXXXXX==
MIME-Version: 1.0
Content-Type: text/cloud-config
Content-Disposition: attachment; filename="part-001"
#cloud-config
write_files:
- path: /tmp/template-write-files.txt
...
runcmd:
- echo "template runcmd ran" > /tmp/template-runcmd.txt
#cloud-config
write_files:
- path: /tmp/user-write-files.txt
...
runcmd:
- echo "user runcmd ran" > /tmp/user-runcmd.txt
--===============XXXXXXX==--
```
Note `Number-Attachments: 1` — the two cloud-configs are concatenated into a
single part, not delivered as two distinct parts. The second `#cloud-config`
line is interpreted as a YAML comment by cloud-init's parser, not a part
separator.
`/var/lib/cloud/instance/cloud-config.txt` confirms cloud-init only saw one
document:
```yaml
#cloud-config
# from 1 files
# part-001
---
runcmd:
- echo "user runcmd ran" > /tmp/user-runcmd.txt
write_files:
- path: /tmp/user-write-files.txt
...
```
The template's `runcmd` and `write_files` were silently discarded due to
YAML duplicate-key resolution.
## Security Impact
This bug means `APPENDONLY` does not actually enforce its documented
guarantee. Operators relying on `APPENDONLY` to deliver mandatory cloud-init
configuration (e.g., security mitigations, logging agents, baseline hardening)
cannot trust that those configurations are applied — any user-supplied
cloud-config with colliding top-level keys will silently override them, with no
error or warning to the operator or the user.
For organizations using `APPENDONLY` to enforce kernel-vulnerability
mitigations or compliance baselines, this is a silent security regression.
### versions
- **CloudStack:** 4.19.3.0
- **Hypervisor:** KVM
- **Userdata datasource:** CloudStack (virtual router)
- **Cloud-init (guest):** 25.2
- **Guest OS reproduced on:** Ubuntu 24.04
Note: this issue is in CloudStack's userdata merge logic on the management
server side, not OS- or hypervisor-specific.
### The steps to reproduce the bug
1. Register a userdata in CloudStack with the following content:
```yaml
#cloud-config
write_files:
- path: /tmp/template-write-files.txt
content: "template write_files ran\n"
runcmd:
- echo "template runcmd ran" > /tmp/template-runcmd.txt
```
2. Link this userdata to a template with override policy `APPENDONLY`.
3. Deploy a VM from this template, supplying additional manual userdata:
```yaml
#cloud-config
write_files:
- path: /tmp/user-write-files.txt
content: "user write_files ran\n"
runcmd:
- echo "user runcmd ran" > /tmp/user-runcmd.txt
```
4. After the VM boots, inspect:
```bash
ls -la /tmp/template-* /tmp/user-*
cat /var/lib/cloud/instance/user-data.txt
cat /var/lib/cloud/instance/user-data.txt.i
cat /var/lib/cloud/instance/cloud-config.txt
```
5. Observe that `/tmp/template-write-files.txt` and
`/tmp/template-runcmd.txt` do not exist, while `/tmp/user-write-files.txt` and
`/tmp/user-runcmd.txt` do. The `.i` metadata file shows `Number-Attachments:
1`, confirming CloudStack delivered both cloud-configs concatenated into a
single MIME part rather than as two distinct parts.
### What to do about it?
## Suggested Fix
CloudStack's userdata append logic should produce a proper multipart MIME
message with two distinct parts, each tagged with the appropriate
`Content-Type` (e.g., `text/cloud-config`), rather than concatenating raw
content. cloud-init's existing handlers will then merge the parts correctly,
and operators can use `merge_how` directives in the template userdata to
control merge behavior per-key (replace, append, recurse).
The relevant code path is the userdata combination logic invoked when policy
is `APPENDONLY` and the user supplies additional userdata via `userdata=` or
`userdataid=` on `deployVirtualMachine`.
## Workarounds Attempted (none sufficient)
- **`merge_how` directives** in the template userdata — do not work, because
cloud-init's merge handlers operate across separate MIME parts, not within a
single concatenated document.
- **Wrapping the template userdata in a multipart MIME envelope** when
registering — does not help; CloudStack still concatenates the result.
- **Using non-colliding keys** (e.g., `bootcmd` in user-supplied userdata
when template uses `runcmd`) — works, but requires every user to know which
keys to avoid, which defeats the security purpose of `APPENDONLY`.
## Related History
Issue #7918 ("Joint UserData must have a header") and PR #9575 ("Fix
userdata append header restrictions", merged Aug 2024, included in 4.19.2+)
addressed a different aspect of joint userdata handling. The header-restriction
fix is present in 4.19.3.0 and is not the issue reported here. This is a
separate problem with the append mechanism producing a single MIME part instead
of multiple distinct parts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
