Userdata APPEND policy concatenates cloud-config as text, causing same-name top-level keys to silently override template-linked userdata

[CodeBleu]

problem

When a template has a linked userdata with policy APPENDONLY/APPEND, and a user deploys a VM with additional cloud-config userdata, CloudStack concatenates the two cloud-config documents as raw text rather than producing a proper multipart MIME message with two distinct parts.

The resulting payload is a single MIME part containing two #cloud-config documents glued together. Because cloud-init parses this as a single YAML document, any duplicate top-level keys (e.g., runcmd, write_files) cause the second occurrence to silently override the first per YAML semantics. The template-linked userdata is therefore not actually enforced — a user who supplies a cloud-config with the same top-level keys as the template silently destroys the template's directives.

This breaks the security guarantee that APPENDONLY is documented to provide. The documentation states:

Don't allow users to override linked UserData but allow users to pass userdata content … which is appended to the linked UserData of the Template.

In practice, override is exactly what occurs for any colliding top-level key.

Expected Behavior

Both the template's directives and the user's directives should execute. With the reproduction steps in the next section, all four files should exist after boot:

• /tmp/template-write-files.txt
• /tmp/template-runcmd.txt
• /tmp/user-write-files.txt
• /tmp/user-runcmd.txt

This requires CloudStack to deliver the appended userdata as a proper multipart MIME message with two distinct text/cloud-config parts. cloud-init would then merge them correctly using its standard merge handlers (or via merge_how directives if specified).

Actual Behavior

Only the user-supplied directives execute. The template's directives are silently lost:

File	Status
/tmp/template-write-files.txt	MISSING
/tmp/template-runcmd.txt	MISSING
/tmp/user-write-files.txt	exists
/tmp/user-runcmd.txt	exists

The cloud-init metadata files reveal why.

/var/lib/cloud/instance/user-data.txt.i shows:

Content-Type: multipart/mixed; boundary="===============XXXXXXX=="
MIME-Version: 1.0
Number-Attachments: 1

--===============XXXXXXX==
MIME-Version: 1.0
Content-Type: text/cloud-config
Content-Disposition: attachment; filename="part-001"

#cloud-config
write_files:
  - path: /tmp/template-write-files.txt
    ...
runcmd:
  - echo "template runcmd ran" > /tmp/template-runcmd.txt

#cloud-config
write_files:
  - path: /tmp/user-write-files.txt
    ...
runcmd:
  - echo "user runcmd ran" > /tmp/user-runcmd.txt
--===============XXXXXXX==--

Note Number-Attachments: 1 — the two cloud-configs are concatenated into a single part, not delivered as two distinct parts. The second #cloud-config line is interpreted as a YAML comment by cloud-init's parser, not a part separator.

/var/lib/cloud/instance/cloud-config.txt confirms cloud-init only saw one document:

#cloud-config
# from 1 files
# part-001
---
runcmd:
- echo "user runcmd ran" > /tmp/user-runcmd.txt
write_files:
- path: /tmp/user-write-files.txt
  ...

The template's runcmd and write_files were silently discarded due to YAML duplicate-key resolution.

Security Impact

This bug means APPENDONLY does not actually enforce its documented guarantee. Operators relying on APPENDONLY to deliver mandatory cloud-init configuration (e.g., security mitigations, logging agents, baseline hardening) cannot trust that those configurations are applied — any user-supplied cloud-config with colliding top-level keys will silently override them, with no error or warning to the operator or the user.

For organizations using APPENDONLY to enforce kernel-vulnerability mitigations or compliance baselines, this is a silent security regression.

versions

• CloudStack: 4.19.3.0
• Hypervisor: KVM
• Userdata datasource: CloudStack (virtual router)
• Cloud-init (guest): 25.2
• Guest OS reproduced on: Ubuntu 24.04

Note: this issue is in CloudStack's userdata merge logic on the management server side, not OS- or hypervisor-specific.

The steps to reproduce the bug

1. Register a userdata in CloudStack with the following content:

   #cloud-config
   write_files:
     - path: /tmp/template-write-files.txt
       content: "template write_files ran\n"
   
   runcmd:
     - echo "template runcmd ran" > /tmp/template-runcmd.txt

2. Link this userdata to a template with override policy APPENDONLY.

3. Deploy a VM from this template, supplying additional manual userdata:

   #cloud-config
   write_files:
     - path: /tmp/user-write-files.txt
       content: "user write_files ran\n"
   
   runcmd:
     - echo "user runcmd ran" > /tmp/user-runcmd.txt

4. After the VM boots, inspect:

   ls -la /tmp/template-* /tmp/user-*
   cat /var/lib/cloud/instance/user-data.txt
   cat /var/lib/cloud/instance/user-data.txt.i
   cat /var/lib/cloud/instance/cloud-config.txt

5. Observe that /tmp/template-write-files.txt and /tmp/template-runcmd.txt do not exist, while /tmp/user-write-files.txt and /tmp/user-runcmd.txt do. The .i metadata file shows Number-Attachments: 1, confirming CloudStack delivered both cloud-configs concatenated into a single MIME part rather than as two distinct parts.

What to do about it?

Suggested Fix

CloudStack's userdata append logic should produce a proper multipart MIME message with two distinct parts, each tagged with the appropriate Content-Type (e.g., text/cloud-config), rather than concatenating raw content. cloud-init's existing handlers will then merge the parts correctly, and operators can use merge_how directives in the template userdata to control merge behavior per-key (replace, append, recurse).

The relevant code path is the userdata combination logic invoked when policy is APPENDONLY and the user supplies additional userdata via userdata= or userdataid= on deployVirtualMachine.

Workarounds Attempted (none sufficient)

• merge_how directives in the template userdata — do not work, because cloud-init's merge handlers operate across separate MIME parts, not within a single concatenated document.
• Wrapping the template userdata in a multipart MIME envelope when registering — does not help; CloudStack still concatenates the result.
• Using non-colliding keys (e.g., bootcmd in user-supplied userdata when template uses runcmd) — works, but requires every user to know which keys to avoid, which defeats the security purpose of APPENDONLY.

Related History

Issue #7918 ("Joint UserData must have a header") and PR #9575 ("Fix userdata append header restrictions", merged Aug 2024, included in 4.19.2+) addressed a different aspect of joint userdata handling. The header-restriction fix is present in 4.19.3.0 and is not the issue reported here. This is a separate problem with the append mechanism producing a single MIME part instead of multiple distinct parts.

[CodeBleu]

cc @nvazquez @sureshanaparti @weizhouapache @harikrishna-patnala @DaanHoogland — this is in the same area as #7918 / #9575 but a separate issue with the merge mechanism itself, would appreciate your input.

[DaanHoogland]

@CodeBleu (and others), isn’t this overriding of the template level definition the exact intention of the append method? I understand this might be a security rtisk in some cases, but append not overiding the template level definition would cause me to complain to my provider.

[CodeBleu]

@CodeBleu (and others), isn’t this overriding of the template level definition the exact intention of the append method? I understand this might be a security rtisk in some cases, but append not overiding the template level definition would cause me to complain to my provider.

As mentioned above. Here is what actual docs say. IMO, APPEND/APPEND_ONLY should do just that, append, otherwise it doesnt make sense.

https://docs.cloudstack.apache.org/en/4.19.3.0/adminguide/virtual_machines/user-data.html

Append Only: Don’t allow users to override linked UserData but allow users to pass userdata content
or ID that should be appended to the linked UserData of the Template. When the users pass userdata it is appended to the Template userdata in the form of a multipart MIME message