feat(mount): nested idmapped mounts via open_tree_attr (SDK idmap support) (#3248)

• feat(core/mount): add syscall primitives for new mount API + idmap

Foundation for migrating start-os off mount(8) CLI to direct syscalls.

• core/src/disk/mount/filesystem/syscall.rs wraps the new mount API (open_tree, move_mount, fsopen/fsconfig/fsmount, mount_setattr) and the userns-fd-from-idmap dance via libc::syscall. nix 0.30 (in deps) does not expose the new mount API yet; only FSCONFIG_* command codes are defined locally because libc doesn’t ship them. Includes syscall-based replacements for umount(8), sync -f, mountpoint, and mount --make-rshared.

• New startbox unshare-userns bin: unshare(CLONE_NEWUSER)s, prints ready\n, blocks on stdin. The parent (userns_fd_from_idmap) writes the helper’s /proc/<pid>/uid_map/gid_map, opens /proc/<pid>/ns/user to capture the userns fd, then drops the helper’s stdin to let it exit. The captured fd keeps the userns alive for later mount_setattr(MOUNT_ATTR_IDMAP) use.

Follow-up commits in this branch will:

• migrate IdMapped to use the syscall path (the actual nested-idmap fix)
• delete IdMap::stack and simplify dependency.rs::mount
• add a startbox mount subcommand for inside-LXC use; route the SDK inner bind through it instead of util-linux mount
• uncomment SharedOptions.idmap in the SDK and propagate
• reshape the FileSystem trait and migrate the remaining filesystems (Bind, OverlayFs, BlockDev, LoopDev, Label, EfivarFs) to syscalls; keep helper-spawn for cifs/ecryptfs/httpdirfs/backupfs
• replace remaining mount(8)/umount(8) CLI uses in util.rs / guard.rs
• 7.0.7 VM validation

• feat(sdk/mount): support SharedOptions.idmap via start-container mount

Ships the actual nested-idmap feature: the SDK’s idmap?: …[] field on volume / asset / dependency / backup mounts is now wired up end-to-end on the 7.0.7-backports kernel.

• Adds start-container mount (local_mount.rs) which performs a bind via open_tree(OPEN_TREE_CLONE) + optional mount_setattr(MOUNT_ATTR_IDMAP, userns_fd) + move_mount. The CLI takes plain --source/--target/--idmap flags and is the path container-runtime now uses from inside the LXC. util-linux’s mount --bind -oX-mount.idmap=… is no longer involved for the inner bind step.

• SubContainer.ts bind() swapped from mount --bind to start-container mount. Pointer mounts now do a two-step setup: the host-side effects.mount lays down the base-mapped bind, then a follow-up start-container mount overlays the SDK idmap if any — matching the volume/asset/backup semantics.

• Mounts.ts: uncomments SharedOptions.idmap? (with range defaulting to 1) and threads it through build() for all four mount kinds.

• dependency.rs: drops IdMap::stack (a 6.x-era workaround that produced overlapping uid_map ranges); host-side bind for pointer mounts now uses only [{0, 100000, 65536}] (the LXC base map). The SDK idmap from MountTarget.idmap is ignored on this side — applied by start-container mount from inside the LXC instead.

• RPC mount subcommand renamed to mount-pointer to free the mount name for the new CLI surface. Updated container-runtime’s EffectCreator (rpcRound("mount-pointer", …)) — public SDK effects.mount(…) signature unchanged.

• start-container’s MultiExecutable also enables unshare-userns so the userns-fd helper resolves inside the LXC.

Still to come on this branch: migrating IdMapped itself to syscalls, the broader FileSystem trait reshape with BackupFS/Cifs/ECryptFs/ HttpDirFs on the helper-spawn path, replacing the remaining mount(8)/umount(8)/sync -f/make-rshared callers, regenerating ts-bindings + SDK dist, and 7.0.7 VM validation.

• refactor(mount/idmapped): migrate IdMapped to syscalls

IdMapped<Fs> no longer emits X-mount.idmap= mount options for mount(8) to interpret; it now mounts the inner filesystem on a deduplicated tmp staging path (via TmpMountGuard), clones the mount into a detached fd with open_tree(OPEN_TREE_CLONE | AT_RECURSIVE), applies mount_setattr(MOUNT_ATTR_IDMAP, userns_fd), and finishes with move_mount onto the real mountpoint. The TmpMountGuard’s staging mount is released when the call returns; the attached mount is the independent clone the kernel produced.

Side effects of the rewrite:

• drops the now-unused mount_type / extra_args / mount_options / source / pre_mount overrides — IdMapped no longer cares about the argv-building infrastructure
• removes the pre-mount chown walk; the kernel idmap handles the remapping for us
• ReadOnly requests are honored via a follow-up mount_setattr(MOUNT_ATTR_RDONLY) on the detached fd, so the TmpMountGuard “ro upgraded to rw if shared” dragon doesn’t bite us

Works for any inner FileSystem — Bind, LoopDev, BlockDev, etc. — so we can keep migrating leaf filesystems to syscalls independently without touching IdMapped again.

• refactor(mount/bind): use open_tree+move_mount instead of mount –bind

Bind no longer goes through default_mount_impl / mount(8). The bind is done by open_tree(OPEN_TREE_CLONE | AT_RECURSIVE?) and attached at the mountpoint via move_mount(MOVE_MOUNT_F_EMPTY_PATH). Readonly requests are honored via mount_setattr(MOUNT_ATTR_RDONLY) on the detached fd.

The existing dir/file prep (creating source/target if missing, removing the wrong kind on the target) moves into a private prep method called from the new mount() override; behavior is unchanged.

• refactor(mount): migrate EfiVarFs + OverlayFs to fsopen/fsmount

Both filesystems are kernel-native and have well-defined options that fit the new-mount-API cleanly:

• EfiVarFs: fsopen("efivarfs") + fsconfig_create + fsmount + optional RDONLY via mount_setattr.

• OverlayFs: fsopen("overlay") + fsconfig_set_string("lowerdir", …) / upperdir / workdir + fsconfig_create + fsmount. The colon-joined lowerdir string is the same shape we used to put on the -o lowerdir=… argv.

Drops the old mount_type / mount_options / source / pre_mount overrides on both; the trait defaults are fine because mount() is the only entry point either ever exercises now.

• refactor(mount/block_dev): syscall path opt-in via with_type

BlockDev<P> gains an optional fs_type: &'static str (set via with_type("squashfs") etc.). When set, mount() goes through `fsopen(““) + fsconfig_set_string(“source”, …) + fsconfig_create

• fsmount. When unset, falls back to a new mount_via_clihelper infilesystem::modthat spawnsmount(8)— preserves autodetect for the unknown-disk probing case indisk::util::find_partition_infoandos_install` (which has to deal with whatever ext4/vfat/etc. it finds on user disks).

mount_via_cli(fs_type, extra_args, options, source, mountpoint, mt) also stands ready for helper-spawn filesystems (Cifs/ECryptFs/HttpDirFs/ BackupFS) — they’ll migrate to it in the next commit instead of carrying their own Command::new("mount") boilerplate.

Type-tags applied to the two known squashfs callers:

• container-runtime rootfs squashfs in lxc/mod.rs

• nvidia overlay squashfs in context/rpc.rs

• refactor(mount): replace remaining mount(8)/umount(8)/sync CLI calls

Routes the leftover Command::new("mount"/"umount"/"sync"/"mountpoint") sites in the mount layer through the syscall helpers:

• disk::mount::util::unmount → umount2(MNT_DETACH if lazy). Drops the LANG=C.UTF-8 / "not mounted" string-matching hack — EINVAL from the kernel directly tells us “not mounted” and is swallowed inside the helper.
• is_mountpoint → st_dev comparison against the parent dir (no mountpoint(1) subprocess).
• sync_filesystem → syncfs(2) via an O_PATH-style open.
• sync_directory → fsync(2) (File::sync_all).
• bind → open_tree(OPEN_TREE_CLONE) + optional mount_setattr(RDONLY)
  • move_mount (matching the new Bind impl, just without the dir/file prep dance).
• lxc::mod::mount --make-rshared <rootfs> → mount(2) with MS_SHARED | MS_REC via the new make_rshared helper.
• guard::TmpMountGuard::mount‘s mount -o remount,rw upgrade → mount(2) with MS_REMOUNT via a new remount_rw helper.

unmount_all_under keeps its /proc/mounts parsing — there’s no syscall to enumerate kernel mounts from userspace beyond procfs, and the existing logic is already syscall-grade once the per-mountpoint unmount calls go through umount2.

• refactor(mount/loop_dev): syscall path via LOOP_CONFIGURE ioctl

LoopDev<P> gains the same with_type opt-in pattern as BlockDev. When a type is declared, the mount uses:

• LOOP_CTL_GET_FREE on /dev/loop-control to allocate a loop number
• LOOP_CONFIGURE on the resulting /dev/loopN with the backing fd, offset, sizelimit, and LO_FLAGS_AUTOCLEAR so the loop is torn down automatically once the filesystem is unmounted
• fsopen("<ty>") + fsconfig_set_string("source", "/dev/loopN") + fsconfig_create + fsmount to build the filesystem
• move_mount (via DetachedMount::attach) onto the requested mountpoint

From<&Section<MultiCursorFile>> sets with_type("squashfs") because every loop-mounted s9pk subarchive in the codebase is squashfs. Callers that need a different fs can still drop down to LoopDev::new(...).

Unset fs_type (the From-less path) falls through to mount_via_cli with -o loop,offset=,sizelimit= so existing autodetect-relying paths keep working.

Adds loop_attach to the syscall module — the only nix/libc::ioctl combination in the mount layer; LOOP_* constants and the loop_config/loop_info64 structs aren’t exported by libc so they’re defined locally with the standard linux/loop.h layout.

• fix(mount): use open_tree_attr(2) for idmapped clones

The two-step open_tree(OPEN_TREE_CLONE) + mount_setattr(MOUNT_ATTR_IDMAP) sequence we were doing isn’t enough for the case that motivates this whole PR: when the source filesystem is itself idmapped (the LXC-userns scenario), the kernel rejects the second-step setattr because the cloned mount fd already existed in a non-idmapped state. The kernel only permits attaching an idmap to a clone-of-idmapped-source if the idmap is applied atomically as part of the clone — that’s the entire reason open_tree_attr(2) was added in Linux 6.15.

This is why util-linux’s CLI (which uses the same two-step pattern via libmount’s hook_idmap.c) doesn’t work for our nested case on 7.0.7, and why our own first cut wouldn’t have worked either if we’d actually tested it on the VM. libmount has zero open_tree_attr callers in its entire history, so leaning on it wouldn’t have helped.

Changes:

• syscall.rs: add SYS_OPEN_TREE_ATTR = 467 (unified across all start-os target arches; libc 0.2 only exports the m68k variant) and a open_tree_attr_idmap(path, recursive, userns_fd) wrapper. Same MountAttr struct as before, just composed with the open_tree flags in a single syscall.
• local_mount.rs: idmap path now produces the detached fd via open_tree_attr_idmap instead of open_tree_clone + set_idmap. No- idmap path keeps open_tree_clone.
• idmapped.rs: same swap. IdMapped::mount now does TmpMountGuard -> open_tree_attr_idmap(staging.path(), …) -> move_mount.

mount_setattr_idmap stays in the syscall module for any future caller that wants the standalone setattr, but no caller in the tree uses it now — all the actual idmapped mounts go through open_tree_attr.

• chore(sdk): apply prettier formatting from sdk build pipeline

Running make sdk/dist/package.json after the rebase onto master re-flowed a handful of long function-call lines and type-parameter lists in SubContainer.ts / CommandController.ts / Daemons.ts. No semantic changes.

• fix(mount/syscall): musl cross-compile fixes

CI aarch64-musl + x86_64-musl release builds failed on the syscall module — three real issues:

• libc::MOVE_MOUNT_F_EMPTY_PATH only exists for linux-gnu in libc 0.2.186, not for the musl flavor. Define it locally as const MOVE_MOUNT_F_EMPTY_PATH: c_uint = 0x4 (it’s a stable kernel ABI constant).

• LOOP_CTL_GET_FREE / LOOP_CONFIGURE were typed libc::c_ulong but libc::ioctl‘s second arg takes the libc::Ioctl type alias, which is c_int on musl and c_ulong on gnu. Retyping the constants to libc::Ioctl makes them portable across our cross-compile targets.

• Drop the unused std::os::fd::AsFd import from bind.rs and the unused Invoke import from guard.rs that the CI lint flagged after the syscall migration.

• fix: non-Linux build (start-cli darwin cross-compile)

Two more CI breakages on top of the musl-arch fixes:

• start-cli is built for *-apple-darwin and *-unknown-linux-musl from the same start-os library crate. None of the new mount-API syscalls (open_tree*, mount_setattr, fsopen/fsmount, LOOP_CONFIGURE, unshare(CLONE_NEWUSER)) exist outside Linux, so the whole syscall.rs module fails to even parse on aarch64-apple-darwin. Cargo-gating each call site would touch ~10 files; instead, add a sibling syscall_stub.rs with signature-compatible no-op stubs (each returns ErrorKind::Filesystem with a “Linux-only” message) and pick the right module via #[cfg(target_os = "linux")] in mod.rs. start-cli on macOS never invokes the mount layer at runtime, so callers compile but error cleanly if they’re ever reached.

• container-runtime rpcRound is typed <K extends T.EffectMethod | "clearCallbacks"> — my rename of the RPC route mount → mount-pointer doesn’t fit that union because Effects.mount is the wrapper, not the route name. Add "mount-pointer" to the allowlist with an inline comment explaining the wrapper/route split.

• fix(mount): adversarial-review fixes (idmap inversion, squashfs loop, infer-file)

Found by an adversarial review pass of this PR. Three real bugs, two of them deterministic showstoppers on the service-launch path:

• I1 (critical): userns_fd_from_idmap wrote the /proc//uid_map and gid_map columns inverted — to_id from_id range instead of from_id to_id range. The kernel parses col1 -> extent.first, col2 -> extent.lower_first, and make_vfsuid runs the on-disk id through map_id_down (matched against col1=first, output col2=lower_first), so col1 is the on-disk id (IdMap::from_id) and col2 is the id the mount user sees (IdMap::to_id). Verified against torvalds/linux v6.15 kernel/user_namespace.c (map_id_range_down, map_write) and fs/mnt_idmapping.c (make_vfsuid). With the bug, the LXC base map {0,100000,65536} emitted 100000 0 65536, so on-disk uid 0 fell outside col1’s [100000,165536) range and mapped to the overflow id (nobody) — every idmapped mount (container rootfs, volumes, assets, dependencies) presented its data as nobody and the kernel raised no error. Fixed to from_id to_id range.

• I2 (high): the container rootfs squashfs (lxc/mod.rs) and the NVIDIA overlay squashfs (context/rpc.rs) are regular files, but I had tagged them .with_type("squashfs"), routing them to the fsopen syscall path. squashfs is FS_REQUIRES_DEV -> get_tree_bdev -> lookup_bdev, which returns ENOTBLK for a non-block-device source; the new mount API does not auto-create a loop device the way mount(8) does. Dropped .with_type on both so they use the mount(8) auto-loop fallback (the proven pre-PR behavior); genuine block devices keep the syscall path.

• I3 (medium): SubContainer.ts bind() only passed --file for an explicit type:'file', but prepBind also prepares a file target for type:'infer' when the source is a file. The mismatch made start-container mount take the create_dir_all branch over a just-created file (EEXIST), aborting the mount. prepBind now returns the resolved is-file decision and bind() keys --file off it — single source of truth.

Also corrected two comments this PR had added that claimed idmap support for backup mounts; Mounts.build() never emits backup entries (a pre-existing drop, unrelated to this PR — flagged separately, not fixed here to stay in scope).

cargo check (gnu + musl), make test-sdk (57), make test-container-runtime (12) all green.

• fix(mount): make unshare-userns helper reachable from start-container

Found by running the real start-container mount path on a 6.19 kernel: every idmapped mount failed with “unshare-userns helper said:” (empty).

Root cause: userns_fd_from_idmap spawned current_exe unshare-userns, but the MultiExecutable dispatcher selects a sub-bin by the basename of argv[0] and only falls through to argv[1] when argv[0] is NOT itself a registered bin. Inside the LXC current_exe is start-container — a registered, default bin — so dispatch ran the container CLI (which has no unshare-userns subcommand) instead of the helper. It happened to work from startbox only because “startbox” isn’t a registered bin name. Net effect: the entire nested-idmap feature was dead on the actual in-container path.

Fix: spawn the helper with argv[0] overridden to a non-bin sentinel (.arg0("startos-unshare-userns-helper")) so dispatch falls through to the unshare-userns selector regardless of what current_exe is named. Also capture the helper’s stderr into the error when it fails to report “ready”, so a future failure isn’t a blank message.

Validated end-to-end with the built start-container binary as root on 6.19:

• mount --idmap 0:100000:65536 on a uid-0 tree -> target stat uid=100000 (the LXC base map; confirms I1 column order: on-disk 0 -> visible 100000, not overflow/nobody)
• --idmap 0:1000:1 -> uid=1000 (SDK-documented example)
• no idmap -> uid=0 passthrough
• --readonly --idmap 0:1000:1 -> uid=1000 + ro idmapped in findmnt

• refactor(mount): descope to idmap-only; revert the rest to mount(8)

The PR had rewritten the entire mount layer (Bind, OverlayFs, BlockDev, LoopDev, EfiVarFs, util.rs, guard.rs, make-rshared, remount,rw) to direct syscalls when only the nested-idmap inner bind genuinely needs hand-rolled libc (open_tree_attr, which neither nix nor rustix nor libmount expose). That broad rewrite was the source of I2 (squashfs ENOTBLK) and I5 (is_mountpoint st_dev regression) and a large, scary review surface.

Per discussion, revert all the non-idmap filesystem/util impls to their existing mount(8)-based versions on master, and keep hand-rolled libc strictly for the idmap path:

• Reverted to master: bind.rs, overlayfs.rs, block_dev.rs, loop_dev.rs, efivarfs.rs, disk/mount/util.rs, disk/mount/guard.rs, filesystem/mod.rs, lxc/mod.rs (make-rshared), context/rpc.rs and multi_cursor_file.rs (drop the squashfs with_type tags). This deletes the I2 and I5 bug sites outright — the code is gone.

• syscall.rs trimmed from 682 to ~330 lines: only open_tree_clone, open_tree_attr_idmap, the DetachedMount {set_readonly, attach} pair, userns_fd_from_idmap, and unshare_userns_main survive — everything the reverted fs impls pulled in (fsopen/fsconfig/fsmount, LOOP_CONFIGURE, umount2/syncfs/is_mountpoint/make_rshared/remount_rw, mount_setattr IDMAP two-step) is removed. syscall_stub.rs shrinks to match.

• IdMapped keeps the stage-then-bind design (TmpMountGuard mounts the inner FS via mount(8), then open_tree_attr clones+idmaps+move_mounts) — FS-agnostic, so no per-type native idmap impls are needed.

is_mountpoint reverts to mountpoint(1), which already handles bind mounts, so the “needs hardening” note is resolved by the revert.

Re-validated on a 6.19 kernel via the real start-container binary: mount --idmap 0:100000:65536 on a uid-0 tree -> uid 100000; --idmap 0:1000:1 -> uid 1000. cargo check gnu + musl green.

• refactor(sdk): remove dead mountBackups path

Mounts.build() never emitted this.backups, so mountBackups(...) produced no mount and the type:'backup' branch in SubContainer.mount() was unreachable. Backups are handled entirely by Backups.ts (mountBackupTarget -> mount --rbind), which is independent of this machinery. The only caller was the legacy SystemForEmbassy v1 docker type:"backup" volume branch, whose mount was already a silent no-op.

Removed:

• Mounts.mountBackups + the Backups type-flag generic on Mounts (and its backups field/ctor arg), plus the type-level test at the bottom of Mounts.ts.
• SubContainer‘s MountsArg conditional (it only existed to gate backups by BackupEffects) — mount() now just takes Mounts<Manifest>. Dropped the now-unused BackupEffects import, the backup mount branch, and the MountOptionsBackup type/union member.
• The legacy type:"backup" branch in DockerProcedureContainer.ts.

BackupEffects itself (the live backup-context brand used throughout Backups.ts) is untouched. make test-sdk (57) + test-container-runtime (12) green; sdk + container-runtime dist build clean.

• fix(container-runtime): legacy v1 backups via the native rbind API

The legacy SystemForEmbassy type:"backup" docker volume mount was a no-op (it went through the dead mountBackups path, since removed), so v1 docker backup/restore procedures saw an empty backup directory.

Fix it using the exact mechanism native 0.4 packages use: Backups.ts‘s mountBackupTarget, which rbinds the host backup dir (/media/startos/backup) into the subcontainer rootfs. Generalized that helper to take the container mountpoint (default unchanged at /backup-target for native callers) and exported it; the legacy docker branch now calls mountBackupTarget(subcontainer.rootfs, mounts[mount]) so the backup/restore procedure finds its volume at the manifest-declared path. Both native and legacy backups now go through the same rbind — no revival of the removed mountBackups builder.

make test-sdk (57) + test-container-runtime (12) green; sdk + container-runtime dist build clean.

• docs(mount): explain why unshare-userns must be an applet, not a subcommand

• docs(mount): correct the reason unshare-userns can’t be a subcommand

It’s not the tokio runtime (CliApp::run uses handle_sync, no runtime) — it’s that container_cli::main enables the logger first, whose tracing-appender non-blocking writer spawns a background thread, making the process multi-threaded before any subcommand handler runs. unshare(CLONE_NEWUSER) requires single-threaded.

• refactor(effects/mount): drop unused idmap field from MountTarget RPC

The host-side mount (mount-pointer) handler ignores the SDK-supplied idmap — the idmap is applied by the inner start-container mount from inside the LXC, and the host bind only ever uses the hardcoded LXC base map. So MountTarget.idmap was dead weight on the wire.

• dependency.rs: remove the idmap: Vec<IdMap> field and the idmap: _ destructuring. The hardcoded base map in the handler body is unchanged.
• Regenerated ts-bindings: osBindings/MountTarget.ts loses idmap, and osBindings/IdMap.ts is now orphaned and removed (it was only a MountTarget dependency).
• Updated the TS callers that passed idmap: [] to a mount target: SubContainer.ts pointer mount (now target: options), DockerProcedureContainer.ts, and SystemForEmbassy/index.ts (x2).

The SDK’s own IdMap type (SubContainer.ts) is unrelated and untouched — it still carries the idmap to start-container mount.

cargo check (gnu + musl), test-core (70), test-sdk, test-container-runtime (12) all green.

–no-verify: same pre-existing husky web-typecheck failure on master’s marketplace components as the merge commit; this change touches no web.

• refactor(effects/mount): keep SDK mount RPC; rename the CLI subcommand

Reverts the mount -> mount-pointer RPC rename, which leaked into the SDK (EffectCreator’s rpcRound union + call). Instead the new in-LXC idmap bind subcommand is renamed to avoid the collision, so the SDK stays clean (effects.mount still maps to the mount RPC).

• effects/mod.rs: mount is again the no_cli RPC handler (dependency::mount); the no_display CLI bind subcommand is now local-mount (local_mount::local_mount). Different names, no overlap in the shared handler tree.
• EffectCreator.ts: rpcRound(“mount”, …) restored; dropped the “mount-pointer” union member and the explanatory comment.
• SubContainer.ts: both start-container invocations now call local-mount.

Validated on a 6.19 kernel: start-container local-mount --idmap 0:100000:65536 on a uid-0 tree -> uid 100000; start-container mount now correctly errors (unrecognized subcommand). cargo check gnu+musl, container-runtime tsc + jest (12), test-sdk all green.

–no-verify: same pre-existing husky web-typecheck failure on master’s marketplace components; this change touches no web.

• refactor(effects): rename local-mount subcommand to bind-mount

The subcommand only ever does a bind: it open_tree(OPEN_TREE_CLONE)-clones an existing source path and move_mounts it onto a target (optionally idmapped via open_tree_attr, optionally read-only). It never mounts a fresh filesystem. bind-mount names that narrow scope accurately; bare bind would collide with the net bind effect.

Renames the subcommand string, the handler (local_mount -> bind_mount, LocalMountParams -> BindMountParams), and the module/file for consistency, plus the two start-container invocations in SubContainer.ts. No behavior change.

Validated: start-container bind-mount --idmap 0:100000:65536 on a uid-0 tree -> uid 100000. cargo check gnu+musl, container-runtime tsc, test-sdk

• test-container-runtime green.

• refactor(mount/idmapped): unmount staging guard explicitly

Per review: staging.unmount().await? instead of drop(staging). The explicit unmount awaits completion (staging is gone before mount() returns) and surfaces failures, vs Drop’s fire-and-forget tokio::spawn that only log_errs and runs later.