ENGINEERING SUPPORT

Preserve evidence first, then narrow down the cloud Mac issue

This engineering guide follows an execution-first workflow. Verify the device and network, then check Xcode, signing, and CI before handling storage, upgrades, and recovery—so you avoid repeatedly testing multiple variables at once.

6 categories Engineering issue entry points
5 Available nodes
365 days Node uptime
Diagnostic runbook RUN / SUPPORT
Check in dependency order

Connection → Environment → Build → Data

READY
01 / Device
Device ID, node, host fingerprint
02 / Network
Local access, routing, ports, and latency
03 / Toolchain
macOS, Xcode, certificates, and dependencies
04 / Task
Reproduction command, logs, exit code, and timestamp
Before opening a ticket Redact logs and record completed steps
GUIDE INDEX

Start at your current blocker—there’s no need to read from the beginning

Six entry points cover identity checks, system changes, developer tools, automated jobs, network paths, and order details. Each explains what to check first, what to preserve, and when to open a ticket.

CONNECTION ORDER

When connection fails, verify the target machine first, then the connection tool

Do not switch clients repeatedly before verifying the host address, port, and credentials. Complete the following four steps in order; each provides verifiable input for the next.

  1. 01

    Read device details

    Sign in to the console and verify the device ID, selected node, host address, SSH port, and current credentials in the relevant order. When using multiple devices in parallel, map each order number to its device ID first.

    Preserve Order number, device ID, node, port
  2. 02

    Verify the host fingerprint

    On the first connection, compare the fingerprint shown by the client with the record in the console. If the fingerprint changes after a device rebuild, confirm the change record first, then remove the old local entry—never ignore the warning.

    ssh-keygen -R example-host
    ssh -p 22 user@example-host
  3. 03

    Establish an SSH baseline first

    From a wired connection, test DNS, the port, and SSH first. After success, record the login time, egress network, and command-line response. If it fails, preserve the complete error instead of capturing only the “connection failed” line.

    Interpretation A timeout usually points to routing or the port; connection refused usually points to the target service; authentication failures should first prompt a check of credentials and permissions.
  4. 04

    Configure the graphical connection next

    Only after the SSH baseline is stable should you verify the address, port, client version, and local firewall required by the graphical interface. If the display stutters, also record the resolution, encoding settings, and network latency.

    Boundary A working graphical interface does not guarantee that high-bitrate media previews will match local display performance. Validate them against the actual network path.
NETWORK MEASUREMENT

Compare median ping latency across five nodes using the same methodology

The table illustrates routing differences from major cities to Singapore, Japan (Tokyo), South Korea (Seoul), Hong Kong, and the western United States. These figures are reference measurements, not guarantees for application throughput, graphical-interface frame rates, or task completion time.

Test window Weekdays, 14:00–16:00 UTC+8
Sample size 50 tests per node
Access method Gigabit wired network
Metric Median round-trip latency
Reference median ping from three major cities to five VMDebug nodes, in milliseconds
Test source Carrier Singapore Japan (Tokyo) South Korea (Seoul) Hong Kong Western United States
Shanghai China Telecom 71 ms 42 ms 46 ms 34 ms 141 ms
Shenzhen China Unicom 44 ms 55 ms 59 ms 18 ms 157 ms
Beijing China Mobile 91 ms 48 ms 39 ms 63 ms 138 ms
Test locally first

Rule out Wi-Fi and local egress fluctuations

Use a direct wired connection and pause large-file sync, video calls, and system updates. Test the local gateway and node address continuously; if both fluctuate, resolve local access issues first.

Then inspect the route

The median does not explain the entire experience

Remote interaction is also affected by packet loss, jitter, upstream bandwidth, and client encoding. When reporting a network issue, include the carrier, city, test window, sample size, and traceroute results.

XCODE DIAGNOSTICS

When signing fails, don’t delete the environment first—narrow it down across five dependency layers

The same error may result from an unavailable certificate, a mismatched provisioning profile, missing Keychain access, contaminated caches, or different build parameters. Save the raw logs first, then check the following in order.

  1. 01

    Certificate

    Confirm that the target certificate exists and has not expired, that it pairs with the private key, and that the current build user can read it. List signing identities first; do not immediately re-import all materials.

    security find-identity -v -p codesigning
  2. 02

    Provisioning Profile

    Check the Bundle Identifier, team identifier, certificate type, device scope, and capability declarations. For manually signed projects, confirm that the configuration points to the target file rather than an old cache.

    CODE_SIGN_STYLE / PROVISIONING_PROFILE_SPECIFIER
  3. 03

    Keychain permissions

    The CI user and interactive login user may use different Keychain search lists. Confirm that the target Keychain is unlocked, included in the search path, and allows the build process to read the private key.

    security list-keychains
  4. 04

    DerivedData

    Record the current path and failed task first, then clear only the affected project’s derived data. Do not delete all caches as a first step, or you will lose comparable build evidence.

    xcodebuild -showBuildSettings
  5. 05

    Build logs

    Save the complete command, Scheme, Configuration, SDK, Xcode version, exit code, and first failure location. Prioritize the first root-cause error, not the final summary line.

    xcodebuild -version
CI/CD RUNNER

Treat the runner as a reproducible run, not a working directory that accumulates indefinitely

A dedicated physical Mac can run build jobs continuously, but stability still depends on identity, directories, caches, secrets, and rollback boundaries. Every change should answer: “What changed, how was it verified, and how can it be reverted?”

Runner integration checklist 5 CHECKS
  1. 01

    Register the runner identity

    Give each device a recognizable runner name and set of tags, and record its registration scope, service user, and startup method. Avoid names that cannot distinguish multiple runners.

  2. 02

    Isolate working directories

    Create separate directories by repository, branch, or job. Do not let concurrent jobs write to the same DerivedData, archive directory, or dependency output path.

  3. 03

    Set cache boundaries

    Separate rebuildable caches from artifacts that must be retained. Before cleaning, record directory size, last-use time, and job ownership to avoid a cold start caused by deleting everything.

  4. 04

    Inject secrets at runtime

    Expose secrets to the process environment or a temporary Keychain only when the job needs them. Disable echoing in logs, delete temporary materials after the job, and lock down access.

  5. 05

    Define failure rollback

    Keep version records before updating the runner, Xcode, or dependencies. If validation fails, restore configuration files, toolchain selection, and cache indexes instead of layering on more changes.

Working directory

Generate a unique path for every job

The path should include the project ID, job ID, and attempt number. Cleanup may target only the current job directory, preventing accidental deletion of concurrent builds.

WORK_ROOT="$HOME/ci-work"
JOB_DIR="$WORK_ROOT/$PROJECT/$RUN_ID"
mkdir -p "$JOB_DIR"
Failure evidence

Archive four categories of information before exit

Save the runner version, toolchain version, complete exit code, and first root-cause log. Include cache hit rate and free disk space in the job summary to make before-and-after comparisons easier.

  • Runner and macOS versions
  • Xcode path and version
  • Command, exit code, and logs
  • Free disk space and cache status
STORAGE & TB5

Create a recoverable copy before mounting additional storage, and map the topology before connecting devices in parallel

Additional storage and Thunderbolt 5 connections change data paths and permission boundaries. Before formatting, changing a mount point, or disconnecting a device, confirm the data copy and that jobs have stopped.

Additional storage

Pre-mount checks

  1. Confirm a data copy

    Keep at least one independent copy of important code, media, build artifacts, and configuration. A mount that has not been verified must not be the only storage location.

  2. Record disk identity

    Record the volume name, file system, capacity, device ID, and intended mount point. Do not identify the target disk by display order alone.

  3. Check access permissions

    Confirm that the user running build or media jobs has the required directory permissions. Do not bypass ownership issues by globally broadening permissions.

  4. Perform read/write validation

    Use a disposable test file to verify creation, reading, renaming, and deletion before moving the real working directory.

Thunderbolt 5

Parallel-device connection order

  1. Map the physical topology

    Mark each Mac, cable direction, shared storage device, and job role. Do not change connections when upstream and downstream devices cannot be identified.

  2. Unify permission boundaries

    Define which device writes and which devices are read-only, and give automated jobs separate directories to prevent concurrent overwrites.

  3. Validate each link individually

    After adding each device, complete a connection, permission, and read/write test. Do not wire everything first and troubleshoot afterward.

  4. Disconnect in reverse job order

    Stop writes and build jobs first, confirm that caches have been flushed, then unmount the volume and disconnect devices from the end of the chain.

UPGRADE & RECOVERY

System upgrades need a validation machine, a baseline, and rollback materials

All nodes run normally year-round, 365 days a year. macOS and toolchain upgrades are scheduled by users around their workloads. Validate changes on non-critical jobs first, then roll them out to long-running runners.

Upgrade run sequence 6 PHASES
  1. 01

    Create a snapshot-style inventory

    Export code state, dependency lock files, the Homebrew list, Xcode path, certificate names, Keychain list, runner configuration, and free disk space.

  2. 02

    Back up non-rebuildable data

    Move private configuration, signing materials, build artifacts, and project data to an independent location, and verify that at least one file can actually be restored.

  3. 03

    Verify toolchain compatibility

    Check compatibility across the target macOS, Xcode, command-line tools, package manager, runner, and project dependencies. Record any older versions that must be retained.

  4. 04

    Pause write jobs

    Stop CI queues, media processing, and data synchronization. Confirm that no process is still writing to working directories, caches, or additional storage.

  5. 05

    Complete minimum acceptance checks

    After the upgrade, verify SSH, disks, the Xcode version, certificate access, dependency restoration, tests, archiving, and artifact downloads in sequence.

  6. 06

    Record the new baseline

    Save the duration, logs, cache status, and free disk space from the first successful job, then gradually restore concurrent jobs. Do not release the entire queue at once.

Stop conditions

Roll back if any of these occur

  • A required certificate or private key cannot be read
  • The Xcode version required by the project is unavailable
  • Additional storage becomes read-only or fails to mount
  • The same baseline command produces a new consistent failure

Do not upgrade multiple components in succession before the root cause is clear, or the logs will not distinguish system, toolchain, and project-configuration effects.

Ticket materials

Give support everything needed to reproduce the issue

  • Order number and device ID
  • Time of issue and selected node
  • macOS, Xcode, and runner versions
  • Minimum reproduction steps and expected result
  • Complete redacted logs and exit code
  • Checks completed and their results
Submit a console ticket
NEXT ACTION

Your logs are redacted—hand the runbook over to support

For issues with an existing order, submit a ticket in the console with the device ID, reproduction steps, and complete error context. For configuration questions before ordering, contact the dedicated support email.

support@vmdebug.com