Skip to main content
ChecklistFundamentalsWorkspacesData & I/O

Migrating from cloud VMs

Move a workload off hand-managed instances without losing reproducibility.

0 of 8 checked(not saved, download a copy to keep)

Download markdown

1Inventory what you have

Write down the current environment
Python version, every pinned dependency, and any system package your code relies on. Undocumented environments are what make migrations drag.
Locate every input and output path
List where data is read from and written to today. Local filesystem paths need a replacement; object storage paths usually carry over unchanged.
Note anything that is not Python
Shell steps, cron entries, and compiled binaries in your pipeline need explicit handling rather than an assumption they will come along.

2Move it

Reproduce the environment in a workspace
Install the same dependencies and confirm your existing code runs unchanged on a small input before touching anything else.
Repoint paths at shared storage
Replace local paths with object storage or workspace-persistent locations, so every worker can reach the same data.
Verify results match the old pipeline
Run the same input through both and compare outputs. Do this before distributing anything, because debugging correctness and parallelism at once is twice the work.

3Then distribute

Distribute only after the serial version is correct
Get it running correctly on one worker first. Only then wrap the per-unit work in a distributed function.
Decommission the old instances
Once the migrated pipeline has run clean a few times, shut the VMs down. Parallel infrastructure kept "just in case" is a standing bill.