Skip to main content
Quick guideData scientistFundamentalsEugoIDEDistributed compute

Quick guide: data scientist

The short version of what a data scientist needs on day one: where code runs, where data has to live, and the one habit that keeps a cluster busy.

0 of 4 checked(not saved, download a copy to keep)

Download markdown

1Day one

Open a session
Your work happens in EugoIDE, a browser-based notebook environment. Files you save in workspace storage persist between sessions; anything else does not.
Run your existing code first
Before distributing anything, confirm your current analysis runs as-is. Most pandas and NumPy code needs no changes.
Learn three API calls
The distribute decorator turns a function into a distributed task, calling it returns a future, and get() resolves futures. That is enough for most workloads.

2The habit that matters most

Build the whole list of futures before waiting on any
Calling get() inside a loop runs your tasks one at a time. Collect every future first, then call get() once on the list.

More like this

All resources
Checklist

Workload readiness checklist

Verify a workload is worth distributing before you spend cluster time on it.

11 items
Checklist

Performance tuning checklist

Work through the usual causes when a distributed run is slower than expected.

9 items
Checklist

Migrating from cloud VMs

Move a workload off hand-managed instances without losing reproducibility.

8 items