Data & I/O

Reading and writing large datasets without I/O becoming the bottleneck.

Courses

Take a single-machine geospatial pipeline to a cluster, and find out where the time actually goes.

Split a dataframe workload across workers without rewriting your analysis.

When I/O is your bottleneck, more workers will not help. Here is what does.

A real geospatial pipeline: 192,000 satellite files, resampled and written, start to finish.

Move a workload off hand-managed instances without losing reproducibility.

The terms this subject uses, defined.

Columnar format : A file format that stores values grouped by column rather than by row, so a reader can fetch only the fields it needs.
I/O-bound : A workload whose runtime is limited by reading and writing data rather than by calculation. Adding workers to an I/O-bound job adds waiting, not throughput.
Object storage : Storage reachable by every node in a cluster. Inputs must live here, or in workspace storage, rather than on one machine’s local disk.
Partition : One independently processable chunk of a dataset. Partition size is the main lever on how well a workload parallelizes.
Persistent storage : Workspace storage that survives between interactive sessions. Files saved elsewhere in a session are lost when it ends.