# Data & I/O

Reading and writing large datasets without I/O becoming the bottleneck.

Source: https://university.eugo.io/topics/data-io

---

## Courses

- [Scaling a real workload](https://university.eugo.io/courses/scaling-a-real-workload.md): Take a single-machine geospatial pipeline to a cluster, and find out where the time actually goes.

## Videos

- [Parallel pandas in practice](https://university.eugo.io/videos/parallel-pandas-in-practice.md): Split a dataframe workload across workers without rewriting your analysis.
- [Reading large datasets without stalling](https://university.eugo.io/videos/reading-large-datasets.md): When I/O is your bottleneck, more workers will not help. Here is what does.
- [Seven terabytes in three minutes](https://university.eugo.io/videos/seven-terabytes-in-three-minutes.md): A real geospatial pipeline: 192,000 satellite files, resampled and written, start to finish.

## Checklists and guides

- [Migrating from cloud VMs](https://university.eugo.io/resources/migrating-from-cloud-vms-checklist.md): Move a workload off hand-managed instances without losing reproducibility.

## Vocabulary

- **Columnar format** — A file format that stores values grouped by column rather than by row, so a reader can fetch only the fields it needs.
- **I/O-bound** — A workload whose runtime is limited by reading and writing data rather than by calculation. Adding workers to an I/O-bound job adds waiting, not throughput.
- **Object storage** — Storage reachable by every node in a cluster. Inputs must live here, or in workspace storage, rather than on one machine’s local disk.
- **Partition** — One independently processable chunk of a dataset. Partition size is the main lever on how well a workload parallelizes.
- **Persistent storage** — Workspace storage that survives between interactive sessions. Files saved elsewhere in a session are lost when it ends.