https://haojian.github.io/DSC102SP24/static_files/presentations/9DataParallelismReplication.pdf

Execution Optimization Tradeoffs

Common practices

• Replication: put a shard on >1 workers
• Caching: store as much data as possible on DRAM / Disk
• Async: more common in ML systems (for params)
• Approximation: carefully subsample data

Hybrid Parallelism

Task vs Data-Parallelism have pros and cons

• Task wastes memory/storage due to replication (DRY)
• Data-par is painful to implement but scales w/o wasting memory/storage → higher network costs

Best of both worlds: run task-par on sharded data

• most scalable data systems today support only full task-par (Dask) or full data-par (RDBMS)

Review