https://haojian.github.io/DSC102SP24/static_files/presentations/10DataflowSystems.pdf

MapReduce

A programming model for parallel programs on sharded data (partitioned across multiple shared-nothing servers) & distributed system architecture

• System handles data distribution, parallelization, fault tolerance under the hood

• Map(): process one record independently

  • could be batch of multiple examples
  • dependences across mappers not allowed
  • allows for diff. input & output data types

• Reduce(): gather all Map outputs across workers sharing same key into an iterator

  • agg. function on iterator

• Ex: count word occurences in corpus

  

  each mapper and reducer is a separate process, reducers face barrier synchronization by Bulk Synchronous Parallelism

Pros

• Map() and Reduce() are highly general for diff. data structures, ETL
• Native scalability, large cluster parallelism
• Fault Tolerance automatically handled
• some jobs are map only (Reduce() not needed where no-cross shared agg. not necessary)

Cons

• Map operates record-wise; Reduce aggregates globally

Ex: emulating in SQL

Spark

Dataflow programming model inspired by Pandas chaining functions that exploits distributed memory to cache data

Architecture

• Driver program (spark context) communicates to cluster manager and executors
• Cluster manager communicates with worker nodes