ShuffleWriter¶

ShuffleWriter[K, V] (of K keys and V values) is an abstraction of shuffle writers that can write out key-value records (of a RDD partition) to a shuffle system.

ShuffleWriter is used when ShuffleMapTask is requested to run (and uses a ShuffleWriteProcessor to write partition records to a shuffle system).

Contract¶

Writing Out Partition Records to Shuffle System¶

write(
  records: Iterator[Product2[K, V]]): Unit

Writes key-value records (of a partition) out to a shuffle system

Used when:

• ShuffleWriteProcessor is requested to write

Stopping ShuffleWriter¶

stop(
  success: Boolean): Option[MapStatus]

Stops (closes) the ShuffleWriter and returns a MapStatus if the writing completed successfully. The success flag is the status of the task execution.

Used when:

• ShuffleWriteProcessor is requested to write

Implementations¶

• BypassMergeSortShuffleWriter
• SortShuffleWriter
• UnsafeShuffleWriter