ShuffleExecutorComponents¶

ShuffleExecutorComponents is an abstraction of executor shuffle builders.

Contract¶

createMapOutputWriter¶

ShuffleMapOutputWriter createMapOutputWriter(
  int shuffleId,
  long mapTaskId,
  int numPartitions) throws IOException

Creates a ShuffleMapOutputWriter

Used when:

• BypassMergeSortShuffleWriter is requested to write records
• UnsafeShuffleWriter is requested to mergeSpills and mergeSpillsUsingStandardWriter
• SortShuffleWriter is requested to write records

createSingleFileMapOutputWriter¶

Optional<SingleSpillShuffleMapOutputWriter> createSingleFileMapOutputWriter(
  int shuffleId,
  long mapId) throws IOException

Creates a SingleSpillShuffleMapOutputWriter

Default: empty

Used when:

• UnsafeShuffleWriter is requested to mergeSpills

initializeExecutor¶

void initializeExecutor(
  String appId,
  String execId,
  Map<String, String> extraConfigs);

Used when:

• SortShuffleManager utility is used to loadShuffleExecutorComponents

Implementations¶

• LocalDiskShuffleExecutorComponents