Skip to content

ExternalBlockHandler

ExternalBlockHandler is an RpcHandler.

Creating Instance

ExternalBlockHandler takes the following to be created:

ExternalBlockHandler creates the following:

ExternalBlockHandler is created when:

  • ExternalShuffleService is requested for an ExternalBlockHandler
  • YarnShuffleService is requested to serviceInit

OneForOneStreamManager

ExternalBlockHandler can be given or creates an OneForOneStreamManager to be created.

ExternalShuffleBlockResolver

ExternalBlockHandler can be given or creates an ExternalShuffleBlockResolver to be created.

ExternalShuffleBlockResolver is used for the following:

ExternalShuffleBlockResolver is used for the following:

ExternalShuffleBlockResolver is closed when is ExternalBlockHandler.

Registered Executors File

ExternalBlockHandler can be given a Java's File (or null) to be created.

This file is simply to create an ExternalShuffleBlockResolver.

Messages

FetchShuffleBlocks

Request to read a set of blocks

"Posted" (created) when:

When received, ExternalBlockHandler requests the OneForOneStreamManager to registerStream (with a ShuffleManagedBufferIterator).

ExternalBlockHandler prints out the following TRACE message to the logs:

Registered streamId [streamId] with [numBlockIds] buffers for client [clientId] from host [remoteAddress]

In the end, ExternalBlockHandler responds with a StreamHandle (of streamId and numBlockIds).

GetLocalDirsForExecutors

OpenBlocks

Note

For backward compatibility and like FetchShuffleBlocks.

RegisterExecutor

RemoveBlocks

ShuffleMetrics

Executor Removed Notification

void executorRemoved(
  String executorId,
  String appId)

executorRemoved requests the ExternalShuffleBlockResolver to executorRemoved.

executorRemoved is used when:

Application Finished Notification

void applicationRemoved(
  String appId,
  boolean cleanupLocalDirs)

applicationRemoved requests the ExternalShuffleBlockResolver to applicationRemoved.

applicationRemoved is used when:

  • ExternalShuffleService is requested to applicationRemoved
  • YarnShuffleService (Spark on YARN) is requested to stopApplication

Logging

Enable ALL logging level for org.apache.spark.network.shuffle.ExternalBlockHandler logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.network.shuffle.ExternalBlockHandler=ALL

Refer to Logging.

Back to top