BlockTransferService¶

BlockTransferService is an extension of the BlockStoreClient abstraction for shuffle clients that can fetch and upload blocks of data (synchronously or asynchronously).

BlockTransferService is a network service available by a host name and a port.

BlockTransferService was introduced in SPARK-3019 Pluggable block transfer interface (BlockTransferService).

Contract¶

Host Name¶

hostName: String

Host name this service is listening on

Used when:

• BlockManager is requested to initialize

Initializing¶

init(
  blockDataManager: BlockDataManager): Unit

Used when:

• BlockManager is requested to initialize

Port¶

port: Int

Used when:

• BlockManager is requested to initialize

Uploading Block Asynchronously¶

uploadBlock(
  hostname: String,
  port: Int,
  execId: String,
  blockId: BlockId,
  blockData: ManagedBuffer,
  level: StorageLevel,
  classTag: ClassTag[_]): Future[Unit]

Used when:

• BlockTransferService is requested to uploadBlockSync

Implementations¶

• NettyBlockTransferService

Uploading Block Synchronously¶

uploadBlockSync(
  hostname: String,
  port: Int,
  execId: String,
  blockId: BlockId,
  blockData: ManagedBuffer,
  level: StorageLevel,
  classTag: ClassTag[_]): Unit

uploadBlockSync uploadBlock and waits till it finishes.

uploadBlockSync is used when:

• BlockManager is requested to replicate
• ShuffleMigrationRunnable is requested to run