Skip to content

BlockId

BlockId is an abstraction of data block identifiers based on an unique name.

Contract

Name

name: String

A globally unique identifier of this Block

Used when:

Implementations

Sealed Abstract Class

BlockId is a Scala sealed abstract class which means that all of the implementations are in the same compilation unit (a single file).

BroadcastBlockId

BlockId for broadcast variable blocks:

  • broadcastId identifier
  • Optional field name (default: empty)

Uses broadcast_ prefix for the name

Used when:

RDDBlockId

BlockId for RDD partitions:

  • rddId identifier
  • splitIndex identifier

Uses rdd_ prefix for the name

Used when:

Compressed when spark.rdd.compress configuration property is enabled

ShuffleBlockBatchId

ShuffleBlockId

BlockId for shuffle blocks:

  • shuffleId identifier
  • mapId identifier
  • reduceId identifier

Uses shuffle_ prefix for the name

Used when:

Compressed when spark.shuffle.compress configuration property is enabled

ShuffleDataBlockId

ShuffleIndexBlockId

StreamBlockId

BlockId for ...FIXME:

  • streamId
  • uniqueId

Uses the following name:

input-[streamId]-[uniqueId]

Used in Spark Streaming

TaskResultBlockId

TempLocalBlockId

TempShuffleBlockId

TestBlockId

Creating BlockId by Name

apply(
  name: String): BlockId

apply creates one of the available BlockIds by the given name (that uses a prefix to differentiate between different BlockIds).

apply is used when:

Back to top