LocalCheckpointRDD¶

LocalCheckpointRDD[T] is a CheckpointRDD.

Creating Instance¶

LocalCheckpointRDD takes the following to be created:

• RDD
• SparkContext
• RDD ID
• Number of Partitions

LocalCheckpointRDD is created when:

• LocalRDDCheckpointData is requested to doCheckpoint

Partitions¶

getPartitions: Array[Partition]

getPartitions is part of the RDD abstraction.

getPartitions creates a CheckpointRDDPartition for every input partition (index).

Computing Partition¶

compute(
  partition: Partition,
  context: TaskContext): Iterator[T]

compute is part of the RDD abstraction.

compute merely throws an SparkException (that explains the reason):

Checkpoint block [RDDBlockId] not found! Either the executor
that originally checkpointed this partition is no longer alive, or the original RDD is
unpersisted. If this problem persists, you may consider using `rdd.checkpoint()`
instead, which is slower than local checkpointing but more fault-tolerant."