Skip to content

AccumulatorV2

AccumulatorV2[IN, OUT] is an abstraction of accumulators

AccumulatorV2 is a Java Serializable.

Contract

Adding Value

add(
  v: IN): Unit

Accumulates (adds) the given v value to this accumulator

Copying Accumulator

copy(): AccumulatorV2[IN, OUT]

Is Zero Value

isZero: Boolean

Merging Updates

merge(
  other: AccumulatorV2[IN, OUT]): Unit

Resetting Accumulator

reset(): Unit

Value

value: OUT

The current value of this accumulator

Used when:

Implementations

Converting this Accumulator to AccumulableInfo

toInfo(
  update: Option[Any],
  value: Option[Any]): AccumulableInfo

toInfo determines whether the accumulator is internal based on the name (and whether it uses the internal.metrics prefix) and uses it to create an AccumulableInfo.

toInfo is used when:

Registering Accumulator

register(
  sc: SparkContext,
  name: Option[String] = None,
  countFailedValues: Boolean = false): Unit

register...FIXME

register is used when:

Serializing AccumulatorV2

writeReplace(): Any

writeReplace is part of the Serializable (Java) abstraction (to designate an alternative object to be used when writing an object to the stream).

writeReplace...FIXME

Deserializing AccumulatorV2

readObject(
  in: ObjectInputStream): Unit

readObject is part of the Serializable (Java) abstraction (for special handling during deserialization).

readObject reads the non-static and non-transient fields of the AccumulatorV2 from the given ObjectInputStream.

If the atDriverSide internal flag is turned on, readObject turns it off (to indicate readObject is executed on an executor). Otherwise, atDriverSide internal flag is turned on.

readObject requests the active TaskContext to register this accumulator.

Back to top