TaskMetrics¶

TaskMetrics is a collection of metrics (accumulators) tracked during execution of a task.

Creating Instance¶

TaskMetrics takes no input arguments to be created.

TaskMetrics is created when:

• Stage is requested to makeNewStageAttempt

Metrics¶

ShuffleWriteMetrics¶

ShuffleWriteMetrics

• shuffle.write.bytesWritten
• shuffle.write.recordsWritten
• shuffle.write.writeTime

ShuffleWriteMetrics is exposed using Dropwizard metrics system using ExecutorSource (when TaskRunner is about to finish running):

• shuffleBytesWritten
• shuffleRecordsWritten
• shuffleWriteTime

ShuffleWriteMetrics can be monitored using:

• StatsReportListener (when a stage completes)
  • shuffle bytes written
• JsonProtocol (when requested to taskMetricsToJson)
  • Shuffle Bytes Written
  • Shuffle Write Time
  • Shuffle Records Written

shuffleWriteMetrics is used when:

• ShuffleWriteProcessor is requested for a ShuffleWriteMetricsReporter
• SortShuffleWriter is created
• AppStatusListener is requested to handle a SparkListenerTaskEnd
• LiveTask is requested to updateMetrics
• ExternalSorter is requested to writePartitionedFile (to create a DiskBlockObjectWriter), writePartitionedMapOutput
• ShuffleExchangeExec (Spark SQL) is requested for a ShuffleWriteProcessor (to create a ShuffleDependency)

Memory Bytes Spilled¶

memoryBytesSpilled metric

memoryBytesSpilled is exposed using Dropwizard metrics system using ExecutorSource as memoryBytesSpilled.

incMemoryBytesSpilled is used when:

• Aggregator is requested to updateMetrics
• CoGroupedRDD is requested to compute
• BlockStoreShuffleReader is requested to read
• ShuffleExternalSorter is requested to spill
• ExternalSorter is requested to writePartitionedFile and writePartitionedMapOutput
• UnsafeExternalSorter is requested to createWithExistingInMemorySorter and spill
• SpillableIterator (of UnsafeExternalSorter) is requested to spill

TaskContext¶

TaskMetrics is available using TaskContext.taskMetrics.

TaskContext.get.taskMetrics

Serializable¶

TaskMetrics is a Serializable (Java).

Task¶

TaskMetrics is part of Task.

task.metrics

SparkListener¶

TaskMetrics is available using SparkListener and intercepting SparkListenerTaskEnd events.

StatsReportListener¶

StatsReportListener can be used for summary statistics at runtime (after a stage completes).

Spark History Server¶

Spark History Server uses EventLoggingListener to intercept post-execution statistics (incl. TaskMetrics).