PluginContainer¶

PluginContainer is an abstraction of plugin containers that can registerMetrics (for the driver and executors).

PluginContainer is created for the driver and executors using apply utility.

Contract¶

Listening to Task Failures¶

onTaskFailed(
  failureReason: TaskFailedReason): Unit

For ExecutorPluginContainer only

Used when:

TaskRunner is requested to run (and the task has failed)

Listening to Task Start¶

onTaskStart(): Unit

For ExecutorPluginContainer only

Used when:

TaskRunner is requested to run (and the task has just started)

Listening to Task Success¶

onTaskSucceeded(): Unit

For ExecutorPluginContainer only

Used when:

TaskRunner is requested to run (and the task has finished successfully)

Registering Metrics¶

registerMetrics(
  appId: String): Unit

Registers metrics for the application ID

For DriverPluginContainer only

Used when:

SparkContext is created

Shutdown¶

shutdown(): Unit

Used when:

SparkContext is requested to stop
Executor is requested to stop

Implementations¶

Sealed Abstract Class

PluginContainer is a Scala sealed abstract class which means that all of the implementations are in the same compilation unit (a single file).

Creating PluginContainer¶

// the driver
apply(
  sc: SparkContext,
  resources: java.util.Map[String, ResourceInformation]): Option[PluginContainer]
// executors
apply(
  env: SparkEnv,
  resources: java.util.Map[String, ResourceInformation]): Option[PluginContainer]
// private helper
apply(
  ctx: Either[SparkContext, SparkEnv],
  resources: java.util.Map[String, ResourceInformation]): Option[PluginContainer]

apply creates a PluginContainer for the driver or executors (based on the type of the first input argument, i.e. SparkContext or SparkEnv, respectively).

apply first loads the SparkPlugins defined by spark.plugins configuration property.

Only when there was at least one plugin loaded, apply creates a DriverPluginContainer or ExecutorPluginContainer.

apply is used when:

SparkContext is created
Executor is created