Skip to content

ExecutorAllocationClient

ExecutorAllocationClient is an abstraction of schedulers that can communicate with a cluster manager to request or kill executors.

Contract

Active Executor IDs

getExecutorIds(): Seq[String]

Used when:

isExecutorActive

isExecutorActive(
  id: String): Boolean

Whether a given executor (by ID) is active (and can be used to execute tasks)

Used when:

  • FIXME

Killing Executors

killExecutors(
  executorIds: Seq[String],
  adjustTargetNumExecutors: Boolean,
  countFailures: Boolean,
  force: Boolean = false): Seq[String]

Requests a cluster manager to kill given executors and returns whether the request has been acknowledged by the cluster manager (true) or not (false).

Used when:

Killing Executors on Host

killExecutorsOnHost(
  host: String): Boolean

Used when:

Requesting Additional Executors

requestExecutors(
  numAdditionalExecutors: Int): Boolean

Requests additional executors from a cluster manager and returns whether the request has been acknowledged by the cluster manager (true) or not (false).

Used when:

Updating Total Executors

requestTotalExecutors(
  resourceProfileIdToNumExecutors: Map[Int, Int],
  numLocalityAwareTasksPerResourceProfileId: Map[Int, Int],
  hostToLocalTaskCount: Map[Int, Map[String, Int]]): Boolean

Updates a cluster manager with the exact number of executors desired. Returns whether the request has been acknowledged by the cluster manager (true) or not (false).

Used when:

Implementations

Killing Single Executor

killExecutor(
  executorId: String): Boolean

killExecutor kill the given executor.

killExecutor is used when:

Decommissioning Executors

decommissionExecutors(
  executorsAndDecomInfo: Array[(String, ExecutorDecommissionInfo)],
  adjustTargetNumExecutors: Boolean,
  triggeredByExecutor: Boolean): Seq[String]

decommissionExecutors kills the given executors.

decommissionExecutors is used when:

Decommissioning Single Executor

decommissionExecutor(
  executorId: String,
  decommissionInfo: ExecutorDecommissionInfo,
  adjustTargetNumExecutors: Boolean,
  triggeredByExecutor: Boolean = false): Boolean

decommissionExecutor...FIXME

decommissionExecutor is used when:

Back to top