Spark Configuration Properties¶

spark.dynamicAllocation.cachedExecutorIdleTimeout¶

How long (in seconds) to keep blocks cached

Default: The largest value representable as an Int

Must be >= 0

Used when:

• ExecutorMonitor is created
• RDD is requested to localCheckpoint (simply to print out a WARN message)

spark.dynamicAllocation.enabled¶

Default: false

Used when:

• Utils utility is requested to isDynamicAllocationEnabled
• SparkSubmitArguments is requested to loadEnvironmentArguments (and validates numExecutors argument)
• RDD is requested to localCheckpoint
• DAGScheduler is requested to checkBarrierStageWithDynamicAllocation

spark.dynamicAllocation.executorAllocationRatio¶

Default: 1.0

Must be between 0 (exclusive) and 1.0 (inclusive)

Used when:

• ExecutorAllocationManager is created

spark.dynamicAllocation.executorIdleTimeout¶

Default: 60

spark.dynamicAllocation.initialExecutors¶

Default: spark.dynamicAllocation.minExecutors

spark.dynamicAllocation.maxExecutors¶

Default: The largest value representable as an Int

spark.dynamicAllocation.minExecutors¶

Default: 0

spark.dynamicAllocation.schedulerBacklogTimeout¶

(in seconds)

Default: 1

spark.dynamicAllocation.shuffleTracking.enabled¶

Default: false

Used when:

• ExecutorMonitor is created

spark.dynamicAllocation.shuffleTracking.timeout¶

(in millis)

Default: The largest value representable as an Int

spark.dynamicAllocation.sustainedSchedulerBacklogTimeout¶

Default: spark.dynamicAllocation.schedulerBacklogTimeout