Spark Configuration Properties of External Shuffle Service¶

The following are configuration properties of External Shuffle Service.

spark.shuffle.service.db.enabled¶

Whether to use db in ExternalShuffleService. Note that this only affects standalone mode.

Default: true

Used when:

• ExternalShuffleService is requested for an ExternalBlockHandler
• Worker (Spark Standalone) is requested to handle a WorkDirCleanup message

spark.shuffle.service.enabled¶

Controls whether to use the External Shuffle Service

Default: false

Used when:

• BlacklistTracker is requested to updateBlacklistForFetchFailure
• ExecutorMonitor is created
• ExecutorAllocationManager is requested to validateSettings
• SparkEnv utility is requested to create a "base" SparkEnv
• ExternalShuffleService is created and started
• Worker (Spark Standalone) is requested to handle a WorkDirCleanup message or started
• ExecutorRunnable (Spark on YARN) is requested to startContainer

spark.shuffle.service.fetch.rdd.enabled¶

Enables ExternalShuffleService for fetching disk persisted RDD blocks.

When enabled with Dynamic Resource Allocation executors having only disk persisted blocks are considered idle after spark.dynamicAllocation.executorIdleTimeout and will be released accordingly.

Default: false

Used when:

• ExternalShuffleBlockResolver is created
• SparkEnv utility is requested to create a "base" SparkEnv
• ExecutorMonitor is created

spark.shuffle.service.port¶

Port of the external shuffle service

Default: 7337

Used when:

• ExternalShuffleService is created
• StorageUtils utility is requested for the port of an external shuffle service