Plugin Framework¶

Plugin Framework is an API for registering custom extensions (plugins) to be executed on the driver and executors.

Plugin Framework uses the following main abstractions:

• PluginContainer
• SparkPlugin

Plugin Framework was introduced in Spark 2.4.4 (that only offered an API for executors) with further changes in Spark 3.0.0 (to cover the driver).

Resources¶

• Advanced Instrumentation in the official documentation of Apache Spark
• Commit for SPARK-29397
• Spark Plugin Framework in 3.0 - Part 1: Introduction by Madhukara Phatak
• Spark Memory Monitor by squito
• SparkPlugins by Luca Canali (CERN)