Configuration Properties¶
The following contains the configuration properties of EventLoggingListener and HistoryServer.
spark.eventLog.dir¶
Directory where Spark events are logged to (e.g. hdfs://namenode:8021/directory)
Default: /tmp/spark-events
The directory must exist before SparkContext is created.
spark.eventLog.buffer.kb¶
Size of the buffer to use when writing to output streams.
Default: 100
spark.eventLog.compress¶
Whether to enable (true) or disable (false) event compression (using a CompressionCodec)
Default: false
spark.eventLog.enabled¶
Whether to enable (true) or disable (false) persisting Spark events.
Default: false
spark.eventLog.logBlockUpdates.enabled¶
Whether EventLoggingListener should log RDD block updates (true) or not (false)
Default: false
spark.eventLog.overwrite¶
Whether to enable (true) or disable (false) deleting (or at least overwriting) an existing .inprogress event log files
Default: false
spark.history.fs.logDirectory¶
The directory for event log files. The directory has to exist before starting History Server.
Default: file:/tmp/spark-events
spark.history.kerberos.enabled¶
Whether to enable (true) or disable (false) security when working with HDFS with security enabled (Kerberos).
Default: false
spark.history.kerberos.keytab¶
Keytab to use for login to Kerberos. Required when spark.history.kerberos.enabled is enabled.
Default: (empty)
spark.history.kerberos.principal¶
Kerberos principal. Required when spark.history.kerberos.enabled is enabled.
Default: (empty)
spark.history.provider¶
Fully-qualified class name of an ApplicationHistoryProvider for HistoryServer.
Default: org.apache.spark.deploy.history.FsHistoryProvider
spark.history.store.path¶
Local directory where to cache application history information (by )
Default: (undefined) (i.e. all history information will be kept in memory)
spark.history.retainedApplications¶
How many Spark applications HistoryServer should retain
Default: 50
spark.history.ui.maxApplications¶
How many Spark applications HistoryServer should show in the UI
Default: (unbounded)
spark.history.ui.port¶
The port of History Server's web UI.
Default: 18080