Skip to content

HistoryServer

HistoryServer is an extension of the web UI for reviewing event logs of running (active) and completed Spark applications with event log collection enabled (based on spark.eventLog.enabled configuration property).

Starting HistoryServer Standalone Application

main(
  argStrings: Array[String]): Unit

main creates a HistoryServerArguments (with the given argStrings arguments).

main initializes security.

main creates an ApplicationHistoryProvider (based on spark.history.provider configuration property).

main creates a HistoryServer (with the ApplicationHistoryProvider and spark.history.ui.port configuration property) and requests it to bind.

main requests the ApplicationHistoryProvider to start.

main registers a shutdown hook that requests the HistoryServer to stop and sleeps...till the end of the world (giving the daemon thread a go).

Creating Instance

HistoryServer takes the following to be created:

When created, HistoryServer initializes itself.

HistoryServer is created when HistoryServer standalone application is started.

ApplicationCacheOperations

HistoryServer is a ApplicationCacheOperations.

UIRoot

HistoryServer is a UIRoot.

Initializing HistoryServer

initialize(): Unit

initialize is part of the WebUI abstraction.

initialize...FIXME

Attaching SparkUI

attachSparkUI(
  appId: String,
  attemptId: Option[String],
  ui: SparkUI,
  completed: Boolean): Unit

attachSparkUI is part of the ApplicationCacheOperations abstraction.

attachSparkUI...FIXME

Spark UI

getAppUI(
  appId: String,
  attemptId: Option[String]): Option[LoadedAppUI]

getAppUI is part of the ApplicationCacheOperations abstraction.

getAppUI requests the ApplicationHistoryProvider for the Spark UI of a Spark application (based on the appId and attemptId).

Logging

Enable ALL logging level for org.apache.spark.deploy.history.HistoryServer logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.deploy.history.HistoryServer=ALL

Refer to Logging.