MetricsConfig¶

MetricsConfig is the configuration of the MetricsSystem (i.e. metrics sources and sinks).

MetricsConfig is <> when MetricsSystem is.

MetricsConfig uses metrics.properties as the default metrics configuration file. It is configured using spark-metrics-properties.md#spark.metrics.conf[spark.metrics.conf] configuration property. The file is first loaded from the path directly before using Spark's CLASSPATH.

MetricsConfig accepts a metrics configuration using spark.metrics.conf.-prefixed configuration properties.

Spark comes with conf/metrics.properties.template file that is a template of metrics configuration.

MetricsConfig <> that the <> are always defined.

[[default-properties]] .MetricsConfig's Default Metrics Properties [cols="1,2",options="header",width="100%"] |=== | Name | Description

| *.sink.servlet.class | org.apache.spark.metrics.sink.MetricsServlet

| *.sink.servlet.path | /metrics/json

| master.sink.servlet.path | /metrics/master/json

| applications.sink.servlet.path | /metrics/applications/json |===

[NOTE]¶

The order of precedence of metrics configuration settings is as follows:

. <> . spark-metrics-properties.md#spark.metrics.conf[spark.metrics.conf] configuration property or metrics.properties configuration file . spark.metrics.conf.-prefixed Spark properties ====

[[creating-instance]] [[conf]] MetricsConfig takes a SparkConf.md[SparkConf] when created.

[[internal-registries]] .MetricsConfig's Internal Registries and Counters [cols="1,2",options="header",width="100%"] |=== | Name | Description

| [[properties]] properties | https://docs.oracle.com/javase/8/docs/api/java/util/Properties.html[java.util.Properties] with metrics properties

Used to <> per-subsystem's <>.

| [[perInstanceSubProperties]] perInstanceSubProperties | Lookup table of metrics properties per subsystem |===

=== [[initialize]] Initializing MetricsConfig -- initialize Method

[source, scala]¶

initialize(): Unit¶

initialize <> and <> (that is defined using spark-metrics-properties.md#spark.metrics.conf[spark.metrics.conf] configuration property).

initialize takes all Spark properties that start with spark.metrics.conf. prefix from <> and adds them to <> (without the prefix).

In the end, initialize splits <> with the default configuration (denoted as *) assigned to all subsystems afterwards.

NOTE: initialize accepts * (star) for the default configuration or any combination of lower- and upper-case letters for Spark subsystem names.

NOTE: initialize is used exclusively when MetricsSystem is created.

=== [[setDefaultProperties]] setDefaultProperties Internal Method

[source, scala]¶

setDefaultProperties(prop: Properties): Unit¶

setDefaultProperties sets the <> (in the input prop).

NOTE: setDefaultProperties is used exclusively when MetricsConfig <>.

=== [[loadPropertiesFromFile]] Loading Custom Metrics Configuration File or metrics.properties -- loadPropertiesFromFile Method

[source, scala]¶

loadPropertiesFromFile(path: Option[String]): Unit¶

loadPropertiesFromFile tries to open the input path file (if defined) or the default metrics configuration file metrics.properties (on CLASSPATH).

If either file is available, loadPropertiesFromFile loads the properties (to <> registry).

In case of exceptions, you should see the following ERROR message in the logs followed by the exception.

ERROR Error loading configuration file [file]

NOTE: loadPropertiesFromFile is used exclusively when MetricsConfig <>.

=== [[subProperties]] Grouping Properties Per Subsystem -- subProperties Method

[source, scala]¶

subProperties(prop: Properties, regex: Regex): mutable.HashMap[String, Properties]¶

subProperties takes prop properties and destructures keys given regex. subProperties takes the matching prefix (of a key per regex) and uses it as a new key with the value(s) being the matching suffix(es).

[source, scala]¶

driver.hello.world => (driver, (hello.world))¶

NOTE: subProperties is used when MetricsConfig <> (to apply the default metrics configuration) and when MetricsSystem registers metrics sources and sinks.

=== [[getInstance]] getInstance Method

[source, scala]¶

getInstance(inst: String): Properties¶

getInstance...FIXME

NOTE: getInstance is used when...FIXME