Serializer¶

Serializer is an abstraction of serializers for serialization and deserialization of tasks (closures) and data blocks in a Spark application.

Contract¶

Creating New SerializerInstance¶

newInstance(): SerializerInstance

Creates a new SerializerInstance

Used when:

• Task is created (only used in tests)
• SerializerSupport (Spark SQL) utility is used to newSerializer
• RangePartitioner is requested to writeObject and readObject
• TorrentBroadcast utility is used to blockifyObject and unBlockifyObject
• TaskRunner is requested to run
• NettyBlockRpcServer is requested to deserializeMetadata
• NettyBlockTransferService is requested to uploadBlock
• PairRDDFunctions is requested to...FIXME
• ParallelCollectionPartition is requested to...FIXME
• RDD is requested to...FIXME
• ReliableCheckpointRDD utility is used to...FIXME
• NettyRpcEnvFactory is requested to create a RpcEnv
• DAGScheduler is created
• others

Implementations¶

• JavaSerializer
• KryoSerializer
• UnsafeRowSerializer (Spark SQL)

Accessing Serializer¶

Serializer is available using SparkEnv as the closureSerializer and serializer.

closureSerializer¶

SparkEnv.get.closureSerializer

serializer¶

SparkEnv.get.serializer

Serialized Objects Relocation Requirements¶

supportsRelocationOfSerializedObjects: Boolean

supportsRelocationOfSerializedObjects is disabled (false) by default.

supportsRelocationOfSerializedObjects is used when:

• BlockStoreShuffleReader is requested to fetchContinuousBlocksInBatch
• SortShuffleManager is requested to create a ShuffleHandle for a given ShuffleDependency (and checks out SerializedShuffleHandle requirements)