HadoopWriteConfigUtil¶
HadoopWriteConfigUtil[K, V] is an abstraction of writer configurers for SparkHadoopWriter to write a key-value RDD (for RDD.saveAsNewAPIHadoopDataset and RDD.saveAsHadoopDataset operators).
Contract¶
assertConf¶
assertConf(
jobContext: JobContext,
conf: SparkConf): Unit
closeWriter¶
closeWriter(
taskContext: TaskAttemptContext): Unit
createCommitter¶
createCommitter(
jobId: Int): HadoopMapReduceCommitProtocol
Creates a HadoopMapReduceCommitProtocol committer
Used when:
SparkHadoopWriteris requested to write data out
createJobContext¶
createJobContext(
jobTrackerId: String,
jobId: Int): JobContext
createTaskAttemptContext¶
createTaskAttemptContext(
jobTrackerId: String,
jobId: Int,
splitId: Int,
taskAttemptId: Int): TaskAttemptContext
Creates a Hadoop TaskAttemptContext
initOutputFormat¶
initOutputFormat(
jobContext: JobContext): Unit
initWriter¶
initWriter(
taskContext: TaskAttemptContext,
splitId: Int): Unit
write¶
write(
pair: (K, V)): Unit
Writes out the key-value pair
Used when:
SparkHadoopWriteris requested to executeTask