OrderedRDDFunctions¶

class OrderedRDDFunctions[
  K: Ordering : ClassTag,
  V: ClassTag,
  P <: Product2[K, V] : ClassTag]

OrderedRDDFunctions adds extra operators to RDDs of (key, value) pairs (RDD[(K, V)]) where the K key is sortable (i.e. any key type K that has an implicit Ordering[K] in scope).

Tip

Learn more about Ordering in the Scala Standard Library documentation.

Creating Instance¶

OrderedRDDFunctions takes the following to be created:

RDD of Ps

OrderedRDDFunctions is created using RDD.rddToOrderedRDDFunctions implicit method.

filterByRange¶

filterByRange(
  lower: K,
  upper: K): RDD[P]

filterByRange...FIXME

repartitionAndSortWithinPartitions¶

repartitionAndSortWithinPartitions(
  partitioner: Partitioner): RDD[(K, V)]

repartitionAndSortWithinPartitions creates a ShuffledRDD with the given Partitioner.

Note

repartitionAndSortWithinPartitions is a generalization of sortByKey operator.

repartitionAndSortWithinPartitions is used when...FIXME

sortByKey¶

sortByKey(
  ascending: Boolean = true,
  numPartitions: Int = self.partitions.length): RDD[(K, V)]

sortByKey creates a ShuffledRDD (with the RDD and a RangePartitioner).

Note

sortByKey is a specialization of repartitionAndSortWithinPartitions operator.

Note

Spark uses sortByKey for RDD.sortBy operator.