MemoryConsumer¶

MemoryConsumer is an abstraction of spillable memory consumers (of TaskMemoryManager).

MemoryConsumers correspond to individual operators and data structures within a task. TaskMemoryManager receives memory allocation requests from MemoryConsumers and issues callbacks to consumers in order to trigger spilling when running low on memory.

A MemoryConsumer basically tracks how much memory is allocated.

Contract¶

spill¶

void spill() // (1)
long spill(
  long size,
  MemoryConsumer trigger)

1. Uses MAX_VALUE for the size and this MemoryConsumer

Used when:

• TaskMemoryManager is requested to acquire execution memory
• UnsafeExternalSorter is requested to createWithExistingInMemorySorter

Implementations¶

• BytesToBytesMap
• LongToUnsafeRowMap
• RowBasedKeyValueBatch
• ShuffleExternalSorter
• Spillable
• UnsafeExternalSorter

Creating Instance¶

MemoryConsumer takes the following to be created:

• TaskMemoryManager
• Page Size
• MemoryMode (ON_HEAP or OFF_HEAP)