ObjectAggregationIterator and ObjectHashAggregateExec

Author: jaceklaskowski

docs/ObjectAggregationIterator.md

@@ -122,7 +122,7 @@ Object getBaseObject()
 
 * `UnsafeWriter` is requested to `write` an `UnsafeRow`
 * `UnsafeExternalRowSorter` is requested to `insertRow` an `UnsafeRow`
-* `UnsafeFixedWidthAggregationMap` is requested to [getAggregationBufferFromUnsafeRow](UnsafeFixedWidthAggregationMap.md#getAggregationBufferFromUnsafeRow)
+* `UnsafeFixedWidthAggregationMap` is requested to [getAggregationBufferFromUnsafeRow](physical-operators/UnsafeFixedWidthAggregationMap.md#getAggregationBufferFromUnsafeRow)
 * `UnsafeKVExternalSorter` is requested to `insertKV`
 * `ExternalAppendOnlyUnsafeRowArray` is requested to [add an UnsafeRow](ExternalAppendOnlyUnsafeRowArray.md#add)
 * `UnsafeHashedRelation` is requested to [get](physical-operators/UnsafeHashedRelation.md#get), [getValue](physical-operators/UnsafeHashedRelation.md#getValue), [getWithKeyIndex](physical-operators/UnsafeHashedRelation.md#getWithKeyIndex), [getValueWithKeyIndex](physical-operators/UnsafeHashedRelation.md#getValueWithKeyIndex), [apply](physical-operators/UnsafeHashedRelation.md#apply)
@@ -179,8 +179,8 @@ void copyFrom(
 
 `copyFrom` is used when:
 
-* `ObjectAggregationIterator` is requested to [processInputs](ObjectAggregationIterator.md#processInputs) (using `SortBasedAggregator`)
-* `TungstenAggregationIterator` is requested to [produce the next UnsafeRow](TungstenAggregationIterator.md#next) and [outputForEmptyGroupingKeyWithoutInput](TungstenAggregationIterator.md#outputForEmptyGroupingKeyWithoutInput)
+* `ObjectAggregationIterator` is requested to [processInputs](physical-operators/ObjectAggregationIterator.md#processInputs) (using `SortBasedAggregator`)
+* `TungstenAggregationIterator` is requested to [produce the next UnsafeRow](physical-operators/TungstenAggregationIterator.md#next) and [outputForEmptyGroupingKeyWithoutInput](physical-operators/TungstenAggregationIterator.md#outputForEmptyGroupingKeyWithoutInput)
 
 ## Deserializing UnsafeRow
 

docs/SortBasedAggregationIterator.md

@@ -52,7 +52,7 @@ Use [SQLConf.ADAPTIVE_CUSTOM_COST_EVALUATOR_CLASS](SQLConf.md#ADAPTIVE_CUSTOM_CO
 
 ## <span id="spark.sql.objectHashAggregate.sortBased.fallbackThreshold"> spark.sql.objectHashAggregate.sortBased.fallbackThreshold
 
-**(internal)** The number of rows of an in-memory hash map (to store aggregation buffer) before [ObjectHashAggregateExec](physical-operators/ObjectHashAggregateExec.md) ([ObjectAggregationIterator](ObjectAggregationIterator.md#processInputs) precisely) falls back to sort-based aggregation
+**(internal)** The number of rows of an in-memory hash map (to store aggregation buffer) before [ObjectHashAggregateExec](physical-operators/ObjectHashAggregateExec.md) ([ObjectAggregationIterator](physical-operators/ObjectAggregationIterator.md#processInputs) precisely) falls back to sort-based aggregation
 
 Default: `128`
 

docs/UnsafeRow.md

@@ -20,7 +20,7 @@
 
 * For `PartialMerge` or `Final` modes, the input to the [AggregateFunction](#aggregateFunction) is [immutable input aggregation buffers](AggregateFunction.md#inputAggBufferAttributes), and the actual children of the `AggregateFunction` is not used
 
-* [AggregateExpression](../AggregationIterator.md#aggregateExpressions)s of a [AggregationIterator](../AggregationIterator.md) cannot have more than 2 distinct modes nor the modes be among `Partial` and `PartialMerge` or `Final` and `Complete` mode pairs
+* [AggregateExpression](../physical-operators/AggregationIterator.md#aggregateExpressions)s of a [AggregationIterator](../physical-operators/AggregationIterator.md) cannot have more than 2 distinct modes nor the modes be among `Partial` and `PartialMerge` or `Final` and `Complete` mode pairs
 
 * `Partial` and `Complete` or `PartialMerge` and `Final` pairs are supported
 

docs/configuration-properties.md

@@ -16,7 +16,7 @@ Used when:
 
 * `EliminateAggregateFilter` logical optimization is executed
 * `AggregatingAccumulator` utility is used to create an `AggregatingAccumulator`
-* `AggregationIterator` is requested for the [generateResultProjection](../AggregationIterator.md#generateResultProjection)
+* `AggregationIterator` is requested for the [generateResultProjection](../physical-operators/AggregationIterator.md#generateResultProjection)
 * `HashAggregateExec` physical operator is requested to [doProduceWithoutKeys](../physical-operators/HashAggregateExec.md#doProduceWithoutKeys) and [generateResultFunction](../physical-operators/HashAggregateExec.md#generateResultFunction)
 * `AggregateProcessor` is [created](../window-functions/AggregateProcessor.md#apply)
 
@@ -32,7 +32,7 @@ Used when:
 
 * `EliminateAggregateFilter` logical optimization is executed
 * `AggregatingAccumulator` utility is used to create an `AggregatingAccumulator`
-* `AggregationIterator` is [created](../AggregationIterator.md#expressionAggInitialProjection)
+* `AggregationIterator` is [created](../physical-operators/AggregationIterator.md#expressionAggInitialProjection)
 * `HashAggregateExec` physical operator is requested to [doProduceWithoutKeys](../physical-operators/HashAggregateExec.md#doProduceWithoutKeys), [createHashMap](../physical-operators/HashAggregateExec.md#createHashMap) and [getEmptyAggregationBuffer](../physical-operators/HashAggregateExec.md#getEmptyAggregationBuffer)
 * `HashMapGenerator` is created
 * `AggregateProcessor` is [created](../window-functions/AggregateProcessor.md#apply)

docs/expressions/AggregateExpression.md

@@ -15,9 +15,9 @@ Used when:
 
 * `EliminateAggregateFilter` logical optimization is executed
 * `AggregatingAccumulator` is requested to `createBuffer`
-* `AggregationIterator` is requested to [initializeBuffer](../AggregationIterator.md#initializeBuffer)
-* `ObjectAggregationIterator` is requested to [initAggregationBuffer](../ObjectAggregationIterator.md#initAggregationBuffer)
-* `TungstenAggregationIterator` is requested to [createNewAggregationBuffer](../TungstenAggregationIterator.md#createNewAggregationBuffer)
+* `AggregationIterator` is requested to [initializeBuffer](../physical-operators/AggregationIterator.md#initializeBuffer)
+* `ObjectAggregationIterator` is requested to [initAggregationBuffer](../physical-operators/ObjectAggregationIterator.md#initAggregationBuffer)
+* `TungstenAggregationIterator` is requested to [createNewAggregationBuffer](../physical-operators/TungstenAggregationIterator.md#createNewAggregationBuffer)
 * `AggregateProcessor` is requested to [initialize](../window-functions/AggregateProcessor.md#initialize)
 
 ### <span id="merge"> merge
@@ -31,7 +31,7 @@ merge(
 Used when:
 
 * `AggregatingAccumulator` is requested to `merge`
-* `AggregationIterator` is requested to [generateProcessRow](../AggregationIterator.md#generateProcessRow)
+* `AggregationIterator` is requested to [generateProcessRow](../physical-operators/AggregationIterator.md#generateProcessRow)
 
 ### <span id="update"> update
 
@@ -44,7 +44,7 @@ update(
 Used when:
 
 * `AggregatingAccumulator` is requested to `add` an `InternalRow`
-* `AggregationIterator` is requested to [generateProcessRow](../AggregationIterator.md#generateProcessRow)
+* `AggregationIterator` is requested to [generateProcessRow](../physical-operators/AggregationIterator.md#generateProcessRow)
 * `AggregateProcessor` is requested to [update](../window-functions/AggregateProcessor.md#update)
 
 ## Implementations

docs/expressions/DeclarativeAggregate.md

@@ -1,6 +1,6 @@
 # AggregationIterators
 
-`AggregationIterator` is an [abstraction](#contract) of [aggregation iterators](#implementations) of [UnsafeRow](UnsafeRow.md)s.
+`AggregationIterator` is an [abstraction](#contract) of [aggregation iterators](#implementations) of [UnsafeRow](../UnsafeRow.md)s.
 
 ```scala
 abstract class AggregationIterator(...)
@@ -22,12 +22,12 @@ From [scala.collection.Iterator]({{ scala.api }}/scala/collection/Iterator.html)
 `AggregationIterator` takes the following to be created:
 
 * <span id="partIndex"> Partition ID
-* <span id="groupingExpressions"> Grouping [NamedExpression](expressions/NamedExpression.md)s
-* <span id="inputAttributes"> Input [Attribute](expressions/Attribute.md)s
-* <span id="aggregateExpressions"> [AggregateExpression](expressions/AggregateExpression.md)s
-* <span id="aggregateAttributes"> Aggregate [Attribute](expressions/Attribute.md)s
+* <span id="groupingExpressions"> Grouping [NamedExpression](../expressions/NamedExpression.md)s
+* <span id="inputAttributes"> Input [Attribute](../expressions/Attribute.md)s
+* <span id="aggregateExpressions"> [AggregateExpression](../expressions/AggregateExpression.md)s
+* <span id="aggregateAttributes"> Aggregate [Attribute](../expressions/Attribute.md)s
 * <span id="initialInputBufferOffset"> Initial input buffer offset
-* <span id="resultExpressions"> Result [NamedExpression](expressions/NamedExpression.md)s
+* <span id="resultExpressions"> Result [NamedExpression](../expressions/NamedExpression.md)s
 * <span id="newMutableProjection"> Function to create a new `MutableProjection` given expressions and attributes (`(Seq[Expression], Seq[Attribute]) => MutableProjection`)
 
 ??? note "Abstract Class"

docs/expressions/ImperativeAggregate.md

@@ -13,10 +13,10 @@
 
 `HashAggregateExec` supports [Java code generation](CodegenSupport.md) (aka _codegen_).
 
-`HashAggregateExec` uses [TungstenAggregationIterator](../TungstenAggregationIterator.md) (to iterate over `UnsafeRows` in partitions) when [executed](#doExecute).
+`HashAggregateExec` uses [TungstenAggregationIterator](TungstenAggregationIterator.md) (to iterate over `UnsafeRows` in partitions) when [executed](#doExecute).
 
 !!! note
-    `HashAggregateExec` uses `TungstenAggregationIterator` that can (theoretically) [switch to a sort-based aggregation when the hash-based approach is unable to acquire enough memory](../TungstenAggregationIterator.md#switchToSortBasedAggregation).
+    `HashAggregateExec` uses `TungstenAggregationIterator` that can (theoretically) [switch to a sort-based aggregation when the hash-based approach is unable to acquire enough memory](TungstenAggregationIterator.md#switchToSortBasedAggregation).
 
     See [testFallbackStartsAt](#testFallbackStartsAt) internal property and [spark.sql.TungstenAggregate.testFallbackStartsAt](../configuration-properties.md#spark.sql.TungstenAggregate.testFallbackStartsAt) configuration property.
 
@@ -33,7 +33,7 @@ supportsAggregate(
   aggregateBufferAttributes: Seq[Attribute]): Boolean
 ```
 
-`supportsAggregate` [checks support for aggregation](../UnsafeFixedWidthAggregationMap.md#supportsAggregationBufferSchema) given the aggregation buffer [Attribute](../expressions/Attribute.md)s.
+`supportsAggregate` [checks support for aggregation](UnsafeFixedWidthAggregationMap.md#supportsAggregationBufferSchema) given the aggregation buffer [Attribute](../expressions/Attribute.md)s.
 
 `supportsAggregate` is used when:
 
@@ -228,10 +228,10 @@ In the end, `doExecute` calculates the <<aggTime, aggTime>> metric and returns a
 
 * A single-element `Iterator[UnsafeRow]` with the <<TungstenAggregationIterator.md#outputForEmptyGroupingKeyWithoutInput, single UnsafeRow>>
 
-* The [TungstenAggregationIterator](../TungstenAggregationIterator.md)
+* The [TungstenAggregationIterator](TungstenAggregationIterator.md)
 
 !!! note
-    The [numOutputRows](#numOutputRows), [peakMemory](#peakMemory), [spillSize](#spillSize) and [avgHashProbe](#avgHashProbe) metrics are used exclusively to create the [TungstenAggregationIterator](../TungstenAggregationIterator.md).
+    The [numOutputRows](#numOutputRows), [peakMemory](#peakMemory), [spillSize](#spillSize) and [avgHashProbe](#avgHashProbe) metrics are used exclusively to create the [TungstenAggregationIterator](TungstenAggregationIterator.md).
 
 !!! note
     `doExecute` (by `RDD.mapPartitionsWithIndex` transformation) adds a new `MapPartitionsRDD` to the RDD lineage. Use `RDD.toDebugString` to see the additional `MapPartitionsRDD`.
@@ -352,7 +352,7 @@ finishAggregate(
 createHashMap(): UnsafeFixedWidthAggregationMap
 ```
 
-`createHashMap` creates a [UnsafeFixedWidthAggregationMap](../UnsafeFixedWidthAggregationMap.md) (with the <<getEmptyAggregationBuffer, empty aggregation buffer>>, the <<bufferSchema, bufferSchema>>, the <<groupingKeySchema, groupingKeySchema>>, the current `TaskMemoryManager`, `1024 * 16` initial capacity and the page size of the `TaskMemoryManager`)
+`createHashMap` creates a [UnsafeFixedWidthAggregationMap](UnsafeFixedWidthAggregationMap.md) (with the <<getEmptyAggregationBuffer, empty aggregation buffer>>, the <<bufferSchema, bufferSchema>>, the <<groupingKeySchema, groupingKeySchema>>, the current `TaskMemoryManager`, `1024 * 16` initial capacity and the page size of the `TaskMemoryManager`)
 
 ## Internal Properties