InSubquery Expression

jaceklaskowski · jaceklaskowski · commit 4e0fb0544acd · 2023-07-02T19:54:29.000+02:00
diff --git a/docs/bloom-filter-join/index.md b/docs/bloom-filter-join/index.md
@@ -7,7 +7,7 @@
 
 Bloom Filter Join uses [BloomFilter](BloomFilter.md)s as runtime filters when [spark.sql.optimizer.runtime.bloomFilter.enabled](../configuration-properties.md#spark.sql.optimizer.runtime.bloomFilter.enabled) configuration property is enabled.
 
-Bloom Filter Join uses [InjectRuntimeFilter](../logical-optimizations/InjectRuntimeFilter.md) logical optimization to inject up to [spark.sql.optimizer.runtimeFilter.number.threshold](../configuration-properties.md#spark.sql.optimizer.runtimeFilter.number.threshold) filters ([BloomFilter](BloomFilter.md)s or `InSubquery`s).
+Bloom Filter Join uses [InjectRuntimeFilter](../logical-optimizations/InjectRuntimeFilter.md) logical optimization to inject up to [spark.sql.optimizer.runtimeFilter.number.threshold](../configuration-properties.md#spark.sql.optimizer.runtimeFilter.number.threshold) filters ([BloomFilter](BloomFilter.md)s or [InSubquery](../expressions/InSubquery.md)s).
 
 ??? note "SPARK-32268"
     Bloom Filter Join was introduced in [SPARK-32268]({{ spark.jira }}/SPARK-32268).
diff --git a/docs/expressions/InSubquery.md b/docs/expressions/InSubquery.md
@@ -0,0 +1,76 @@
+---
+title: InSubquery
+---
+
+# InSubquery Expression
+
+`InSubquery` is a [Predicate](Predicate.md) that represents the following [IN](../sql/AstBuilder.md#withPredicate) SQL predicate in a logical query plan:
+
+```sql
+NOT? IN '(' query ')'
+```
+
+`InSubquery` can also be used internally for other use cases (e.g., [Runtime Filtering](../runtime-filtering/index.md), [Dynamic Partition Pruning](../dynamic-partition-pruning/index.md)).
+
+## Creating Instance
+
+`InSubquery` takes the following to be created:
+
+* <span id="values"> Values ([Expression](Expression.md)s)
+* <span id="query"> [ListQuery](ListQuery.md)
+
+`InSubquery` is created when:
+
+* [InjectRuntimeFilter](../logical-optimizations/InjectRuntimeFilter.md) logical optimization is executed (and [injectInSubqueryFilter](../logical-optimizations/InjectRuntimeFilter.md#injectInSubqueryFilter))
+* `AstBuilder` is requested to [withPredicate](../sql/AstBuilder.md#withPredicate) (for `NOT? IN '(' query ')'` SQL predicate)
+* [PlanDynamicPruningFilters](../physical-optimizations/PlanDynamicPruningFilters.md) physical optimization is executed (with [spark.sql.optimizer.dynamicPartitionPruning.enabled](../configuration-properties.md#spark.sql.optimizer.dynamicPartitionPruning.enabled) enabled)
+* `RowLevelOperationRuntimeGroupFiltering` logical optimization is executed
+
+## Unevaluable { #Unevaluable }
+
+`InSubquery` is an [Unevaluable](Unevaluable.md) expression.
+
+`InSubquery` can be converted to a [Join](../logical-operators/Join.md) operator at logical optimization using [RewritePredicateSubquery](../logical-optimizations/RewritePredicateSubquery.md):
+
+* [Left-Semi Join](../logical-operators/Join.md) unless it is a `NOT IN` that becomes a [Left-Anti Join](../logical-operators/Join.md) (among the other _less_ important cases)
+
+`InSubquery` can also be converted to [InSubqueryExec](InSubqueryExec.md) expression (over a [SubqueryExec](../physical-operators/SubqueryExec.md)) in [PlanSubqueries](../physical-optimizations/PlanSubqueries.md) physical optimization.
+
+## Logical Analysis
+
+`InSubquery` is resolved using the following logical analysis rules:
+
+* [ResolveSubquery](../logical-analysis-rules/ResolveSubquery.md)
+* `InConversion`
+
+## Logical Optimization
+
+`InSubquery` is optimized using the following logical optimizations:
+
+* [NullPropagation](../logical-optimizations/NullPropagation.md) (so `null` values give `null` results)
+* [InjectRuntimeFilter](../logical-optimizations/InjectRuntimeFilter.md)
+* [RewritePredicateSubquery](../logical-optimizations/RewritePredicateSubquery.md)
+
+## Physical Optimization
+
+`InSubquery` is optimized using the following physical optimizations:
+
+* [PlanSubqueries](../physical-optimizations/PlanSubqueries.md)
+* [InsertAdaptiveSparkPlan](../physical-optimizations/InsertAdaptiveSparkPlan.md)
+* [PlanAdaptiveSubqueries](../physical-optimizations/PlanAdaptiveSubqueries.md)
+
+## Catalyst DSL
+
+`InSubquery` can be created using [in](#in) operator using [Catalyst DSL](../catalyst-dsl/index.md) (via `ImplicitOperators`).
+
+## nodePatterns { #nodePatterns }
+
+??? note "TreeNode"
+
+    ```scala
+    nodePatterns: Seq[TreePattern]
+    ```
+
+    `nodePatterns` is part of the [TreeNode](../catalyst/TreeNode.md#nodePatterns) abstraction.
+
+`nodePatterns` is [IN_SUBQUERY](../catalyst/TreePattern.md#IN_SUBQUERY).
diff --git a/docs/expressions/InSubqueryExec.md b/docs/expressions/InSubqueryExec.md
@@ -1,6 +1,10 @@
-# InSubqueryExec
+---
+title: InSubqueryExec
+---
 
-`InSubqueryExec` is a [ExecSubqueryExpression](ExecSubqueryExpression.md) that represents `InSubquery` and [DynamicPruningSubquery](DynamicPruningSubquery.md) expressions at execution time.
+# InSubqueryExec Expression
+
+`InSubqueryExec` is an [ExecSubqueryExpression](ExecSubqueryExpression.md) that represents [InSubquery](InSubquery.md) and [DynamicPruningSubquery](DynamicPruningSubquery.md) expressions at execution time.
 
 ## Creating Instance
 
@@ -13,13 +17,12 @@
 
 `InSubqueryExec` is created when:
 
-* [PlanSubqueries](../physical-optimizations/PlanSubqueries.md) physical optimization is executed (and plans `InSubquery` expressions)
-* [PlanAdaptiveSubqueries](../physical-optimizations/PlanAdaptiveSubqueries.md) physical optimization is executed (and plans `InSubquery` expressions)
+* [PlanSubqueries](../physical-optimizations/PlanSubqueries.md) physical optimization is executed (and plans [InSubquery](InSubquery.md) expressions)
+* [PlanAdaptiveSubqueries](../physical-optimizations/PlanAdaptiveSubqueries.md) physical optimization is executed (and plans [InSubquery](InSubquery.md) expressions)
 * [PlanDynamicPruningFilters](../physical-optimizations/PlanDynamicPruningFilters.md) physical optimization is executed (and plans [DynamicPruningSubquery](DynamicPruningSubquery.md) expressions)
 
-## Broadcasted Result
+## Broadcasted Result { #resultBroadcast }
 
-<span id="resultBroadcast">
 ```scala
 resultBroadcast: Broadcast[Array[Any]]
 ```
@@ -28,12 +31,16 @@ resultBroadcast: Broadcast[Array[Any]]
 
 `resultBroadcast` is updated when `InSubqueryExec` is requested to [update the collected result](#updateResult).
 
-## <span id="eval"> Interpreted Expression Evaluation
+## <span id=""> Interpreted Expression Evaluation { #eval }
 
-```scala
-eval(
-  input: InternalRow): Any
-```
+??? note "Expression"
+
+    ```scala
+    eval(
+      input: InternalRow): Any
+    ```
+
+    `eval` is part of the [Expression](Expression.md#eval) abstraction.
 
 `eval` [prepareResult](#prepareResult).
 
@@ -44,45 +51,45 @@ eval(
 * `null` for `null` evaluation result
 * `true` when the [result](#result) contains the evaluation result or `false`
 
-`eval` is part of the [Expression](Expression.md#eval) abstraction.
+## Code-Generated Expression Evaluation { #doGenCode }
 
-## Code-Generated Expression Evaluation
+??? note "Expression"
 
-<span id="doGenCode">
-```scala
-doGenCode(
-  ctx: CodegenContext,
-  ev: ExprCode): ExprCode
-```
+    ```scala
+    doGenCode(
+      ctx: CodegenContext,
+      ev: ExprCode): ExprCode
+    ```
+
+    `doGenCode` is part of the [Expression](Expression.md#doGenCode) abstraction.
 
 `doGenCode` [prepareResult](#prepareResult).
 
 `doGenCode` creates a [InSet](InSet.md) expression (with the [child](#child) expression and [result](#result)) and requests it to [doGenCode](Expression.md#doGenCode).
 
-`doGenCode` is part of the [Expression](Expression.md#doGenCode) abstraction.
+## Updating Result { #updateResult }
 
-## Updating Result
+??? note "ExecSubqueryExpression"
 
-<span id="updateResult">
-```scala
-updateResult(): Unit
-```
+    ```scala
+    updateResult(): Unit
+    ```
+
+    `updateResult` is part of the [ExecSubqueryExpression](ExecSubqueryExpression.md#updateResult) abstraction.
 
 `updateResult` requests the [BaseSubqueryExec](#plan) to [executeCollect](../physical-operators/SparkPlan.md#executeCollect).
 
 `updateResult` uses the collected result to update the [result](#result) and [resultBroadcast](#resultBroadcast) registries.
 
-`updateResult` is part of the [ExecSubqueryExpression](ExecSubqueryExpression.md#updateResult) abstraction.
-
-## <span id="result"> result Registry
+## result
 
 ```scala
 result: Array[Any]
 ```
 
 `result`...FIXME
 
-## <span id="prepareResult"> prepareResult
+## prepareResult { #prepareResult }
 
 ```scala
 prepareResult(): Unit
@@ -96,4 +103,8 @@ prepareResult(): Unit
 [this] has not finished
 ```
 
-`prepareResult` is used when `InSubqueryExec` expression is evaluated ([interpreted](#eval) or [code-generated](#doGenCode)).
+---
+
+`prepareResult` is used when:
+
+* `InSubqueryExec` expression is evaluated ([interpreted](#eval) or [code-generated](#doGenCode)).
diff --git a/docs/logical-analysis-rules/ResolveSubquery.md b/docs/logical-analysis-rules/ResolveSubquery.md
@@ -11,7 +11,7 @@
 
 * [ScalarSubquery](../expressions/ExecSubqueryExpression-ScalarSubquery.md)
 * [Exists](../expressions/Exists.md)
-* [ListQuery](../expressions/ListQuery.md) (in `InSubquery` expressions)
+* [ListQuery](../expressions/ListQuery.md) (in [InSubquery](../expressions/InSubquery.md) expressions)
 
 `ResolveSubquery` is part of [Resolution](../Analyzer.md#Resolution) rule batch of the [Logical Analyzer](../Analyzer.md).
 
@@ -79,7 +79,7 @@ resolveSubQueries(
 
 * [ScalarSubquery](../expressions/ExecSubqueryExpression-ScalarSubquery.md)
 * [Exists](../expressions/Exists.md)
-* [ListQuery](../expressions/ListQuery.md) (in `InSubquery` expressions)
+* [ListQuery](../expressions/ListQuery.md) (in [InSubquery](../expressions/InSubquery.md) expressions)
 
 `resolveSubQueries` is used when `ResolveSubquery` is [executed](#apply).
 
diff --git a/docs/logical-optimizations/InjectRuntimeFilter.md b/docs/logical-optimizations/InjectRuntimeFilter.md
@@ -126,9 +126,9 @@ Property | Value
 Unless the `Aggregate` logical operator [canBroadcastBySize](../JoinSelectionHelper.md#canBroadcastBySize), `injectInSubqueryFilter` returns the given `filterApplicationSidePlan` logical plan (and basically throws away all the work so far).
 
 !!! note
-    `injectInSubqueryFilter` skips the `InSubquery` filter if the size of the `Aggregate` is beyond [broadcast join threshold](../JoinSelectionHelper.md#canBroadcastBySize) and the semi-join will be a shuffle join, which is not worthwhile.
+    `injectInSubqueryFilter` skips the [InSubquery](../expressions/InSubquery.md) filter if the size of the `Aggregate` is beyond [broadcast join threshold](../JoinSelectionHelper.md#canBroadcastBySize) and the semi-join will be a shuffle join, which is not worthwhile.
 
-`injectInSubqueryFilter` creates an `InSubquery` logical operator with the following:
+`injectInSubqueryFilter` creates an [InSubquery](../expressions/InSubquery.md) expression with the following:
 
 * The given `filterApplicationSideExp` (possibly [mayWrapWithHash](#mayWrapWithHash))
 * [ListQuery](../expressions/ListQuery.md) expression with the `Aggregate`
diff --git a/docs/physical-operators/SparkPlan.md b/docs/physical-operators/SparkPlan.md
@@ -520,7 +520,7 @@ executeCollect(): Array[InternalRow]
 
 * `SparkPlan` is requested to <<executeCollectPublic, executeCollectPublic>>
 
-* `ScalarSubquery` and `InSubquery` plan expressions are requested to `updateResult`
+* [ScalarSubquery](../expressions/ScalarSubquery.md) and [InSubquery](../expressions/InSubquery.md) plan expressions are requested to `updateResult`
 
 ## <span id="outputPartitioning"> Output Data Partitioning Requirements
 
diff --git a/docs/physical-optimizations/InsertAdaptiveSparkPlan.md b/docs/physical-optimizations/InsertAdaptiveSparkPlan.md
@@ -112,7 +112,7 @@ buildSubqueryMap(
   plan: SparkPlan): Map[Long, SubqueryExec]
 ```
 
-`buildSubqueryMap` finds [ScalarSubquery](../expressions/ScalarSubquery) and [ListQuery](../expressions/ListQuery.md) (in `InSubquery`) expressions (unique by expression ID to reuse the execution plan from another sub-query) in the given [physical query plan](../physical-operators/SparkPlan.md).
+`buildSubqueryMap` finds [ScalarSubquery](../expressions/ScalarSubquery) and [ListQuery](../expressions/ListQuery.md) (in [InSubquery](../expressions/InSubquery.md)) expressions (unique by expression ID to reuse the execution plan from another sub-query) in the given [physical query plan](../physical-operators/SparkPlan.md).
 
 For every `ScalarSubquery` and `ListQuery` expressions, `buildSubqueryMap` [compileSubquery](#compileSubquery), [verifyAdaptivePlan](#verifyAdaptivePlan) and registers a new [SubqueryExec](../physical-operators/SubqueryExec.md) operator.
 
diff --git a/docs/runtime-filtering/index.md b/docs/runtime-filtering/index.md
@@ -1,5 +1,5 @@
 # Runtime Filtering
 
-**Runtime Filtering** is an optimization of join queries by pre-filtering one side of a join using [Bloom Filter](../bloom-filter-join/index.md) or `InSubquery` predicate based on the values from the other side of the join.
+**Runtime Filtering** is an optimization of join queries by pre-filtering one side of a join using [Bloom Filter](../bloom-filter-join/index.md) or [InSubquery](../expressions/InSubquery.md) predicate based on the values from the other side of the join.
 
-Runtime Filtering uses [InjectRuntimeFilter](../logical-optimizations/InjectRuntimeFilter.md) logical optimization to inject either [Bloom Filter](../bloom-filter-join/index.md) or `InSubquery` predicate based on [spark.sql.optimizer.runtime.bloomFilter.enabled](../configuration-properties.md#spark.sql.optimizer.runtime.bloomFilter.enabled) configuration property.
+Runtime Filtering uses [InjectRuntimeFilter](../logical-optimizations/InjectRuntimeFilter.md) logical optimization to inject either [Bloom Filter](../bloom-filter-join/index.md) or [InSubquery](../expressions/InSubquery.md) predicate based on [spark.sql.optimizer.runtime.bloomFilter.enabled](../configuration-properties.md#spark.sql.optimizer.runtime.bloomFilter.enabled) configuration property.
diff --git a/docs/sql/AstBuilder.md b/docs/sql/AstBuilder.md
@@ -763,7 +763,7 @@ Creates an [UnresolvedHaving](../logical-operators/UnresolvedHaving.md) for the
 HAVING booleanExpression
 ```
 
-### <span id="withHints"> withHints
+### withHints { #withHints }
 
 Adds an [UnresolvedHint](../logical-operators/UnresolvedHint.md) for `/*+ hint */` in `SELECT` queries.
 
@@ -826,17 +826,19 @@ For regular `SELECT` (no `TRANSFORM`, `MAP` or `REDUCE` qualifiers), `withQueryS
 
 1. [UnresolvedHint](#withHints) unary logical operator (if used in the parsed SQL text)
 
-### withPredicate
+### withPredicate { #withPredicate }
 
-* `NOT? IN '(' query ')'` adds an [In](../expressions/In.md) predicate expression with a [ListQuery](../expressions/ListQuery.md) subquery expression
+Creates a [InSubquery](../expressions/InSubquery.md) over a [ListQuery](../expressions/ListQuery.md) (possibly "inverted" using `Not` unary expression)
 
-* `NOT? IN '(' expression (',' expression)* ')'` adds an [In](../expressions/In.md) predicate expression
+```sql
+NOT? IN '(' query ')'
+```
 
-### <span id="withPivot"> withPivot
+### withPivot { #withPivot }
 
 Creates a [Pivot](../logical-operators/Pivot.md) unary logical operator for the following SQL clause:
 
-```text
+```sql
 PIVOT '(' aggregates FOR pivotColumn IN '(' pivotValue (',' pivotValue)* ')' ')'
 ```
 
diff --git a/mkdocs.yml b/mkdocs.yml
@@ -329,6 +329,7 @@ nav:
       - In: expressions/In.md
       - Inline: expressions/Inline.md
       - InSet: expressions/InSet.md
+      - InSubquery: expressions/InSubquery.md
       - InSubqueryExec: expressions/InSubqueryExec.md
       - InterpretedProjection: expressions/InterpretedProjection.md
       - JsonToStructs: expressions/JsonToStructs.md
