BasePredicate Expressions

Author: jaceklaskowski

docs/BindReferences.md

@@ -0,0 +1,32 @@
+# BindReferences
+
+`BindReferences` utility allows [bindReferences](#bindReferences).
+
+## <span id="bindReferences"> bindReferences
+
+```scala
+bindReference[A <: Expression](
+  expression: A,
+  input: AttributeSeq,
+  allowFailures: Boolean = false): A
+bindReferences[A <: Expression](
+  expressions: Seq[A],
+  input: AttributeSeq): Seq[A] // (1)!
+```
+
+1. `bindReference` of all the given `expressions` to the `input` schema
+
+For every `AttributeReference` expression in the given `expression`, `bindReferences` finds the `ExprId` in the `input` schema and creates a [BoundReference](expressions/BoundReference.md) expression.
+
+`bindReferences` throws an `IllegalStateException` when an `AttributeReference` could not be found in the input schema:
+
+```text
+Couldn't find [a] in [input]
+```
+
+---
+
+`bindReferences` is used when:
+
+* `ExpressionEncoder` is requested to [resolveAndBind](ExpressionEncoder.md#resolveAndBind)
+* _others_

docs/ExpressionEncoder.md

@@ -124,35 +124,41 @@ resolveAndBind(
 
 `resolveAndBind`...FIXME
 
+---
+
 `resolveAndBind` is used when:
 
-* `ResolveEncodersInUDF` analysis rule is executed
 * `Dataset` is requested for [resolvedEnc](Dataset.md#resolvedEnc)
-* `TypedAggregateExpression` is created
-* `ResolveEncodersInScalaAgg` extended resolution rule is executed
 * _others_
 
 ### Demo
 
-```text
+```scala
 case class Person(id: Long, name: String)
 import org.apache.spark.sql.Encoders
 val schema = Encoders.product[Person].schema
+```
 
+```scala
 import org.apache.spark.sql.catalyst.encoders.{RowEncoder, ExpressionEncoder}
 import org.apache.spark.sql.Row
 val encoder: ExpressionEncoder[Row] = RowEncoder.apply(schema).resolveAndBind()
+val deserializer = encoder.deserializer
+```
 
+```scala
 import org.apache.spark.sql.catalyst.InternalRow
-val row = InternalRow(1, "Jacek")
-
-val deserializer = encoder.deserializer
+val input = InternalRow(1, "Jacek")
+```
 
-scala> deserializer.eval(row)
+```text
+scala> deserializer.eval(input)
 java.lang.UnsupportedOperationException: Only code-generated evaluation is supported
   at org.apache.spark.sql.catalyst.expressions.objects.CreateExternalRow.eval(objects.scala:1105)
   ... 54 elided
+```
 
+```scala
 import org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext
 val ctx = new CodegenContext
 val code = deserializer.genCode(ctx).code

docs/expressions/BasePredicate.md

@@ -0,0 +1,25 @@
+# BasePredicate Expressions
+
+`BasePredicate` is an [abstraction](#contract) of [predicate expressions](#implementations) that can be [evaluated](#eval) to a `Boolean` value.
+
+`BasePredicate` is created using [Predicate.create](Predicate.md#create) utility.
+
+## Contract
+
+### <span id="eval"> Evaluating
+
+```scala
+eval(
+  r: InternalRow): Boolean
+```
+
+### <span id="initialize"> Initializing
+
+```scala
+initialize(
+  partitionIndex: Int): Unit
+```
+
+## Implementations
+
+* `InterpretedPredicate`

docs/expressions/Predicate.md

@@ -1,23 +1,14 @@
 # Predicate Expressions
 
-`Predicate` is an extension of the [Expression](Expression.md) abstraction for [expressions](#implementations) that return a [boolean](#dataType) value (_predicates_).
+`Predicate` is an extension of the [Expression](Expression.md) abstraction for [predicate expressions](#implementations) that evaluate to a value of [BooleanType](#dataType) type.
 
 ## Implementations
 
-* And
-* AtLeastNNonNulls
 * [BinaryComparison](BinaryComparison.md)
-* DynamicPruning
 * [Exists](Exists.md)
 * [In](In.md)
 * [InSet](InSet.md)
-* `InSubquery`
-* IsNaN
-* IsNotNull
-* IsNull
-* Not
-* Or
-* StringPredicate
+* _others_
 
 ## <span id="dataType"> DataType
 
@@ -27,4 +18,18 @@ dataType: DataType
 
 `dataType` is part of the [Expression](Expression.md#dataType) abstraction.
 
+---
+
 `dataType` is always [BooleanType](../types/DataType.md#BooleanType).
+
+## <span id="create"> Creating BasePredicate for Bound Expression
+
+```scala
+create(
+  e: Expression): BasePredicate
+create(
+  e: Expression,
+  inputSchema: Seq[Attribute]): BasePredicate
+```
+
+`create` [creates a BasePredicate](#createObject) for the given [Expression](Expression.md) that is [bound](#bindReference) to the input schema ([Attribute](Attribute.md)s).

mkdocs.yml

@@ -365,6 +365,7 @@ nav:
       - Aggregator: expressions/Aggregator.md
       - AttributeSeq: expressions/AttributeSeq.md
       - Attribute: expressions/Attribute.md
+      - BasePredicate: expressions/BasePredicate.md
       - BinaryComparison: expressions/BinaryComparison.md
       - BinaryOperator: expressions/BinaryOperator.md
       - BoundReference: expressions/BoundReference.md
@@ -943,6 +944,7 @@ nav:
     - Hive Partitioned Parquet Table and Partition Pruning: demo/hive-partitioned-parquet-table-partition-pruning.md
     - Using JDBC Data Source to Access PostgreSQL: demo/using-jdbc-data-source-to-access-postgresql.md
   - Misc:
+    - BindReferences: BindReferences.md
     - IntervalUtils: IntervalUtils.md
     - ExplainUtils: ExplainUtils.md
     - PartitionedFileUtil: PartitionedFileUtil.md