WriteToDataSourceV2(Exec) migrates to Structured Streaming

jaceklaskowski · jaceklaskowski · commit 08dbad78583c · 2022-10-01T14:03:13.000+02:00
diff --git a/docs/execution-planning-strategies/DataSourceV2Strategy.md b/docs/execution-planning-strategies/DataSourceV2Strategy.md
@@ -7,7 +7,7 @@ Logical Operator | Physical Operator
  [DataSourceV2ScanRelation](../logical-operators/DataSourceV2ScanRelation.md) with [V1Scan](../connector/V1Scan.md) | [RowDataSourceScanExec](../physical-operators/RowDataSourceScanExec.md)
  [DataSourceV2ScanRelation](../logical-operators/DataSourceV2ScanRelation.md) | [BatchScanExec](../physical-operators/BatchScanExec.md)
  `StreamingDataSourceV2Relation` |
- [WriteToDataSourceV2](../logical-operators/WriteToDataSourceV2.md) | [WriteToDataSourceV2Exec](../physical-operators/WriteToDataSourceV2Exec.md)
+ `WriteToDataSourceV2` ([Spark Structured Streaming]({{ book.structured_streaming }}/logical-operators/WriteToDataSourceV2)) | `WriteToDataSourceV2Exec` ([Spark Structured Streaming]({{ book.structured_streaming }}/physical-operators/WriteToDataSourceV2Exec))
  [CreateTableAsSelect](../logical-operators/CreateTableAsSelect.md) | `AtomicCreateTableAsSelectExec` or [CreateTableAsSelectExec](../physical-operators/CreateTableAsSelectExec.md)
  `RefreshTable` | `RefreshTableExec`
  `ReplaceTable` | `AtomicReplaceTableExec` or `ReplaceTableExec`
diff --git a/docs/logical-operators/AppendData.md b/docs/logical-operators/AppendData.md
@@ -2,9 +2,6 @@
 
 `AppendData` is a [V2WriteCommand](V2WriteCommand.md) that represents appending data (the result of executing a [structured query](#query)) to a [table](#table) (with the [columns matching](#isByName) by [name](#byName) or [position](#byPosition)).
 
-!!! note
-    `AppendData` has replaced the deprecated [WriteToDataSourceV2](WriteToDataSourceV2.md) logical operator.
-
 ## Creating Instance
 
 `AppendData` takes the following to be created:
diff --git a/docs/logical-operators/LogicalPlan.md b/docs/logical-operators/LogicalPlan.md
@@ -27,13 +27,11 @@ Logical operators with a single [child](../catalyst/TreeNode.md#children) logica
 * [CreateTable](CreateTable.md)
 * [IgnoreCachedData](IgnoreCachedData.md)
 * [NamedRelation](NamedRelation.md)
-* `ObjectProducer`
 * [ParsedStatement](ParsedStatement.md)
 * [SupportsSubquery](SupportsSubquery.md)
-* `Union`
 * [V2CreateTablePlan](V2CreateTablePlan.md)
 * [View](View.md)
-* [WriteToDataSourceV2](WriteToDataSourceV2.md)
+* _others_
 
 ## <span id="statsCache"> Statistics Cache
 
diff --git a/docs/logical-operators/WriteToDataSourceV2.md b/docs/logical-operators/WriteToDataSourceV2.md
diff --git a/docs/new-and-noteworthy/datasource-v2.md b/docs/new-and-noteworthy/datasource-v2.md
@@ -24,13 +24,7 @@ When executed, `DataSourceV2ScanExec` physical operator creates a [DataSourceRDD
 
 ## Data Writing
 
-DataSource V2 uses [WriteToDataSourceV2](../logical-operators/WriteToDataSourceV2.md) and [AppendData](../logical-operators/AppendData.md) logical operators to represent data writing (over a [DataSourceV2Relation](../logical-operators/DataSourceV2Relation.md) logical operator). As of Spark SQL 2.4.0, `WriteToDataSourceV2` operator was deprecated for the more specific `AppendData` operator (compare _"data writing"_ to _"data append"_ which is certainly more specific).
-
-NOTE: One of the differences between `WriteToDataSourceV2` and `AppendData` logical operators is that the former (`WriteToDataSourceV2`) uses [DataSourceWriter](../logical-operators/WriteToDataSourceV2.md#writer) directly while the latter (`AppendData`) uses [DataSourceV2Relation](../logical-operators/AppendData.md#table) to [get the DataSourceWriter from](../logical-operators/DataSourceV2Relation.md#newWriter).
-
-[WriteToDataSourceV2](../logical-operators/WriteToDataSourceV2.md) and [AppendData](../logical-operators/AppendData.md) (with [DataSourceV2Relation](../logical-operators/DataSourceV2Relation.md)) logical operators are planned as (_translated to_) a [WriteToDataSourceV2Exec](../physical-operators/WriteToDataSourceV2Exec.md) physical operator.
-
-When executed, `WriteToDataSourceV2Exec` physical operator...FIXME
+DataSource V2 uses `WriteToDataSourceV2` ([Spark Structured Streaming]({{ book.structured_streaming }}/logical-operators/WriteToDataSourceV2)) and [AppendData](../logical-operators/AppendData.md) logical operators to represent data writing (over a [DataSourceV2Relation](../logical-operators/DataSourceV2Relation.md) logical operator). As of Spark SQL 2.4.0, `WriteToDataSourceV2` operator was deprecated for the more specific `AppendData` operator (compare _"data writing"_ to _"data append"_ which is certainly more specific).
 
 ## <span id="filter-pushdown"> Filter Pushdown Performance Optimization
 
diff --git a/docs/physical-operators/V2TableWriteExec.md b/docs/physical-operators/V2TableWriteExec.md
@@ -16,7 +16,7 @@ query: SparkPlan
 
 * [TableWriteExecHelper](TableWriteExecHelper.md)
 * [V2ExistingTableWriteExec](V2ExistingTableWriteExec.md)
-* [WriteToDataSourceV2Exec](WriteToDataSourceV2Exec.md)
+* `WriteToDataSourceV2Exec` ([Spark Structured Streaming]({{ book.structured_streaming }}/physical-operators/WriteToDataSourceV2Exec))
 
 ## <span id="writeWithV2"> writeWithV2
 
@@ -25,9 +25,9 @@ writeWithV2(
   batchWrite: BatchWrite): Seq[InternalRow]
 ```
 
-`writeWithV2` requests the [physical query plan](#query) to [execute](SparkPlan.md#execute) (that gives a `RDD[InternalRow]`).
+`writeWithV2` requests the [physical query plan](#query) to [execute](SparkPlan.md#execute) (and produce a `RDD[InternalRow]`).
 
-`writeWithV2` requests the given `BatchWrite` for a [DataWriterFactory](../connector/BatchWrite.md#createBatchWriterFactory).
+`writeWithV2` requests the given [BatchWrite](../connector/BatchWrite.md) to [create a DataWriterFactory](../connector/BatchWrite.md#createBatchWriterFactory) (with the number of partitions of the `RDD`)
 
 `writeWithV2` prints out the following INFO message to the logs:
 
@@ -51,10 +51,16 @@ Data source write support [batchWrite] is committing.
 Data source write support [batchWrite] committed.
 ```
 
-In the end, `writeWithV2` returns no `InternalRow`s.
+In the end, `writeWithV2` returns an empty collection (of `InternalRow`s).
+
+---
 
 `writeWithV2` is used when:
 
-* `WriteToDataSourceV2Exec` is [executed](WriteToDataSourceV2Exec.md#run)
-* `V2ExistingTableWriteExec` is [executed](V2ExistingTableWriteExec.md#run)
 * `TableWriteExecHelper` is requested to [writeToTable](TableWriteExecHelper.md#writeToTable)
+* `V2ExistingTableWriteExec` is [executed](V2ExistingTableWriteExec.md#run)
+* `WriteToDataSourceV2Exec` ([Spark Structured Streaming]({{ book.structured_streaming }}/physical-operators/WriteToDataSourceV2Exec)) is executed
+
+## Logging
+
+`V2TableWriteExec` is a Scala trait and logging is configured using the logger of the [implementations](#implementations).
diff --git a/docs/physical-operators/WriteToDataSourceV2Exec.md b/docs/physical-operators/WriteToDataSourceV2Exec.md
diff --git a/mkdocs.yml b/mkdocs.yml
@@ -312,7 +312,6 @@ nav:
       - Window: logical-operators/Window.md
       - WithCTE: logical-operators/WithCTE.md
       - WithWindowDefinition: logical-operators/WithWindowDefinition.md
-      - WriteToDataSourceV2: logical-operators/WriteToDataSourceV2.md
     - SparkSession Registries:
       - Catalog:
         - Catalog: Catalog.md
@@ -531,7 +530,6 @@ nav:
       - WindowExec:
         - WindowExec: physical-operators/WindowExec.md
       - WindowExecBase: physical-operators/WindowExecBase.md
-      - WriteToDataSourceV2Exec: physical-operators/WriteToDataSourceV2Exec.md
       - Distribution and Partitioning:
         - Distribution: physical-operators/Distribution.md
         - Partitioning: physical-operators/Partitioning.md
