[DP] GraphElementRegistry and the pipeline graph elements

Author: jaceklaskowski

docs/declarative-pipelines/GraphElementRegistry.md

@@ -0,0 +1,61 @@
+# GraphElementRegistry
+
+`GraphElementRegistry` is an [abstraction](#contract) of [graph element registries](#implementations).
+
+## Contract
+
+### register_dataset { #register_dataset }
+
+```py
+register_dataset(
+    self,
+    dataset: Dataset,
+) -> None
+```
+
+See:
+
+* [SparkConnectGraphElementRegistry](SparkConnectGraphElementRegistry.md#register_dataset)
+
+Used when:
+
+* [@create_streaming_table](./index.md#create_streaming_table), [@table](./index.md#table), [@materialized_view](./index.md#materialized_view), [@temporary_view](./index.md#temporary_view) decorators are used
+
+### register_flow { #register_flow }
+
+```py
+register_flow(
+    self,
+    flow: Flow,
+) -> None
+```
+
+See:
+
+* [SparkConnectGraphElementRegistry](SparkConnectGraphElementRegistry.md#register_flow)
+
+Used when:
+
+* [@append_flow](./index.md#append_flow), [@table](./index.md#table), [@materialized_view](./index.md#materialized_view), [@temporary_view](./index.md#temporary_view) decorators are used
+
+### register_sql { #register_sql }
+
+```py
+register_sql(
+    self,
+    sql_text: str,
+    file_path: Path,
+) -> None
+```
+
+See:
+
+* [SparkConnectGraphElementRegistry](SparkConnectGraphElementRegistry.md#register_sql)
+
+Used when:
+
+* `pyspark.pipelines.cli` is requested to [register_definitions](#register_definitions)
+
+## Implementations
+
+* [SparkConnectGraphElementRegistry](SparkConnectGraphElementRegistry.md)

docs/declarative-pipelines/MaterializedView.md

@@ -0,0 +1,51 @@
+# MaterializedView
+
+`MaterializedView` is a `Table` that represents a materialized view in a pipeline dataflow graph.
+
+`MaterializedView` is created using [@materialized_view](#materialized_view) decorator.
+
+`MaterializedView` is a Python class.
+
+## materialized_view
+
+```py
+materialized_view(
+    query_function: Optional[QueryFunction] = None,
+    *,
+    name: Optional[str] = None,
+    comment: Optional[str] = None,
+    spark_conf: Optional[Dict[str, str]] = None,
+    table_properties: Optional[Dict[str, str]] = None,
+    partition_cols: Optional[List[str]] = None,
+    schema: Optional[Union[StructType, str]] = None,
+    format: Optional[str] = None,
+) -> Union[Callable[[QueryFunction], None], None]
+```
+
+`materialized_view` uses `query_function` for the parameters unless they are specified explicitly.
+
+`materialized_view` uses the name of the decorated function as the name of the materialized view unless specified explicitly.
+
+`materialized_view` makes sure that [GraphElementRegistry](GraphElementRegistry.md) has been set (using `graph_element_registration_context` context manager).
+
+??? note "Demo"
+
+    ```py
+    from pyspark.pipelines.graph_element_registry import (
+        graph_element_registration_context,
+        get_active_graph_element_registry,
+    )
+    from pyspark.pipelines.spark_connect_graph_element_registry import (
+        SparkConnectGraphElementRegistry,
+    )
+
+    dataflow_graph_id = "demo_dataflow_graph_id"
+    registry = SparkConnectGraphElementRegistry(spark, dataflow_graph_id)
+    with graph_element_registration_context(registry):
+        graph_registry = get_active_graph_element_registry()
+        assert graph_registry == registry
+    ```
+
+`materialized_view` creates a new `MaterializedView` and requests the `GraphElementRegistry` to [register_dataset](GraphElementRegistry.md#register_dataset) it.
+
+`materialized_view` creates a new `Flow` and requests the `GraphElementRegistry` to [register_flow](GraphElementRegistry.md#register_flow) it.

docs/declarative-pipelines/SparkConnectGraphElementRegistry.md

@@ -0,0 +1,31 @@
+# SparkConnectGraphElementRegistry
+
+`SparkConnectGraphElementRegistry` is a [GraphElementRegistry](GraphElementRegistry.md).
+
+## Creating Instance
+
+`SparkConnectGraphElementRegistry` takes the following to be created:
+
+* <span id="spark"> `SparkSession` (`SparkConnectClient`)
+* <span id="dataflow_graph_id"> Dataflow Graph ID
+
+`SparkConnectGraphElementRegistry` is created when:
+
+* `pyspark.pipelines.cli` is requested to [run](#run)
+
+## register_dataset { #register_dataset }
+
+??? note "GraphElementRegistry"
+
+    ```py
+    register_dataset(
+        self,
+        dataset: Dataset,
+    ) -> None
+    ```
+
+    `register_dataset` is part of the [GraphElementRegistry](GraphElementRegistry.md#register_dataset) abstraction.
+
+`register_dataset` makes sure that the given `Dataset` is either `MaterializedView`, `StreamingTable` or `TemporaryView`.
+
+`register_dataset` requests this [SparkSession](#spark) to [execute](#execute_command) a `PipelineCommand.DefineDataset`.

docs/declarative-pipelines/index.md

@@ -44,7 +44,7 @@ As of this [Commit 6ab0df9]({{ spark.commit }}/6ab0df9287c5a9ce49769612c2bb0a1da
 from pyspark import pipelines as dp
 ```
 
-## Python Decorators for Datasets and Flows { #python-decorators }
+## Python Decorators for Tables and Flows { #python-decorators }
 
 Declarative Pipelines uses the following [Python decorators](https://peps.python.org/pep-0318/) to describe tables and views:
 
@@ -73,6 +73,8 @@ from pyspark import pipelines as dp
 
 ### @dp.materialized_view { #materialized_view }
 
+Creates a [MaterializedView](MaterializedView.md) (for a table whose contents are defined to be the result of a query).
+
 ### @dp.table { #table }
 
 ### @dp.temporary_view { #temporary_view }