Skip to content

Commit d0dba06

Browse files
jaceklaskowskijaceklaskowski
committed
DelegatingCatalogExtension et al.
1 parent 9f35184 commit d0dba06

File tree

6 files changed

+89
-38
lines changed

6 files changed

+89
-38
lines changed

‎docs/BaseSessionStateBuilder.md‎

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -70,13 +70,12 @@ catalog: SessionCatalog
7070
catalogManager: CatalogManager
7171
```
7272

73-
[CatalogManager](connector/catalog/CatalogManager.md) that is created for the session-specific [SQLConf](#conf), [V2SessionCatalog](#v2SessionCatalog) and [SessionCatalog](#catalog).
73+
[CatalogManager](connector/catalog/CatalogManager.md) that is created with this [V2SessionCatalog](#v2SessionCatalog) and this [SessionCatalog](#catalog).
7474

7575
`catalogManager` is used when:
7676

77-
* `BaseSessionStateBuilder` is requested for [Analyzer](#analyzer) and [Optimizer](#optimizer)
78-
79-
* `HiveSessionStateBuilder` is requested for [Analyzer](hive/HiveSessionStateBuilder.md#analyzer)
77+
* `BaseSessionStateBuilder` is requested for the [Analyzer](#analyzer) and the [Optimizer](#optimizer)
78+
* `HiveSessionStateBuilder` is requested for the [Analyzer](hive/HiveSessionStateBuilder.md#analyzer)
8079

8180
### SQLConf { #conf }
8281

@@ -129,11 +128,13 @@ When requested for the first time (as a `lazy val`), `tableFunctionRegistry` req
129128
v2SessionCatalog: V2SessionCatalog
130129
```
131130

132-
[V2SessionCatalog](V2SessionCatalog.md) that is created for the session-specific [SessionCatalog](#catalog) and [SQLConf](#conf).
131+
[V2SessionCatalog](V2SessionCatalog.md) that is created with this [SessionCatalog](#catalog).
132+
133+
This `V2SessionCatalog` is used when:
133134

134-
`v2SessionCatalog` is used when `BaseSessionStateBuilder` is requested for the [CatalogManager](#catalogManager).
135+
* `BaseSessionStateBuilder` is requested for the [CatalogManager](#catalogManager)
135136

136-
## <span id="customOperatorOptimizationRules"> Custom Operator Optimization Rules
137+
## Custom Operator Optimization Rules { #customOperatorOptimizationRules }
137138

138139
```scala
139140
customOperatorOptimizationRules: Seq[Rule[LogicalPlan]]

‎docs/configuration-properties.md‎

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -307,6 +307,16 @@ Used when:
307307

308308
* `InMemoryRelation` is requested for the [CachedBatchSerializer](logical-operators/InMemoryRelation.md#getSerializer)
309309

310+
## <span id="V2_SESSION_CATALOG_IMPLEMENTATION"> spark.sql.catalog.spark_catalog { #spark.sql.catalog.spark_catalog }
311+
312+
The fully-qualified class name of a [CatalogPlugin](connector/catalog/CatalogPlugin.md) to handle `spark_catalog` built-in session catalog
313+
314+
Default: [defaultSessionCatalog](connector/catalog/CatalogManager.md#defaultSessionCatalog)
315+
316+
Used when:
317+
318+
* `CatalogManager` is requested for the [user-specified v2 session catalog](connector/catalog/CatalogManager.md#v2SessionCatalog)
319+
310320
## spark.sql.codegen { #spark.sql.codegen }
311321

312322
### aggregate.fastHashMap.capacityBit { #spark.sql.codegen.aggregate.fastHashMap.capacityBit }
@@ -1625,12 +1635,6 @@ It is highly discouraged to turn on case sensitive mode.
16251635

16261636
Use [SQLConf.caseSensitiveAnalysis](SQLConf.md#caseSensitiveAnalysis) method to access the current value.
16271637

1628-
## <span id="spark.sql.catalog.spark_catalog"><span id="V2_SESSION_CATALOG_IMPLEMENTATION"> spark.sql.catalog.spark_catalog
1629-
1630-
The [CatalogPlugin](connector/catalog/CatalogPlugin.md) for `spark_catalog`
1631-
1632-
Default: [defaultSessionCatalog](connector/catalog/CatalogManager.md#defaultSessionCatalog)
1633-
16341638
## <span id="spark.sql.cbo.enabled"> spark.sql.cbo.enabled
16351639

16361640
Enables [Cost-Based Optimization](cost-based-optimization/index.md) (CBO) for estimation of plan statistics when `true`.

‎docs/connector/catalog/CatalogExtension.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,14 +4,16 @@
44

55
## Contract
66

7-
### <span id="setDelegateCatalog"> setDelegateCatalog
7+
### setDelegateCatalog { #setDelegateCatalog }
88

99
```java
1010
void setDelegateCatalog(
1111
CatalogPlugin delegate)
1212
```
1313

14-
Used when `CatalogManager` is requested to [loadV2SessionCatalog](CatalogManager.md#loadV2SessionCatalog)
14+
Used when:
15+
16+
* `CatalogManager` is requested to [loadV2SessionCatalog](CatalogManager.md#loadV2SessionCatalog)
1517

1618
## Implementations
1719

‎docs/connector/catalog/CatalogManager.md‎

Lines changed: 32 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -12,25 +12,25 @@
1212

1313
* `BaseSessionStateBuilder` is requested for a [CatalogManager](../../BaseSessionStateBuilder.md#catalogManager)
1414

15-
## <span id="defaultSessionCatalog"> Default Session Catalog
15+
## Default Session Catalog { #defaultSessionCatalog }
1616

1717
```scala
1818
defaultSessionCatalog: CatalogPlugin
1919
```
2020

21-
`CatalogManager` is given a [CatalogPlugin](CatalogPlugin.md) when [created](#creating-instance) for the **default session catalog**.
21+
`CatalogManager` is given a [CatalogPlugin](CatalogPlugin.md) when [created](#creating-instance) for the **default session catalog** (which is a [V2SessionCatalog](../../BaseSessionStateBuilder.md#v2SessionCatalog)).
2222

2323
`defaultSessionCatalog` is used as the [delegate catalog](CatalogExtension.md#setDelegateCatalog) when `CatalogManager` is requested to [load a V2SessionCatalog](#loadV2SessionCatalog).
2424

25-
`defaultSessionCatalog` is used as the fall-back catalog when `CatalogManager` is requested to [load a custom V2CatalogPlugin](#v2SessionCatalog).
25+
`defaultSessionCatalog` is used as the fallback catalog when `CatalogManager` is requested to [load a custom V2CatalogPlugin](#v2SessionCatalog).
2626

27-
## <span id="SESSION_CATALOG_NAME"> Default Catalog Name
27+
## spark_catalog Default Catalog Name { #SESSION_CATALOG_NAME }
2828

2929
`CatalogManager` defines `spark_catalog` as the name of the default catalog ([V2SessionCatalog](../../V2SessionCatalog.md)).
3030

3131
`spark_catalog` is used as the default value of [spark.sql.defaultCatalog](../../configuration-properties.md#spark.sql.defaultCatalog) configuration property.
3232

33-
## <span id="_currentCatalogName"> Current Catalog Name
33+
## Current Catalog Name { #_currentCatalogName }
3434

3535
```scala
3636
_currentCatalogName: Option[String]
@@ -40,7 +40,7 @@ _currentCatalogName: Option[String]
4040

4141
`_currentCatalogName` can be changed using [setCurrentCatalog](#setCurrentCatalog).
4242

43-
## <span id="currentCatalog"> Current CatalogPlugin
43+
## Current CatalogPlugin { #currentCatalog }
4444

4545
```scala
4646
currentCatalog: CatalogPlugin
@@ -56,22 +56,24 @@ currentCatalog: CatalogPlugin
5656

5757
* `ViewHelper` utility is requested to `generateViewProperties`
5858

59-
## <span id="currentNamespace"> Current Namespace
59+
## Current Namespace { #currentNamespace }
6060

6161
```scala
6262
currentNamespace: Array[String]
6363
```
6464

6565
`currentNamespace`...FIXME
6666

67+
---
68+
6769
`currentNamespace` is used when:
6870

6971
* `ResolveNamespace` analysis rule is executed
7072
* `GetCurrentDatabase` analysis rule is executed
7173
* `CatalogAndIdentifier` extractor utility is requested to `unapply`
7274
* `ViewHelper` utility is requested to `generateViewProperties`
7375

74-
## <span id="setCurrentNamespace"> Setting Current Namespace
76+
## Setting Current Namespace { #setCurrentNamespace }
7577

7678
```scala
7779
setCurrentNamespace(
@@ -80,9 +82,13 @@ setCurrentNamespace(
8082

8183
`setCurrentNamespace`...FIXME
8284

83-
`setCurrentNamespace` is used when `SetCatalogAndNamespaceExec` physical command is executed.
85+
---
86+
87+
`setCurrentNamespace` is used when:
8488

85-
## <span id="setCurrentCatalog"> Changing Current Catalog Name
89+
* `SetCatalogAndNamespaceExec` physical command is executed
90+
91+
## Changing Current Catalog Name { #setCurrentCatalog }
8692

8793
```scala
8894
setCurrentCatalog(
@@ -100,7 +106,7 @@ Only if the names are different, `setCurrentCatalog` makes it [_currentCatalogNa
100106
* `SetCatalogCommand` logical command is executed
101107
* [SetCatalogAndNamespaceExec](../../physical-operators/SetCatalogAndNamespaceExec.md) physical command is executed
102108

103-
## <span id="catalog"> Finding CatalogPlugin by Name
109+
## Finding CatalogPlugin by Name { #catalog }
104110

105111
```scala
106112
catalog(
@@ -111,15 +117,15 @@ catalog(
111117

112118
Otherwise, `catalog` looks up the name in [catalogs](#catalogs) internal registry. When not found, `catalog` tries to [load a CatalogPlugin by name](Catalogs.md#load) (and registers it in [catalogs](#catalogs) internal registry).
113119

120+
---
121+
114122
`catalog` is used when:
115123

116124
* `CatalogManager` is requested to [isCatalogRegistered](#isCatalogRegistered) and [currentCatalog](#currentCatalog)
117-
118125
* `CatalogV2Util` utility is requested to [getTableProviderCatalog](CatalogV2Util.md#getTableProviderCatalog)
119-
120126
* `CatalogAndMultipartIdentifier`, `CatalogAndNamespace` and `CatalogAndIdentifier` utilities are requested to extract a [CatalogPlugin](CatalogPlugin.md) (`unapply`)
121127

122-
## <span id="isCatalogRegistered"> isCatalogRegistered
128+
## isCatalogRegistered { #isCatalogRegistered }
123129

124130
```scala
125131
isCatalogRegistered(
@@ -130,32 +136,37 @@ isCatalogRegistered(
130136

131137
`isCatalogRegistered` is used when `Analyzer` is requested to [expandRelationName](../../Analyzer.md#expandRelationName).
132138

133-
## <span id="v2SessionCatalog"> v2SessionCatalog
139+
## v2SessionCatalog { #v2SessionCatalog }
134140

135141
```scala
136142
v2SessionCatalog: CatalogPlugin
137143
```
138144

139145
`v2SessionCatalog`...FIXME
140146

147+
---
148+
141149
`v2SessionCatalog` is used when:
142150

143151
* `CatalogManager` is requested to [look up a CatalogPlugin by name](#catalog)
144-
145152
* `CatalogV2Util` is requested to `getTableProviderCatalog`
146-
147153
* `CatalogAndIdentifier` utility is requested to extract a CatalogPlugin and an identifier from a multi-part name (`unapply`)
148154

149-
## <span id="loadV2SessionCatalog"> loadV2SessionCatalog
155+
## loadV2SessionCatalog { #loadV2SessionCatalog }
150156

151157
```scala
152158
loadV2SessionCatalog(): CatalogPlugin
153159
```
154160

155-
`loadV2SessionCatalog` [loads](Catalogs.md#load) the default [spark_catalog](#SESSION_CATALOG_NAME).
161+
`loadV2SessionCatalog` [loads](Catalogs.md#load) the default [CatalogPlugin](CatalogPlugin.md) that handles [spark_catalog](#SESSION_CATALOG_NAME) catalog.
162+
163+
!!! note "spark.sql.catalog.spark_catalog"
164+
[spark.sql.catalog.spark_catalog](../../configuration-properties.md#spark.sql.catalog.spark_catalog) configuration property is used to specify the fully-qualified class name of the session [CatalogPlugin](CatalogPlugin.md).
156165

157-
If it is of type [CatalogExtension](CatalogExtension.md), `loadV2SessionCatalog` requests it to [setDelegateCatalog](CatalogExtension.md#setDelegateCatalog) with the [defaultSessionCatalog](#defaultSessionCatalog).
166+
Only if the `CatalogPlugin` is a [CatalogExtension](CatalogExtension.md), `loadV2SessionCatalog` assigns this [default session CatalogPlugin](#defaultSessionCatalog) to be the [delegate CatalogPlugin](CatalogExtension.md#setDelegateCatalog).
167+
168+
---
158169

159170
`loadV2SessionCatalog` is used when:
160171

161-
* `CatalogManager` is requested for a [CatalogPlugin](#v2SessionCatalog)
172+
* `CatalogManager` is requested for the [user-specified v2 session catalog](#v2SessionCatalog)

‎docs/connector/catalog/DelegatingCatalogExtension.md‎

Lines changed: 33 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,35 @@
11
# DelegatingCatalogExtension
22

3-
`DelegatingCatalogExtension` is an [extension](#contract) of the [CatalogExtension](CatalogExtension.md) abstraction for catalogs that delegate unsupported catalog functions to the built-in session catalog.
3+
`DelegatingCatalogExtension` is an [extension](#contract) of the [CatalogExtension](CatalogExtension.md) abstraction for catalogs that delegate unsupported catalog extensions to the [Delegate Catalog](#delegate).
4+
5+
`DelegatingCatalogExtension` is a convenience abstraction so that Spark extensions developers can focus on a subset of the [CatalogExtension](CatalogExtension.md) features.
6+
7+
## Delegate Catalog { #delegate }
8+
9+
`DelegatingCatalogExtension` can be given a [CatalogPlugin](CatalogPlugin.md) to handle the following (unless overriden):
10+
11+
* [name](CatalogPlugin.md#name)
12+
* [defaultNamespace](CatalogPlugin.md#defaultNamespace)
13+
* [asTableCatalog](#asTableCatalog)
14+
* [asNamespaceCatalog](#asNamespaceCatalog)
15+
* [asFunctionCatalog](#asFunctionCatalog)
16+
17+
The `CatalogPlugin` is assigned at [setDelegateCatalog](#setDelegateCatalog).
18+
19+
## setDelegateCatalog { #setDelegateCatalog }
20+
21+
??? note "CatalogExtension"
22+
23+
```java
24+
void setDelegateCatalog(
25+
CatalogPlugin delegate)
26+
```
27+
28+
`setDelegateCatalog` is part of the [CatalogExtension](CatalogExtension.md#setDelegateCatalog) abstraction.
29+
30+
`setDelegateCatalog` sets this [CatalogPlugin](#delegate).
31+
32+
??? note "Final Method"
33+
`setDelegateCatalog` is a Java **final method** to prevent [subclasses](#implementations) from overriding or hiding it.
34+
35+
Learn more in the [Java Language Specification]({{ java.spec }}/jls-8.html#jls-8.4.3.3).

‎mkdocs.yml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -117,8 +117,9 @@ extra:
117117
version: 2.3.9
118118
api: https://svn.apache.org/repos/infra/websites/production/hive/content/javadocs/r2.3.9/api
119119
java:
120+
version: 17
120121
api: https://docs.oracle.com/en/java/javase/17/docs/api/java.base
121-
spec: https://docs.oracle.com/javase/specs/jls/se11/html
122+
spec: https://docs.oracle.com/javase/specs/jls/se17/html
122123
kafka:
123124
version: 3.4.1
124125
api: https://kafka.apache.org/34/javadoc

0 commit comments

Comments
 (0)