Layout NG: Folder structure and naming things, focusing on ETL and CDC

- Dissolve individual pages in category section `etl`, relocating them
  into dedicated items within the backbone section `integrate` instead.

- Relocated items: Azure Functions, Apache Iceberg, InfluxDB, MongoDB,
  MySQL and MariaDB, RisingWave, Streamsets.

- Dissolve weird page toc assembly on ETL and CDC category index pages,
  using `toctree` only for now.

Author: amotl

docs/_include/links.md

@@ -27,6 +27,7 @@
 [HNSW paper]: https://arxiv.org/pdf/1603.09320
 [HoloViews]: https://www.holoviews.org/
 [Indexing, Columnar Storage, and Aggregations]: https://cratedb.com/product/features/indexing-columnar-storage-aggregations
+[InfluxDB]: https://github.com/influxdata/influxdb
 [inverted index]: https://en.wikipedia.org/wiki/Inverted_index
 [JOIN]: inv:crate-reference#sql_joins
 [JSON Database]: https://cratedb.com/solutions/json-database
@@ -38,9 +39,12 @@
 [langchain-rag-sql-binder]: https://mybinder.org/v2/gh/crate/cratedb-examples/main?labpath=topic%2Fmachine-learning%2Flangchain%2Fcratedb-vectorstore-rag-openai-sql.ipynb
 [langchain-rag-sql-colab]: https://colab.research.google.com/github/crate/cratedb-examples/blob/main/topic/machine-learning/langchain/cratedb-vectorstore-rag-openai-sql.ipynb
 [langchain-rag-sql-github]: https://github.com/crate/cratedb-examples/blob/main/topic/machine-learning/langchain/cratedb-vectorstore-rag-openai-sql.ipynb
-[MongoDB CDC Relay]: https://cratedb-toolkit.readthedocs.io/io/mongodb/cdc.html
+[MongoDB]: https://www.mongodb.com/docs/manual/
+[MongoDB Atlas]: https://www.mongodb.com/docs/atlas/
+[MongoDB CDC Relay]: inv:ctk:*:label#mongodb-cdc-relay
 [MongoDB Change Streams]: https://www.mongodb.com/docs/manual/changeStreams/
-[MongoDB Table Loader]: https://cratedb-toolkit.readthedocs.io/io/mongodb/loader.html
+[MongoDB collections and databases]: https://www.mongodb.com/docs/php-library/current/databases-collections/
+[MongoDB Table Loader]: inv:ctk:*:label#mongodb-loader
 [Multi-model Database]: https://cratedb.com/solutions/multi-model-database
 [nearest neighbor search]: https://en.wikipedia.org/wiki/Nearest_neighbor_search
 [Nested Data Structure]: https://cratedb.com/product/features/nested-data-structure

docs/ingest/cdc/index.md

@@ -5,9 +5,8 @@
 :::
 
 :::{div}
-You have a variety of options to connect and integrate with 3rd-party
+CrateDB provides a variety of options to connect and integrate with third-party
 CDC applications, mostly using [CrateDB's PostgreSQL interface].
-
 CrateDB also provides a few native adapter components that can be used
 to leverage its advanced features. 
 
@@ -17,83 +16,9 @@ to use them optimally.
 Please also have a look at support for [generic ETL](#etl) solutions.
 :::
 
-(cdc-dms)=
-## AWS DMS
-
-:::{div}
-[AWS Database Migration Service (AWS DMS)] is a managed migration and replication
-service that helps move your database and analytics workloads between different
-kinds of databases quickly, securely, and with minimal downtime and zero data
-loss.
-
-AWS DMS supports migration between 20-plus database and analytics engines, either
-on-premises, or per EC2 instance databases. Supported data migration sources are:
-Amazon Aurora, Amazon DocumentDB, Amazon S3, IBM DB2, MariaDB, Azure SQL Database,
-Microsoft SQL Server, MongoDB, MySQL, Oracle, PostgreSQL, SAP ASE.
-
-The [AWS DMS Integration with CrateDB] uses Amazon Kinesis Data Streams as
-a DMS target, combined with a CrateDB-specific downstream processor element.
-
-CrateDB provides two variants how to conduct data migrations using AWS DMS.
-Either use it standalone / on your own premises, or use it in a completely
-managed environment with services of AWS and CrateDB Cloud.
-AWS DMS supports both `full-load` and `cdc` operation modes, often used in
-combination with each other (`full-load-and-cdc`).
-:::
-
-(cdc-kinesis)=
-## AWS Kinesis
-You can use Amazon Kinesis Data Streams to collect and process large streams of data
-records in real time. A typical Kinesis Data Streams application reads data from a
-data stream as data records.
-
-As such, a common application is to relay DynamoDB table change stream events to a
-Kinesis Stream, and consume that from an adapter to a consolidation database.
-:::{div}
-- About: [Amazon Kinesis Data Streams]
-- See: [](#cdc-dynamodb)
-:::
-
-## Debezium
-
+- {ref}`aws-dms`
+- {ref}`aws-dynamodb`
+- {ref}`aws-kinesis`
 - {ref}`debezium`
-
-(cdc-dynamodb)=
-## DynamoDB
-:::{div}
-Support for loading DynamoDB tables into CrateDB (full-load), as well as
-[Amazon DynamoDB Streams] and [Amazon Kinesis Data Streams],
-to relay CDC events from DynamoDB into CrateDB.
-
-- [DynamoDB Table Loader]
-- [DynamoDB CDC Relay]
-
-If you are looking into serverless replication using AWS Lambda:
-- [DynamoDB CDC Relay with AWS Lambda]
-- Blog: [Replicating CDC events from DynamoDB to CrateDB]
-:::
-
-## MongoDB
-:::{div}
-Support for loading MongoDB collections and databases into CrateDB (full-load),
-and [MongoDB Change Streams], to relay CDC events from MongoDB into CrateDB.
-
-- [MongoDB Table Loader]
-- [MongoDB CDC Relay]
-:::
-
-## StreamSets
-
-The [StreamSets Data Collector] is a lightweight and powerful engine that
-allows you to build streaming, batch and change-data-capture (CDC) pipelines
-that can ingest and transform data from a variety of different sources.
-
-StreamSets Data Collector Engine makes it easy to run data pipelines from Kafka,
-Oracle, Salesforce, JDBC, Hive, and more to Snowflake, Databricks, S3, ADLS, Kafka
-and more. Data Collector Engine runs on-premises or any cloud, wherever your data
-lives.
-
+- {ref}`mongodb`
 - {ref}`streamsets`
-
-
-[StreamSets Data Collector]: https://www.softwareag.com/en_corporate/platform/integration-apis/data-collector-engine.html

docs/ingest/etl/index.md

@@ -7,161 +7,36 @@
 :::
 
 :::{div}
-You have a variety of options to connect and integrate with 3rd-party
+CrateDB provides a variety of options to connect and integrate with third-party
 ETL applications, mostly using [CrateDB's PostgreSQL interface].
-:::
+CrateDB also provides a few native adapter components that can be used
+to leverage its advanced features.
 
 This documentation section lists corresponding ETL applications and
 frameworks which can be used together with CrateDB, and outlines how
 to use them optimally.
 Please also have a look at support for [](#cdc) solutions.
+:::
 
 
-## Apache Airflow / Astronomer
-
 - {ref}`apache-airflow`
-
-## Apache Flink
-
 - {ref}`apache-flink`
-
-## Apache Hop
-
 - {ref}`apache-hop`
-
-## Apache Iceberg / RisingWave
-:::{div}
-- {ref}`iceberg-risingwave`
-:::
-
-```{toctree}
-:hidden:
-
-iceberg-risingwave
-```
-
-## Apache Kafka
-
+- {ref}`apache-iceberg`
 - {ref}`apache-kafka`
-
-## Apache NiFi
-
 - {ref}`apache-nifi`
-
-## AWS DMS
-
-:::{div}
-[AWS Database Migration Service (AWS DMS)] is a managed migration and replication
-service that helps move your database and analytics workloads between different
-kinds of databases quickly, securely, and with minimal downtime and zero data
-loss. It supports migration between 20-plus database and analytics engines.
-
-AWS DMS supports migration between 20-plus database and analytics engines, either
-on-premises, or per EC2 instance databases. Supported data migration sources are:
-Amazon Aurora, Amazon DocumentDB, Amazon S3, IBM DB2, MariaDB, Azure SQL Database,
-Microsoft SQL Server, MongoDB, MySQL, Oracle, PostgreSQL, SAP ASE.
-
-The [AWS DMS Integration with CrateDB] uses Amazon Kinesis Data Streams as
-a DMS target, combined with a CrateDB-specific downstream processor element.
-
-CrateDB provides two variants how to conduct data migrations using AWS DMS.
-Either use it standalone / on your own premises, or use it in a completely
-managed environment with services of AWS and CrateDB Cloud.
-:::
-
-
-## AWS Kinesis
-
-Amazon Kinesis Data Streams is a serverless streaming data service that
-simplifies the capture, processing, and storage of data streams at any
-scale, such as application logs, website clickstreams, and IoT telemetry
-data, for machine learning (ML), analytics, and other applications.
-:::{div}
-The [DynamoDB CDC Relay] pipeline uses Amazon Kinesis to relay a table
-change stream from a DynamoDB table into a CrateDB table, see also
-[DynamoDB CDC](#cdc-dynamodb).
-:::
-
-
-## Azure Functions
-
+- {ref}`aws-dms`
+- {ref}`aws-dynamodb`
+- {ref}`aws-kinesis`
 - {ref}`azure-functions`
-
-```{toctree}
-:hidden:
-
-azure-functions
-```
-
-
-## dbt
-
 - {ref}`dbt`
-
-## DynamoDB
-:::{div}
-- [DynamoDB Table Loader]
-- [DynamoDB CDC Relay]
-:::
-
-
-## Estuary
-
 - {ref}`estuary`
-
-## InfluxDB
-
-- {ref}`integrate-influxdb`
-
-## Kestra
-
+- {ref}`influxdb`
 - {ref}`kestra`
-
-## Meltano
-
 - {ref}`meltano`
-
-## MongoDB
-:::{div}
-- Tutorial: {ref}`integrate-mongodb`
-- Documentation: [MongoDB Table Loader]
-- Documentation: [MongoDB CDC Relay]
-:::
-```{toctree}
-:hidden:
-
-mongodb
-```
-
-
-## MySQL
-
-- {ref}`integrate-mysql`
-
-```{toctree}
-:hidden:
-
-mysql
-```
-
-## Node-RED
-
+- {ref}`mongodb`
+- {ref}`mysql`
 - {ref}`node-red`
-
-## RisingWave
-
 - {ref}`risingwave`
-
-## SQL Server Integration Services
-
 - {ref}`sql-server`
-
-## StreamSets
-
 - {ref}`streamsets`
-
-```{toctree}
-:hidden:
-
-streamsets
-```

docs/ingest/telemetry/index.md

@@ -3,17 +3,14 @@
 (integrate-metrics)=
 # Telemetry data
 
+:::{div}
 CrateDB integrations with metrics collection agents, brokers, and stores.
 This documentation section lists applications and daemons which can
 be used together with CrateDB, and educates about how to use them optimally.
 
 Storing metrics data for the long term is a common need in systems monitoring
 scenarios. CrateDB offers corresponding integration adapters.
-
-## Prometheus
+:::
 
 - {ref}`prometheus`
-
-## Telegraf
-
 - {ref}`telegraf`

docs/integrate/apache-iceberg/index.md

@@ -0,0 +1,19 @@
+(apache-iceberg)=
+# Apache Iceberg
+
+:::{rubric} About
+:::
+The [Iceberg table format] is designed to manage a large, slow-changing collection
+of files in a distributed file system or key-value store as a database table.
+
+:::{rubric} Learn
+:::
+CrateDB provides integration capabilities with Apache Iceberg implementations,
+see {ref}`risingwave-iceberg`.
+
+:::{todo}
+🚧 _Please note this page is a work in progress._ 🚧
+:::
+
+
+[Iceberg table format]: https://iceberg.apache.org/spec/

docs/integrate/aws-dms/index.md

@@ -0,0 +1,36 @@
+(aws-dms)=
+(cdc-dms)=
+# AWS Database Migration Service
+
+:::{include} /_include/links.md
+:::
+
+:::{rubric} About
+:::
+
+:::{div}
+[AWS Database Migration Service (AWS DMS)] is a managed migration and replication
+service that helps move your database and analytics workloads between different
+kinds of databases quickly, securely, and with minimal downtime and zero data
+loss.
+
+AWS DMS supports migration between 20-plus database and analytics engines, either
+on-premises, or per EC2 instance databases. Supported data migration sources are:
+Amazon Aurora, Amazon DocumentDB, Amazon S3, IBM DB2, MariaDB, Azure SQL Database,
+Microsoft SQL Server, MongoDB, MySQL, Oracle, PostgreSQL, SAP ASE.
+:::
+
+:::{rubric} Learn
+:::
+
+:::{div}
+The [AWS DMS Integration with CrateDB] uses Amazon Kinesis Data Streams as
+a DMS target, combined with a CrateDB-specific downstream processor element.
+
+CrateDB provides two variants how to conduct data migrations using AWS DMS.
+Either use it standalone / on your own premises, or use it in a completely
+managed environment with services of AWS and CrateDB Cloud.
+
+AWS DMS supports both `full-load` and `cdc` operation modes, often used in
+combination with each other (`full-load-and-cdc`).
+:::

docs/integrate/aws-dynamodb/index.md

@@ -0,0 +1,27 @@
+(aws-dynamodb)=
+(cdc-dynamodb)=
+# Amazon DynamoDB
+
+:::{include} /_include/links.md
+:::
+
+:::{rubric} About
+:::
+
+:::{div}
+The [DynamoDB Table Loader] supports loading DynamoDB tables into CrateDB (full-load),
+while the [DynamoDB CDC Relay] pipeline uses [Amazon DynamoDB Streams] or [Amazon Kinesis
+Data Streams] to relay table change stream CDC events from a DynamoDB table into CrateDB.
+
+:::{rubric} Learn
+:::
+
+:::{div}
+It is a common application to relay DynamoDB table change stream events to a
+Kinesis Stream, and consume that from an adapter to write into an analytical
+or long-term storage consolidation database.
+
+If you are looking into serverless replication using AWS Lambda:
+- [DynamoDB CDC Relay with AWS Lambda]
+- Blog: [Replicating CDC events from DynamoDB to CrateDB]
+:::